<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div>#gold<br><br>[b]</div><div><br>On 23 Oct 2015, at 3:31 AM, Christopher Pollock <<a href="mailto:cpollock@twitch.tv">cpollock@twitch.tv</a>> wrote:<br><br></div><blockquote type="cite"><div><div dir="ltr">I wrote this down so as to not forget it, about my visit to 30 Ross. I call it "Reason For Outage: Wolverine"<div><br></div><div><p style="margin:0.357143em 0px;padding:0px;font-size:14px;line-height:1.3em;font-family:verdana,arial,helvetica,sans-serif">There are a lot of public datacentres that people use in Australia. Most of them have an affilliation or ownership in some way with a major telco, because what better way to connect with customers than to make it easy and available?</p><p style="margin:0.357143em 0px;padding:0px;font-size:14px;line-height:1.3em;font-family:verdana,arial,helvetica,sans-serif">It was a normal day, started like any other. Walked through the office, grateful that there was no water anywhere. Checked the airconditioning temperatures, and nothing was blowing up. It looked like the start of a good day.</p><p style="margin:0.357143em 0px;padding:0px;font-size:14px;line-height:1.3em;font-family:verdana,arial,helvetica,sans-serif">The NSW IX was in Sydney, nearly 1000 km from me, but the state itself was unmanned. Nothing /really/ ever went wrong there so we made do. Until today.</p><p style="margin:0.357143em 0px;padding:0px;font-size:14px;line-height:1.3em;font-family:verdana,arial,helvetica,sans-serif">I noticed an alert for a customer being offline in the public colo DC. Unusual, but not unheard of. Customers reboot routers all the time. When it didn’t come back, I decided they had disconnected the session for a reason, and sent them an email ‘hey do you know your peering is offline?’. They wrote back saying they thought it was us, so I checked the switch they were connected to, and sure enough the port was down. Hmm. Okay, probably a cabling fault or dislodged cable. I’d schedule in a time to go down and check it out, when I had more work lined up. Then another one droppped off. And a third. By the time we were five peers down we were in full panic station mode. Was our switch dying? God I hope not.</p><p style="margin:0.357143em 0px;padding:0px;font-size:14px;line-height:1.3em;font-family:verdana,arial,helvetica,sans-serif">15 minutes later I was on a train to the airport carrying only what I had on me, hopped on a plane, and two hours later I was in Sydney. By the time I got there, another 10 peers had dropped offline and the phones were running hot. I turned mine off so as to stop getting unhelpful calls and diverted it all back to reception.</p><p style="margin:0.357143em 0px;padding:0px;font-size:14px;line-height:1.3em;font-family:verdana,arial,helvetica,sans-serif">Now, to explain a little about how public datacentres often work, generally the colo provider would charge you an exorbinant amount to install cabling between racks or to run patch leads, in the thousands. However, anyone with a carrier license & cabling license and the right tools could run up their own in 15 minutes. This happened many times. Thousands of times. I would not be underestimating it to say that there were at least 5,000 unregulated, unregistered cables in that datacentre floor.</p><p style="margin:0.357143em 0px;padding:0px;font-size:14px;line-height:1.3em;font-family:verdana,arial,helvetica,sans-serif">When I finally rocked up to the DC, 15 of our 20 or so peers were offline. I ran over to our rack and checked the switch. It was fine, no errors. I ran TDR testing on the ports to check for cable lengths, connectivity, shorts, any kind of Layer 1 or 2 problem. All the cables registered as an open pair, meaning they were not connected at the other end. This was thoroughly confusing. So I checked the actual lengths on these TDR traces and they were actually showing as only 15m away. What the hell? Most of these cable runs with 50 - 80m - why did they stop at 15m?</p><p style="margin:0.357143em 0px;padding:0px;font-size:14px;line-height:1.3em;font-family:verdana,arial,helvetica,sans-serif">I walked out to about 15m and walked a circumference around the rack. When I rounded the corner, the blood drained from my face (as it so often does in these situations). I knew exactly what had happened.</p><p style="margin:0.357143em 0px;padding:0px;font-size:14px;line-height:1.3em;font-family:verdana,arial,helvetica,sans-serif">A new tech for the colo provider was not aware of a little thing called the Telecommunications Act which allows you to run these kinds of cables. So he’d gone through all the locally-paid patches, which were done in a specific colour, and figured out that anything not bright yellow must have been ‘illegal’. He had four floor tiles removed, and was standing over the cable pits, dual-weilding side-cutters, one in each hand. Cutting anything the wrong colour, like a boxer pounding away with left-right combos over and over. Slashing away at our infrastructure like Wolverine berzerker style. There was a pile of cables next to him that, I shit you not, was the size of a small car.</p><p style="margin:0.357143em 0px;padding:0px;font-size:14px;line-height:1.3em;font-family:verdana,arial,helvetica,sans-serif">Yelling and sprinting over, I demanded that he stop what he was doing. I was about to say ‘..and put them back the way they were when I realised he must have been at it for 6+ hours and reconnecting them all was going to be impossible. He’d destroyed the infrastructure for god knows how many businesses. Now, I’m pretty calm most of the time, even in the face of danger, but this .. this made me lose my shit.</p><blockquote style="margin:10px 0px 0px 15px;padding:0px 8px 1px;color:rgb(0,102,0);border:1px;font-size:14px;font-family:verdana,arial,helvetica,sans-serif"><p style="margin:0px;padding:0px;font-size:1em;line-height:1.3em">Me: WHAT ARE YOU DOING STOP<br style="margin-left:0px;margin-right:0px;margin-top:0px">DC Tech: I’m removing the inactive and unauthorised patches. I have an order from management to do it.<br style="margin-left:0px;margin-right:0px;margin-bottom:0px">Me: ARE YOU A F**KING IDIOT? DO YOU REALISE THAT THESE ARE ACTIVE TELECOMMUNICATIONS SERVICES AND THAT INTERFERING WITH OR DISCONNECTING THEM IS A FEDERAL OFFENSE UNDER THE TELECOMMUNICATIONS ACT 1997!? YOU CAN GO TO FUCKING JAIL FOR 10 YEARS FOR ONE AND YOU’VE DONE LIKE TWO HUNDRED NOW STOP BEFORE YOU RUIN ANYONE ELSE’S BUSINESS</p></blockquote><p style="margin:0.357143em 0px;padding:0px;font-size:14px;line-height:1.3em;font-family:verdana,arial,helvetica,sans-serif">Finally, it was someone else’s face going pale. He agreed to stop, and I ran to the nearest supply store, bought a few boxes of cables and supplies and set to furiously running new cables to all our customers. He helped me re-run cabling for all of our customers, and within maybe two hours they were all back online.</p><p style="margin:0.357143em 0px;padding:0px;font-size:14px;line-height:1.3em;font-family:verdana,arial,helvetica,sans-serif">We sent the colo provider an invoice for the expenses incurred during troubleshooting / rectification and they grudgingly agreed to pay for it.</p><p style="margin:0.357143em 0px;padding:0px;font-size:14px;line-height:1.3em;font-family:verdana,arial,helvetica,sans-serif">I still can’t get the image of that giant ball of cables out of my head. It was a horribly hybrid of a giant aborted fetus and an ugly medusa, thousands of RJ45 heads pointing in all directions.</p><p style="margin:0.357143em 0px;padding:0px;font-size:14px;line-height:1.3em;font-family:verdana,arial,helvetica,sans-serif">Heading back to the airport, I sat with my head in my hands, regretting a lost day’s work, and trying to figure out how I would word this Post Incident Report.</p><p style="margin:0.357143em 0px;padding:0px;font-size:14px;line-height:1.3em;font-family:verdana,arial,helvetica,sans-serif">Reason for outage: AAPT is the worst.</p><p style="margin:0.357143em 0px;padding:0px;font-size:14px;line-height:1.3em;font-family:verdana,arial,helvetica,sans-serif">Eventually I handed the PIR job over to someone else, as I’d long since lost the ability to be civil about it, there was only one thing left to do.</p><p style="margin:0.357143em 0px;padding:0px;font-size:14px;line-height:1.3em;font-family:verdana,arial,helvetica,sans-serif">Go to the pub, and cleanse the day with purifying beer.</p></div><div><br></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><font color="#666666" style="font-size:13.3333339691162px"><font style="font-family:Helvetica">--</font></font></div><div dir="ltr"><font color="#666666" style="font-size:13.3333339691162px"><font style="font-family:Helvetica">Christopher Pollock </font></font><font color="#999999" style="font-size:13.3333339691162px;font-family:Helvetica"> </font><font color="#cccccc" style="font-size:13.3333339691162px;font-family:Helvetica">| </font><font color="#999999" style="font-size:13.3333339691162px;font-family:Helvetica"> </font><b style="color:rgb(17,85,204);font-size:13.3333339691162px"><font color="#674ea7"><a href="http://Twitch.tv" target="_blank"><font color="#1155cc"><span style="font-size:13.3333339691162px">Twitc</span></font>h.tv</a></font></b><font color="#999999" style="font-size:13.3333339691162px;font-family:Helvetica"><a href="http://www.twitch.tv" target="_blank"> </a> </font><font color="#cccccc" style="font-size:13.3333339691162px;font-family:Helvetica">| </font><font color="#666666" style="font-size:13.3333339691162px;font-family:Helvetica"> Network Development Engineer</font><font style="font-size:13.3333339691162px;font-family:Helvetica"><font color="#666666"> </font><font style="color:rgb(102,102,102)"> </font><font color="#c0c0c0">| </font><font color="#666666"> 415-361-3042</font></font><font style="font-size:13.3333339691162px;font-family:Helvetica"><font color="#666666"> </font><font style="color:rgb(102,102,102)"> </font><font color="#c0c0c0">|</font><font style="color:rgb(102,102,102)"> </font><font color="#666666"> Skype–christopherpollock</font></font><font style="font-size:13.3333339691162px;font-family:Helvetica"><font color="#666666"> </font><font style="color:rgb(102,102,102)"> </font><font color="#c0c0c0">|</font><font style="color:rgb(102,102,102)"> </font><font color="#666666"> T</font></font><font color="#666666" style="font-size:13.3333339691162px;font-family:Helvetica">witter–chhopsky</font><div style="font-size:13.3333339691162px"><div><font color="#666666"><br></font></div><div><span style="border-collapse:collapse;font-family:Helvetica"><b><font color="#666666">Twitch in the News:</font></b></span></div><div><span style="border-collapse:collapse;font-family:Helvetica"><font color="#6600cc"><font color="#6600cc"><a href="http://www.cbsnews.com/8301-501465_162-57369949-501465/play-video-games-for-a-living-twitchtv-is-making-it-happen/" style="color:rgb(17,85,204)" target="_blank"><font color="#6600cc">CBS News</font></a><b> </b></font><b> </b></font><font color="#666666"> "For a video game lover, like myself, the site is addictive. "</font></span></div></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Thu, Oct 22, 2015 at 1:28 AM, Purdon, Bob <span dir="ltr"><<a href="mailto:bobp@purdon.id.au" target="_blank">bobp@purdon.id.au</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><span class=""><div class="gmail_quote">On 22 October 2015 at 13:54, Bevan Slattery <span dir="ltr"><<a href="mailto:bevan@slattery.net.au" target="_blank">bevan@slattery.net.au</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><div>OMG. Only to be topped by 530 Collins. I remember PIPE taking over the Ausbone rack there. I remember Steve or Bob saying they couldn't get the floor tiles to sit on the raised structure meaning the tiles were effectively "floating" on the cables :). The cabling in that rack was possibly the worst in living memory.</div></div></blockquote><br></div></span>I know I said that once, and Steve may well have also. IIRC, the raised floor was 300mm above the slab and the cables were pressed up against the underside of the floor tiles. I've probably got a photo somewhere. Had to stand and/or gently jump on one particular tile to get it to sit down on the stringers again.<br><br></div><div class="gmail_extra">In a move that obviously went against the convention at that site, I did in fact remove quite a few cables as part of cleaning that rack up, which in turn helped that tile fit just a little bit better.<br><br></div><div class="gmail_extra">I also recall running a cable or two in there before I was at PIPE and despite there being trays (which I actually used), the general cable routing convention seemed to be just go point A to point B, and if that means going diagonally across the room then that's what happened.<br></div><br><div class="gmail_extra">Wonder how many tonnes of cable there were under that floor? :-)<br></div></div>
<br>_______________________________________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a><br>
<a href="http://lists.ausnog.net/mailman/listinfo/ausnog" rel="noreferrer" target="_blank">http://lists.ausnog.net/mailman/listinfo/ausnog</a><br>
<br></blockquote></div><br></div>
</div></blockquote></body></html>