<div>Maybe I expect too much, but...</div><div><br></div>I wish the news would report more along the lines of what Aaron said, instead of saying that Telstras international link "stopped processing requests." Maybe not quite as technically detailed, but at least an accurate account.<div>
<br></div><div>Regarding a *hardware* failure on a *Cisco* border router somehow causing the breakage... How does a *hardware* failure somehow cause this (unless it just happens to be a perfect storm of configuration corruption), and for what purpose does dropping a vendor name achieve anything? Do vendors really sit back and let the reputation of their gear be tarnished by some epically fail disaster PR?</div>
<div><br></div><div>Is it really necessary to dumb it down to the lowest conceivable level? To me it just seems that if journos report this kind of stuff, the general public then thinks thats what *actually* happened (some probably literally), and you havent really informed anyone or made anyone any wiser or smarter.</div>
<div><br></div><div>I guess it really is just easier to blame it on some gremlins than to admit some people cocked up (big time) and there is actually someone to blame, and I do expect too much. :-)</div><div><br><br><div class="gmail_quote">
On 23 February 2012 23:58, Aaron Swayn <span dir="ltr"><<a href="mailto:aaron@swayn.com">aaron@swayn.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-AU" link="blue" vlink="purple"><div><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">From what I understand is the BGP interface between Telstra AUS (AS1221) and Reach aka Telstra worldwide (AS4637) went down because….<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Dodo advertised 390k prefixes to Telstra, which they accepted.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Telstra then advertised the 390k prefixes to Reach<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Reach, correctly assuming that Telstra should never have this many routes and shut down BGP due to ‘max-prefixes’ being breached.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">This causes much route flapping and some ISPs with route dampening, did just that to AS1221 prefixes to prevent CPU overload (Telstra advertise normally something like 800+ prefixes, or something around that number)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Telstra should have had a max-prefix in place on the Dodo peer to protect from this (although should be filtered correctly to protect its own customer base, but to what level is debatable. But as Dodo is not a Teir 1 carrier, I don’t think they should be that relaxed in the peering configuration IMHO. Only 3 carriers are Tier 1 in Australia and only they should be that relaxed to allow all prefixes. It seems Reach however doesn’t trust Telstra though).<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Reach did the right thing, as they are one of the few true global Tier 1 peering provider. Hence, Reach never expects to see the internet come from Telstra, only domestic routes which Telstra peers with.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I’m sure the Instructor lead training courses for CCNP and BGP will talk about this incident for the next 20 years on what not to do. I seem to recall one comment “You <u>don’t</u> want to become famous, so always check what you’re doing before you interface with the internet”.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> <a href="mailto:ausnog-bounces@lists.ausnog.net" target="_blank">ausnog-bounces@lists.ausnog.net</a> [mailto:<a href="mailto:ausnog-bounces@lists.ausnog.net" target="_blank">ausnog-bounces@lists.ausnog.net</a>] <b>On Behalf Of </b>Will Tardy</span></p>
<div class="im"><br><b>Sent:</b> Friday, 24 February 2012 10:30 AM<br></div><div><div class="h5"><b>To:</b> <a href="mailto:ausnog@ausnog.net" target="_blank">ausnog@ausnog.net</a><br><b>Subject:</b> Re: [AusNOG] International link issue<u></u><u></u></div>
</div><p></p><div><div class="h5"><p class="MsoNormal"><u></u> <u></u></p><div><p class="MsoNormal">Telstra claims they had an international link down:<u></u><u></u></p></div><div><p class="MsoNormal"><u></u> <u></u></p></div>
<p class="MsoNormal"><a href="http://www.zdnet.com.au/telstra-hit-by-nationwide-data-outage-339332310.htm" target="_blank">http://www.zdnet.com.au/telstra-hit-by-nationwide-data-outage-339332310.htm</a><u></u><u></u></p><div>
<p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal" style="margin-bottom:12.0pt">If that happened at the same time as DODO incorrectly sending Telstra the full BGP table, could that explain why Telstra black-holed all-routes plus pumped all of it's own traffic via dodo?<u></u><u></u></p>
<div><p class="MsoNormal">On 24 February 2012 10:02, Wade Millican <<a href="mailto:Wade.Millican@echoent.com.au" target="_blank">Wade.Millican@echoent.com.au</a>> wrote:<u></u><u></u></p><div><div><p class="MsoNormal">
<span style="font-size:10.5pt;font-family:"Calibri","sans-serif"">Hi All,<u></u><u></u></span></p></div><div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif""><u></u> <u></u></span></p>
</div><div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif"">What I'm yet to understand about this outage is why DODO's AS_PATH was seen as shorter than anything Telstra already had.<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif""><u></u> <u></u></span></p></div><div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif"">An earlier posted look at routes(below), thanks Gavin, shows all routes from Telstra taking hops to DODO, then Optus or PIPE before moving to the destination. Surely Telstra would have had better routes than pushing all traffic 2 hops out of it's way.<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif""><u></u> <u></u></span></p></div><div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif"">AS_PATH does not explain how Telstra accepted these as the active routes. Even if all routes were accepted, Telstra still has better routes.<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif""><u></u> <u></u></span></p></div><div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif"">Can anyone explain what BGP Metric was modified/used that pushed traffic over longer AS_PATHs? <u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif""><u></u> <u></u></span></p></div><div><pre style="text-align:-webkit-auto;white-space:pre-wrap;word-spacing:0px">
<u></u> <u></u></pre><pre>*> <a href="http://1.22.161.0/24" target="_blank">1.22.161.0/24</a> 165.228.157.73 100 80 0 1221 38285 7474 7473 55410 45528 i<u></u><u></u></pre><pre>*> <a href="http://1.22.162.0/24" target="_blank">1.22.162.0/24</a> 165.228.157.73 100 80 0 1221 38285 7474 7473 55410 45528 i<u></u><u></u></pre>
<pre>*> <a href="http://1.22.163.0/24" target="_blank">1.22.163.0/24</a> 165.228.157.73 100 80 0 1221 38285 7474 7473 55410 45528 i<u></u><u></u></pre><pre>*> <a href="http://1.22.167.0/24" target="_blank">1.22.167.0/24</a> 165.228.157.73 100 80 0 1221 38285 7474 7473 6453 4755 45528 i<u></u><u></u></pre>
<pre>*> <a href="http://1.22.168.0/24" target="_blank">1.22.168.0/24</a> 165.228.157.73 100 80 0 1221 38285 7474 7473 6453 4755 45528 i<u></u><u></u></pre><pre>..<u></u><u></u></pre><pre>* <a href="http://14.201.64.0/24" target="_blank">14.201.64.0/24</a> 165.228.157.73 100 80 0 1221 38285 18398 7545 7545 i<u></u><u></u></pre>
</div><div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif""><u></u> <u></u></span></p></div><div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif"">Thanks,<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif""><u></u> <u></u></span></p></div><div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif"">Wade<u></u><u></u></span></p>
</div><div><div><div><div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif"">-- <u></u><u></u></span></p></div><div><p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Georgia","serif";color:#959595">Wade Millican <br>
Technical Consultant Team Lead<br></span><span style="font-size:9.0pt;font-family:"Georgia","serif";color:#989898">Hemisphere Infrastructure Support<br></span><span style="font-size:9.0pt;font-family:"Georgia","serif";color:#959595">Information Technology<br>
<b>Echo Entertainment Group Limited</b> <br><br>2 Edward St<br>Pyrmont NSW 2009 <br><br>T: <a href="tel:%2B61%202%209657%207460" target="_blank">+61 2 9657 7460</a><br>M: <a href="tel:%2B61%20%280%29%20400%20192%20485" target="_blank">+61 (0) 400 192 485</a><br>
</span><u><span style="font-size:9.0pt;color:blue"><a href="mailto:wade.millican@echoent.com.au" target="_blank">wade.millican@echoent.com.au</a><br></span></u><span style="font-size:9.0pt;color:#959595"><a href="http://www.echoentertainment.com.au" target="_blank">www.echoentertainment.com.au</a><br>
</span><span style="font-size:10.0pt;font-family:"Arial Narrow","sans-serif";color:#7f7f7f"><img border="0" width="309" height="89" src="cid:image001.png@01CCF2E3.4D4BE820"></span><span style="font-size:10.5pt;font-family:"Calibri","sans-serif""><u></u><u></u></span></p>
</div></div></div></div><div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm"><p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From: </span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">"Ramsay, Paul" <<a href="mailto:pramsay@uecomm.com.au" target="_blank">pramsay@uecomm.com.au</a>><br>
<b>Date: </b>Wed, 22 Feb 2012 22:20:41 -0800<br><b>To: </b>"<a href="mailto:ausnog@ausnog.net" target="_blank">ausnog@ausnog.net</a>" <<a href="mailto:ausnog@ausnog.net" target="_blank">ausnog@ausnog.net</a>><br>
<b>Subject: </b>Re: [AusNOG] International link issue<u></u><u></u></span></p></div><div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif""><u></u> <u></u></span></p>
</div><div><div><div><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Yes, this reinforces the Rule of Trust. Don’t trust your BGP peers and ensure your filters are in place, configured correctly and working, you can’t transfer blame.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">It can cost you big $$ and pain if you inadvertently turn yourself into a transit peer because your upstreams may prefer to send traffic where they can make $$ from.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p><div><div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> <a href="mailto:ausnog-bounces@lists.ausnog.net" target="_blank">ausnog-bounces@lists.ausnog.net</a> [<a href="mailto:ausnog-bounces@lists.ausnog.net" target="_blank">mailto:ausnog-bounces@lists.ausnog.net</a>] <b>On Behalf Of </b>Sean K. Finn<br>
<b>Sent:</b> Thursday, 23 February 2012 5:09 PM<br><b>To:</b> <a href="mailto:'ausnog@ausnog.net" target="_blank">'ausnog@ausnog.net</a>'<br><b>Subject:</b> Re: [AusNOG] International link issue</span><u></u><u></u></p>
</div></div><p class="MsoNormal"> <u></u><u></u></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">It’s easy to describe for all the media types watching..</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">(And I’m not sure why its not being put out there in Laymans terms).</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">From the routes seen at various points, and reported on the WAIX mailing list earlier..</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Dodo told Telstra that Dodo was the rest of the Internet.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Telstra Believed Dodo.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Telstra entire system tried to use DODO as their ISP instead of everyone else Telstra is connected to.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Needless to say this didn’t work, the pipes got Jammed.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Telstra should have filtered the announcement from Dodo, butdidn’t.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Filtering is in place as a form of control (which is used instead of trust).</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Filtering obviously wasn’t in place, or didn’t work, so anything that Dodo told Telstra about where to find the Internet, Telstra believed.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">This happens quite often, I’ve heard of this happening on peering exchanges within Australia, too. Just never at an organizational level as big as Telstra.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Over and Out.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p></div><div style="border:none;border-top:solid windowtext 1.0pt;padding:1.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;layout-grid-mode:line"> </span><u></u><u></u></p></div><p style="margin:0cm;margin-bottom:.0001pt"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial","sans-serif"">This message and its attachments may contain legally privileged or confidential information. It is for the intended addressee(s) only.</span><span style="font-size:10.5pt;font-family:"Calibri","sans-serif""><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";layout-grid-mode:line">If you are not the intended recipient you must not disclose or use the information contained in it. If you have received this email in error please notify us immediately by return email and delete the document.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;font-family:"Arial","sans-serif";layout-grid-mode:line">Any views expressed in this message are those of the individual sender, except where the sender specifies and with authority, states them to be the views of the Company.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";layout-grid-mode:line">Uecomm accepts no liability for any damage caused by this email or its attachments due to viruses, interference, interception, corruption or unauthorised access.</span><u></u><u></u></p>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif""><hr size="2" width="100%" align="center"></span></div><p class="MsoNormal">
<span style="font-size:10.5pt;font-family:"Calibri","sans-serif"">This e-mail message has been scanned for Viruses and Content and cleared by <strong><span style="font-family:"Calibri","sans-serif";color:#ff8000">NetIQ MailMarshal</span></strong><strong><span style="font-family:"Calibri","sans-serif""> </span></strong><u></u><u></u></span></p>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif""><hr size="2" width="100%" align="center"></span></div></div></div></div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>_______________________________________________<br>AusNOG mailing list<br><a href="mailto:AusNOG@lists.ausnog.net" target="_blank">AusNOG@lists.ausnog.net</a><br><a href="http://lists.ausnog.net/mailman/listinfo/ausnog" target="_blank">http://lists.ausnog.net/mailman/listinfo/ausnog</a><u></u><u></u></p>
</div><p class="MsoNormal"><u></u> <u></u></p></div></div></div></div></div><br>_______________________________________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a><br>
<a href="http://lists.ausnog.net/mailman/listinfo/ausnog" target="_blank">http://lists.ausnog.net/mailman/listinfo/ausnog</a><br>
<br></blockquote></div><br></div>