<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>On 24/02/2012, at 10:14 AM, McDonald Richards wrote:</div><div><br class="Apple-interchange-newline"><blockquote type="cite"><div bgcolor="#FFFFFF"><div>I'm pretty sure no hardware or cables were an issue here. It's nice to blame them, but somebody somewhere made a mistake in a route-map.</div></div></blockquote><div><br></div><div>My read is that a primary router at ISP D with correct filtering failed, allowing a backup router with a separate Telstra interface to become the preferred path - perhaps as a result of a rushed replacement, or something that no-one noticed before. One of those scenarios that few of us are game to test on a production network.</div><div><br></div><div>I've Experienced a backup link with T that didn't have filtering applied, because the provisioning was drawn out over a long time - yet the localpref was absolutely applied.</div><div><br></div><div>The preference of yours truly is for multiple layers of filtering - tag routes wherever they come in to the network based on the same rules as the filter (hopefully making the lack of a filter more noticeable!), and then only allow routes into the mesh with a tag limiting mistakes to a single router - deny by default, filter trusted peers using community strings, and even use tags on statics and connected interfaces. There's a little more work on the route-maps, but you end up with a network that defaults to not making headlines.</div><div><br></div><div>John</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><br><blockquote type="cite"><div bgcolor="#FFFFFF"><div><br></div><div>Anybody have private capacity on endeavour and able to confirm if it did indeed "stop processing requests" ? ;)</div><div><br></div><div>Macca<br><br><div><br></div></div><div><br>On 24/02/2012, at 10:30 AM, Will Tardy <<a href="mailto:will@fetchtv.com.au">will@fetchtv.com.au</a>> wrote:<br><br></div><div></div><blockquote type="cite"><div><div>Telstra claims they had an international link down:</div><div><br></div><a href="http://www.zdnet.com.au/telstra-hit-by-nationwide-data-outage-339332310.htm">http://www.zdnet.com.au/telstra-hit-by-nationwide-data-outage-339332310.htm</a><div>
<br></div><div>If that happened at the same time as DODO incorrectly sending Telstra the full BGP table, could that explain why Telstra black-holed all-routes plus pumped all of it's own traffic via dodo?<br><br><div class="gmail_quote">
On 24 February 2012 10:02, Wade Millican <span dir="ltr"><<a href="mailto:Wade.Millican@echoent.com.au">Wade.Millican@echoent.com.au</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="font-size:14px;font-family:Calibri,sans-serif;word-wrap:break-word"><div>Hi All,</div><div><br></div><div>What I'm yet to understand about this outage is why DODO's AS_PATH was seen as shorter than anything Telstra already had.</div>
<div><br></div><div>An earlier posted look at routes(below), thanks Gavin, shows all routes from Telstra taking hops to DODO, then Optus or PIPE before moving to the destination. Surely Telstra would have had better routes than pushing all traffic 2 hops out of it's way.</div>
<div><br></div><div>AS_PATH does not explain how Telstra accepted these as the active routes. Even if all routes were accepted, Telstra still has better routes.</div><div><br></div><div>Can anyone explain what BGP Metric was modified/used that pushed traffic over longer AS_PATHs? </div>
<div><br></div><div><pre style="line-height:normal;text-indent:0px;letter-spacing:normal;text-align:-webkit-auto;font-variant:normal;text-transform:none;font-style:normal;white-space:pre-wrap;font-weight:normal;word-spacing:0px">*> <a href="http://1.22.161.0/24" target="_blank">1.22.161.0/24</a> 165.228.157.73 100 80 0 1221 38285 7474 7473 55410 45528 i
*> <a href="http://1.22.162.0/24" target="_blank">1.22.162.0/24</a> 165.228.157.73 100 80 0 1221 38285 7474 7473 55410 45528 i
*> <a href="http://1.22.163.0/24" target="_blank">1.22.163.0/24</a> 165.228.157.73 100 80 0 1221 38285 7474 7473 55410 45528 i
*> <a href="http://1.22.167.0/24" target="_blank">1.22.167.0/24</a> 165.228.157.73 100 80 0 1221 38285 7474 7473 6453 4755 45528 i
*> <a href="http://1.22.168.0/24" target="_blank">1.22.168.0/24</a> 165.228.157.73 100 80 0 1221 38285 7474 7473 6453 4755 45528 i
..
* <a href="http://14.201.64.0/24" target="_blank">14.201.64.0/24</a> 165.228.157.73 100 80 0 1221 38285 18398 7545 7545 i</pre></div><div><br></div><div>Thanks,</div><div><br></div><div>Wade</div><div>
<div><div><div>-- </div><div><font color="#959595"><font size="1"><font face="Georgia,Times New Roman"><span style="font-size:9pt">Wade Millican <br>Technical Consultant Team Lead<br></span></font></font></font><font size="1"><font face="Georgia, Times New Roman"><span style="font-size:9pt"><font color="#989898">Hemisphere Infrastructure Support<br>
</font><font color="#959595">Information Technology<br><b>Echo Entertainment Group Limited</b> <br><br>2 Edward St<br>Pyrmont NSW 2009 <br><br>T: <a href="tel:%2B61%202%209657%207460" value="+61296577460" target="_blank">+61 2 9657 7460</a><br>
M: <a href="tel:%2B61%20%280%29%20400%20192%20485" value="+61400192485" target="_blank">+61 (0) 400 192 485</a><br></font></span></font><span style="font-size:9pt"><font color="#0000FF"><font face="Times New Roman"><u><a>wade.millican@echoent.com.au</a><br>
</u></font></font><font face="Times New Roman"><font color="#959595"><a href="http://www.echoentertainment.com.au/" target="_blank">www.echoentertainment.com.au</a><br></font></font></span></font><font color="#7F7F7F"><font><font face="Arial Narrow"><span style="font-size:10pt"><78BFFC55-58BE-42A5-94D5-509927E7B33A.png></span></font></font></font></div>
</div></div></div><span><div style="border-right:medium none;padding-right:0in;padding-left:0in;padding-top:3pt;text-align:left;font-size:11pt;border-bottom:medium none;font-family:Calibri;border-top:#b5c4df 1pt solid;padding-bottom:0in;border-left:medium none">
<span style="font-weight:bold">From: </span> "Ramsay, Paul" <<a href="mailto:pramsay@uecomm.com.au" target="_blank">pramsay@uecomm.com.au</a>><br><span style="font-weight:bold">Date: </span> Wed, 22 Feb 2012 22:20:41 -0800<br>
<span style="font-weight:bold">To: </span> "<a href="mailto:ausnog@ausnog.net" target="_blank">ausnog@ausnog.net</a>" <<a href="mailto:ausnog@ausnog.net" target="_blank">ausnog@ausnog.net</a>><br><span style="font-weight:bold">Subject: </span> Re: [AusNOG] International link issue<br>
</div><div><br></div><div><div lang="EN-AU" link="blue" vlink="purple"><div><p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125);font-family:Calibri,sans-serif">Yes, this reinforces the Rule of Trust. Don’t trust your BGP
peers and ensure your filters are in place, configured correctly and working,
you can’t transfer blame.<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125);font-family:Calibri,sans-serif">It can cost you big $$ and pain if you inadvertently turn
yourself into a transit peer because your upstreams may prefer to send traffic
where they can make $$ from.<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125);font-family:Calibri,sans-serif"><u></u> <u></u></span></p><div><div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm"><p class="MsoNormal"><b><span lang="EN-US" style="font-size:10pt;font-family:Tahoma,sans-serif">From:</span></b><span lang="EN-US" style="font-size:10pt;font-family:Tahoma,sans-serif"> <a href="mailto:ausnog-bounces@lists.ausnog.net" target="_blank">ausnog-bounces@lists.ausnog.net</a>
[<a href="mailto:ausnog-bounces@lists.ausnog.net" target="_blank">mailto:ausnog-bounces@lists.ausnog.net</a>] <b>On Behalf Of </b>Sean K. Finn<br><b>Sent:</b> Thursday, 23 February 2012 5:09 PM<br><b>To:</b> <a href="mailto:'ausnog@ausnog.net" target="_blank">'ausnog@ausnog.net</a>'<br>
<b>Subject:</b> Re: [AusNOG] International link issue<u></u><u></u></span></p></div></div><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;color:rgb(31,73,125);font-family:Calibri,sans-serif">It’s easy to describe for all the media types watching..<u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;color:rgb(31,73,125);font-family:Calibri,sans-serif">(And I’m not sure why its not being put out there in Laymans
terms).<u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;color:rgb(31,73,125);font-family:Calibri,sans-serif"><u></u> <u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;color:rgb(31,73,125);font-family:Calibri,sans-serif">From the routes seen at various points, and reported on the WAIX
mailing list earlier..<u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;color:rgb(31,73,125);font-family:Calibri,sans-serif"><u></u> <u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;color:rgb(31,73,125);font-family:Calibri,sans-serif"><u></u> <u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;color:rgb(31,73,125);font-family:Calibri,sans-serif"><u></u> <u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;color:rgb(31,73,125);font-family:Calibri,sans-serif">Dodo told Telstra that Dodo was the rest of the Internet.<u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;color:rgb(31,73,125);font-family:Calibri,sans-serif"><u></u> <u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;color:rgb(31,73,125);font-family:Calibri,sans-serif">Telstra Believed Dodo.<u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;color:rgb(31,73,125);font-family:Calibri,sans-serif"><u></u> <u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;color:rgb(31,73,125);font-family:Calibri,sans-serif">Telstra entire system tried to use DODO as their ISP instead of
everyone else Telstra is connected to.<u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;color:rgb(31,73,125);font-family:Calibri,sans-serif"><u></u> <u></u></span></p><p class="MsoNormal">
<span lang="EN-US" style="font-size:11pt;color:rgb(31,73,125);font-family:Calibri,sans-serif">Needless to say this didn’t work, the pipes got Jammed.<u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;color:rgb(31,73,125);font-family:Calibri,sans-serif"><u></u> <u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;color:rgb(31,73,125);font-family:Calibri,sans-serif">Telstra should have filtered the announcement from Dodo, butdidn’t.<u></u><u></u></span></p><p class="MsoNormal">
<span lang="EN-US" style="font-size:11pt;color:rgb(31,73,125);font-family:Calibri,sans-serif"><u></u> <u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;color:rgb(31,73,125);font-family:Calibri,sans-serif">Filtering is in place as a form of control (which is used
instead of trust).<u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;color:rgb(31,73,125);font-family:Calibri,sans-serif"><u></u> <u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;color:rgb(31,73,125);font-family:Calibri,sans-serif">Filtering obviously wasn’t in place, or didn’t work, so anything
that Dodo told Telstra about where to find the Internet, Telstra believed.<u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;color:rgb(31,73,125);font-family:Calibri,sans-serif"><u></u> <u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;color:rgb(31,73,125);font-family:Calibri,sans-serif">This happens quite often, I’ve heard of this happening on
peering exchanges within Australia, too. Just never at an organizational level
as big as Telstra.<u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;color:rgb(31,73,125);font-family:Calibri,sans-serif"><u></u> <u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;color:rgb(31,73,125);font-family:Calibri,sans-serif">Over and Out.<u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;color:rgb(31,73,125);font-family:Calibri,sans-serif"><u></u> <u></u></span></p></div><div style="BORDER-RIGHT:medium none;PADDING-RIGHT:0cm;BORDER-TOP:windowtext 1pt solid;PADDING-LEFT:0cm;PADDING-BOTTOM:0cm;BORDER-LEFT:medium none;PADDING-TOP:1pt;BORDER-BOTTOM:medium none"><div style="border-right-width: medium; border-right-style: none; border-right-color: initial; padding-right: 0cm; border-top-width: medium; border-top-style: none; border-top-color: initial; padding-left: 0cm; padding-bottom: 0cm; margin-top: 0cm; margin-right: 0cm; margin-bottom: 0pt; margin-left: 0cm; border-left-width: medium; border-left-style: none; border-left-color: initial; padding-top: 0cm; border-bottom-width: medium; border-bottom-style: none; border-bottom-color: initial; "><span lang="EN-US" style="FONT-SIZE:9pt;LAYOUT-GRID-MODE:line"><u></u> <u></u></span></div>
</div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0pt; margin-left: 0cm; "><span lang="EN-US" style="FONT-SIZE:10pt;FONT-FAMILY:Arial">This
message and its attachments may contain legally privileged or confidential information. It is for the intended addressee(s) only.</span><span><u></u><u></u></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0pt; margin-left: 0cm; "><span style="FONT-SIZE:9pt;LAYOUT-GRID-MODE:line;FONT-FAMILY:Arial">If
you are not the intended recipient you must not disclose or use the information
contained in it. If you have received this email in error please notify us immediately by return email and delete the document.<u></u><u></u></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0pt; margin-left: 0cm; "><span lang="EN-US" style="FONT-SIZE:9pt;LAYOUT-GRID-MODE:line;FONT-FAMILY:Arial">Any
views expressed in this message are those of the individual sender, except where
the sender specifies and with authority, states them to be the views of the
Company.<u></u><u></u></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0pt; margin-left: 0cm; "><span style="FONT-SIZE:9pt;LAYOUT-GRID-MODE:line;FONT-FAMILY:Arial">Uecomm
accepts no liability for any damage caused by this email or its attachments due
to viruses, interference, interception, corruption or unauthorised
access.<u></u><u></u></span></div><div><br class="webkit-block-placeholder"></div><hr>
This e-mail message has been scanned for Viruses and Content and cleared by
<strong><font color="#ff8000">NetIQ MailMarshal</font> </strong><hr></div></div></span></div>
<br>_______________________________________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a><br>
<a href="http://lists.ausnog.net/mailman/listinfo/ausnog" target="_blank">http://lists.ausnog.net/mailman/listinfo/ausnog</a><br>
<br></blockquote></div><br></div>
</div></blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>AusNOG mailing list</span><br><span><a href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a></span><br><span><a href="http://lists.ausnog.net/mailman/listinfo/ausnog">http://lists.ausnog.net/mailman/listinfo/ausnog</a></span><br></div></blockquote></div>_______________________________________________<br>AusNOG mailing list<br><a href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a><br>http://lists.ausnog.net/mailman/listinfo/ausnog<br></blockquote></div><br></body></html>