[AusNOG] International link issue
XiTatiON
xitation at gmail.com
Fri Feb 24 11:40:07 EST 2012
You haven't lived until you break the internet... :)
On 24/02/2012 11:22 AM, Burt Mascareigne wrote:
>
> Can you imagine that engineers face, when it all blew up. Oh.. I wish
> I was there to see it L
>
> *From:*ausnog-bounces at lists.ausnog.net
> [mailto:ausnog-bounces at lists.ausnog.net] *On Behalf Of *Aaron Swayn
> *Sent:* Friday, 24 February 2012 11:20 AM
> *To:* ausnog at ausnog.net
> *Subject:* Re: [AusNOG] International link issue
>
> I think the spin was from Dodo to blame the vendor, but me thinks the
> fault went down something like this...
>
> -Dodo engineer troubleshooting BGP advertisement issue
>
> -Can't figure it out (can't see why, these things do work properly and
> you can debug them easily enough without having to take risks)
>
> -Removed outbound route filter from BGP session to Telstra to see if
> that fixes the problem
>
> And we know the rest...
>
> *From:*ausnog-bounces at lists.ausnog.net
> [mailto:ausnog-bounces at lists.ausnog.net] *On Behalf Of *Evan Weston
> *Sent:* Friday, 24 February 2012 10:55 AM
> *To:* ausnog at ausnog.net
> *Subject:* Re: [AusNOG] International link issue
>
> Nice spin from Telstra. Blame the hardware vendor, blame the customer
> but never admit that it was actually **our fault** for not filtering
> properly.
>
> *From:*ausnog-bounces at lists.ausnog.net
> <mailto:ausnog-bounces at lists.ausnog.net>
> [mailto:ausnog-bounces at lists.ausnog.net]
> <mailto:[mailto:ausnog-bounces at lists.ausnog.net]> *On Behalf Of *Will
> Tardy
> *Sent:* Friday, 24 February 2012 10:30 AM
> *To:* ausnog at ausnog.net <mailto:ausnog at ausnog.net>
> *Subject:* Re: [AusNOG] International link issue
>
> Telstra claims they had an international link down:
>
> http://www.zdnet.com.au/telstra-hit-by-nationwide-data-outage-339332310.htm
>
> If that happened at the same time as DODO incorrectly sending Telstra
> the full BGP table, could that explain why Telstra black-holed
> all-routes plus pumped all of it's own traffic via dodo?
>
> On 24 February 2012 10:02, Wade Millican <Wade.Millican at echoent.com.au
> <mailto:Wade.Millican at echoent.com.au>> wrote:
>
> Hi All,
>
> What I'm yet to understand about this outage is why DODO's AS_PATH was
> seen as shorter than anything Telstra already had.
>
> An earlier posted look at routes(below), thanks Gavin, shows all
> routes from Telstra taking hops to DODO, then Optus or PIPE before
> moving to the destination. Surely Telstra would have had better routes
> than pushing all traffic 2 hops out of it's way.
>
> AS_PATH does not explain how Telstra accepted these as the active
> routes. Even if all routes were accepted, Telstra still has better routes.
>
> Can anyone explain what BGP Metric was modified/used that pushed
> traffic over longer AS_PATHs?
>
>
> *> 1.22.161.0/24 <http://1.22.161.0/24> 165.228.157.73 100 80 0 1221 38285 7474 7473 55410 45528 i
> *> 1.22.162.0/24 <http://1.22.162.0/24> 165.228.157.73 100 80 0 1221 38285 7474 7473 55410 45528 i
> *> 1.22.163.0/24 <http://1.22.163.0/24> 165.228.157.73 100 80 0 1221 38285 7474 7473 55410 45528 i
> *> 1.22.167.0/24 <http://1.22.167.0/24> 165.228.157.73 100 80 0 1221 38285 7474 7473 6453 4755 45528 i
> *> 1.22.168.0/24 <http://1.22.168.0/24> 165.228.157.73 100 80 0 1221 38285 7474 7473 6453 4755 45528 i
> ..
> *14.201.64.0/24 <http://14.201.64.0/24> 165.228.157.73 100 80 0 1221 38285 18398 7545 7545 i
>
> Thanks,
>
> Wade
>
> --
>
> Wade Millican
> Technical Consultant Team Lead
> Hemisphere Infrastructure Support
> Information Technology
> *Echo Entertainment Group Limited*
>
> 2 Edward St
> Pyrmont NSW 2009
>
> T: +61 2 9657 7460 <tel:%2B61%202%209657%207460>
> M: +61 (0) 400 192 485 <tel:%2B61%20%280%29%20400%20192%20485>
> _wade.millican at echoent.com.au <mailto:wade.millican at echoent.com.au>
> _www.echoentertainment.com.au <http://www.echoentertainment.com.au>
>
> *From: *"Ramsay, Paul" <pramsay at uecomm.com.au
> <mailto:pramsay at uecomm.com.au>>
> *Date: *Wed, 22 Feb 2012 22:20:41 -0800
> *To: *"ausnog at ausnog.net <mailto:ausnog at ausnog.net>"
> <ausnog at ausnog.net <mailto:ausnog at ausnog.net>>
> *Subject: *Re: [AusNOG] International link issue
>
> Yes, this reinforces the Rule of Trust. Don't trust your BGP peers and
> ensure your filters are in place, configured correctly and working,
> you can't transfer blame.
>
> It can cost you big $$ and pain if you inadvertently turn yourself
> into a transit peer because your upstreams may prefer to send traffic
> where they can make $$ from.
>
> *From:*ausnog-bounces at lists.ausnog.net
> <mailto:ausnog-bounces at lists.ausnog.net>
> [mailto:ausnog-bounces at lists.ausnog.net] *On Behalf Of *Sean K. Finn
> *Sent:* Thursday, 23 February 2012 5:09 PM
> *To:* 'ausnog at ausnog.net <mailto:%27ausnog at ausnog.net>'
> *Subject:* Re: [AusNOG] International link issue
>
> It's easy to describe for all the media types watching..
>
> (And I'm not sure why its not being put out there in Laymans terms).
>
> From the routes seen at various points, and reported on the WAIX
> mailing list earlier..
>
> Dodo told Telstra that Dodo was the rest of the Internet.
>
> Telstra Believed Dodo.
>
> Telstra entire system tried to use DODO as their ISP instead of
> everyone else Telstra is connected to.
>
> Needless to say this didn't work, the pipes got Jammed.
>
> Telstra should have filtered the announcement from Dodo, butdidn't.
>
> Filtering is in place as a form of control (which is used instead of
> trust).
>
> Filtering obviously wasn't in place, or didn't work, so anything that
> Dodo told Telstra about where to find the Internet, Telstra believed.
>
> This happens quite often, I've heard of this happening on peering
> exchanges within Australia, too. Just never at an organizational level
> as big as Telstra.
>
> Over and Out.
>
> This message and its attachments may contain legally privileged or
> confidential information. It is for the intended addressee(s) only.
>
> If you are not the intended recipient you must not disclose or use the
> information contained in it. If you have received this email in error
> please notify us immediately by return email and delete the document.
>
> Any views expressed in this message are those of the individual
> sender, except where the sender specifies and with authority, states
> them to be the views of the Company.
>
> Uecomm accepts no liability for any damage caused by this email or its
> attachments due to viruses, interference, interception, corruption or
> unauthorised access.
>
> ------------------------------------------------------------------------
>
> This e-mail message has been scanned for Viruses and Content and
> cleared by *NetIQ MailMarshal***
>
> ------------------------------------------------------------------------
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net <mailto:AusNOG at lists.ausnog.net>
> http://lists.ausnog.net/mailman/listinfo/ausnog
>
> ------------------------------------------------------------------------
>
> Stockland Notice: If this communication has been sent to you by
> mistake, please delete and notify us. If it has been sent to you by
> mistake, legal privilege is not waived or lost and you are not
> entitled to use it in any way. Stockland and its subsidiaries reserve
> the right to monitor e-mail communication through its networks.
>
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20120224/0f460062/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 13740 bytes
Desc: not available
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20120224/0f460062/attachment.png>
More information about the AusNOG
mailing list