[AusNOG] International link issue
Matthew Moyle-Croft
mmc at mmc.com.au
Fri Feb 24 12:26:28 EST 2012
Suspect what happened was:
1) Dodo starts advertising full table to Telstra
2) Telstra don't filter, don't max prefix Dodo so carried this into their BGP Table.
3) Telstra started advertising Dodo's advertisements out their transit/peering ports as Dodo's AS is customer one and they don't filter that properly either.
4) "International" links go down as Telstra's transit provider and peers (eg. Reach etc) start hitting their max-prefix filters and dropping BGP hence why AS1221 went away.
5) Hilarity rains down upon us all.
One thing I didn't see (was behind someone using AS1221 at the time :-( ) was what happened to Telstra's IPv6 table? They have IPv6 and wondering if AS1221's IPv6 routes disappeared at the same time?
MMC
On 24/02/2012, at 10:28 AM, Andree Toonk wrote:
> That would explain why all of Telstra and many other prefixes (~1400)
> disappeared from the BGP tables:
>
> http://www.bgpmon.net/telstra-feb23-2012.png
> https://twitter.com/#!/bgpmon/status/172608854855647233/photo/1
>
> Though, surely Telstra has more than one international link...
>
> Andree
>
>
> .-- My secret spy satellite informs me that at 12-02-23 3:30 PM Will
> Tardy wrote:
>> Telstra claims they had an international link down:
>>
>> http://www.zdnet.com.au/telstra-hit-by-nationwide-data-outage-339332310.htm
>>
>> If that happened at the same time as DODO incorrectly sending Telstra
>> the full BGP table, could that explain why Telstra black-holed
>> all-routes plus pumped all of it's own traffic via dodo?
>>
>> On 24 February 2012 10:02, Wade Millican <Wade.Millican at echoent.com.au
>> <mailto:Wade.Millican at echoent.com.au>> wrote:
>>
>> Hi All,
>>
>> What I'm yet to understand about this outage is why DODO's AS_PATH
>> was seen as shorter than anything Telstra already had.
>>
>> An earlier posted look at routes(below), thanks Gavin, shows all
>> routes from Telstra taking hops to DODO, then Optus or PIPE before
>> moving to the destination. Surely Telstra would have had better
>> routes than pushing all traffic 2 hops out of it's way.
>>
>> AS_PATH does not explain how Telstra accepted these as the active
>> routes. Even if all routes were accepted, Telstra still has better
>> routes.
>>
>> Can anyone explain what BGP Metric was modified/used that pushed
>> traffic over longer AS_PATHs?
>>
>> *> 1.22.161.0/24 <http://1.22.161.0/24> 165.228.157.73 100 80 0 1221 38285 7474 7473 55410 45528 i
>> *> 1.22.162.0/24 <http://1.22.162.0/24> 165.228.157.73 100 80 0 1221 38285 7474 7473 55410 45528 i
>> *> 1.22.163.0/24 <http://1.22.163.0/24> 165.228.157.73 100 80 0 1221 38285 7474 7473 55410 45528 i
>> *> 1.22.167.0/24 <http://1.22.167.0/24> 165.228.157.73 100 80 0 1221 38285 7474 7473 6453 4755 45528 i
>> *> 1.22.168.0/24 <http://1.22.168.0/24> 165.228.157.73 100 80 0 1221 38285 7474 7473 6453 4755 45528 i
>> ..
>> * 14.201.64.0/24 <http://14.201.64.0/24> 165.228.157.73 100 80 0 1221 38285 18398 7545 7545 i
>>
>>
>> Thanks,
>>
>> Wade
>> --
>> Wade Millican
>> Technical Consultant Team Lead
>> Hemisphere Infrastructure Support
>> Information Technology
>> *Echo Entertainment Group Limited*
>>
>> 2 Edward St
>> Pyrmont NSW 2009
>>
>> T: +61 2 9657 7460 <tel:%2B61%202%209657%207460>
>> M: +61 (0) 400 192 485 <tel:%2B61%20%280%29%20400%20192%20485>
>> _wade.millican at echoent.com.au
>> _www.echoentertainment.com.au <http://www.echoentertainment.com.au>
>> From: "Ramsay, Paul" <pramsay at uecomm.com.au
>> <mailto:pramsay at uecomm.com.au>>
>> Date: Wed, 22 Feb 2012 22:20:41 -0800
>> To: "ausnog at ausnog.net <mailto:ausnog at ausnog.net>"
>> <ausnog at ausnog.net <mailto:ausnog at ausnog.net>>
>> Subject: Re: [AusNOG] International link issue
>>
>> Yes, this reinforces the Rule of Trust. Don’t trust your BGP peers
>> and ensure your filters are in place, configured correctly and
>> working, you can’t transfer blame.____
>>
>> It can cost you big $$ and pain if you inadvertently turn yourself
>> into a transit peer because your upstreams may prefer to send
>> traffic where they can make $$ from.____
>>
>> __ __
>>
>> *From:*ausnog-bounces at lists.ausnog.net
>> <mailto:ausnog-bounces at lists.ausnog.net>
>> [mailto:ausnog-bounces at lists.ausnog.net] *On Behalf Of *Sean K. Finn
>> *Sent:* Thursday, 23 February 2012 5:09 PM
>> *To:* 'ausnog at ausnog.net <mailto:'ausnog at ausnog.net>'
>> *Subject:* Re: [AusNOG] International link issue____
>>
>> __ __
>>
>> It’s easy to describe for all the media types watching..____
>>
>> (And I’m not sure why its not being put out there in Laymans terms).____
>>
>> __ __
>>
>> From the routes seen at various points, and reported on the WAIX
>> mailing list earlier..____
>>
>> __ __
>>
>> __ __
>>
>> __ __
>>
>> Dodo told Telstra that Dodo was the rest of the Internet.____
>>
>> __ __
>>
>> Telstra Believed Dodo.____
>>
>> __ __
>>
>> Telstra entire system tried to use DODO as their ISP instead of
>> everyone else Telstra is connected to.____
>>
>> __ __
>>
>> Needless to say this didn’t work, the pipes got Jammed.____
>>
>> __ __
>>
>> Telstra should have filtered the announcement from Dodo, butdidn’t.____
>>
>> __ __
>>
>> Filtering is in place as a form of control (which is used instead of
>> trust).____
>>
>> __ __
>>
>> Filtering obviously wasn’t in place, or didn’t work, so anything
>> that Dodo told Telstra about where to find the Internet, Telstra
>> believed.____
>>
>> __ __
>>
>> This happens quite often, I’ve heard of this happening on peering
>> exchanges within Australia, too. Just never at an organizational
>> level as big as Telstra.____
>>
>> __ __
>>
>> Over and Out.____
>>
>> __ __
>>
>> __ __
>>
>> This message and its attachments may contain legally privileged or
>> confidential information. It is for the intended addressee(s) only.____
>>
>> If you are not the intended recipient you must not disclose or use
>> the information contained in it. If you have received this email in
>> error please notify us immediately by return email and delete the
>> document.____
>>
>> Any views expressed in this message are those of the individual
>> sender, except where the sender specifies and with authority, states
>> them to be the views of the Company.____
>>
>> Uecomm accepts no liability for any damage caused by this email or
>> its attachments due to viruses, interference, interception,
>> corruption or unauthorised access.____
>>
>> ------------------------------------------------------------------------
>> This e-mail message has been scanned for Viruses and Content and
>> cleared by *NetIQ MailMarshal *
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net <mailto:AusNOG at lists.ausnog.net>
>> http://lists.ausnog.net/mailman/listinfo/ausnog
>>
>>
>>
>>
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> http://lists.ausnog.net/mailman/listinfo/ausnog
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog
More information about the AusNOG
mailing list