[AusNOG] International link issue
Mark Doorey
SBS.User at netmark.net.au
Fri Feb 24 13:08:16 EST 2012
I was using IPv6 and IPv4 at the time all the IPv6 traffic seemed fine.
Whereas the IPv4 traffic had a meltdown.
Netmark Computer Solutions
If you go online unprotected you are a bloody idiot
Mark Dooréy
On 24/02/2012 12:26 PM, Matthew Moyle-Croft wrote:
> Suspect what happened was:
>
> 1) Dodo starts advertising full table to Telstra
> 2) Telstra don't filter, don't max prefix Dodo so carried this into their BGP Table.
> 3) Telstra started advertising Dodo's advertisements out their transit/peering ports as Dodo's AS is customer one and they don't filter that properly either.
> 4) "International" links go down as Telstra's transit provider and peers (eg. Reach etc) start hitting their max-prefix filters and dropping BGP hence why AS1221 went away.
> 5) Hilarity rains down upon us all.
>
> One thing I didn't see (was behind someone using AS1221 at the time :-( ) was what happened to Telstra's IPv6 table? They have IPv6 and wondering if AS1221's IPv6 routes disappeared at the same time?
>
> MMC
>
> On 24/02/2012, at 10:28 AM, Andree Toonk wrote:
>
>> That would explain why all of Telstra and many other prefixes (~1400)
>> disappeared from the BGP tables:
>>
>> http://www.bgpmon.net/telstra-feb23-2012.png
>> https://twitter.com/#!/bgpmon/status/172608854855647233/photo/1
>>
>> Though, surely Telstra has more than one international link...
>>
>> Andree
>>
>>
>> .-- My secret spy satellite informs me that at 12-02-23 3:30 PM Will
>> Tardy wrote:
>>> Telstra claims they had an international link down:
>>>
>>> http://www.zdnet.com.au/telstra-hit-by-nationwide-data-outage-339332310.htm
>>>
>>> If that happened at the same time as DODO incorrectly sending Telstra
>>> the full BGP table, could that explain why Telstra black-holed
>>> all-routes plus pumped all of it's own traffic via dodo?
>>>
>>> On 24 February 2012 10:02, Wade Millican<Wade.Millican at echoent.com.au
>>> <mailto:Wade.Millican at echoent.com.au>> wrote:
>>>
>>> Hi All,
>>>
>>> What I'm yet to understand about this outage is why DODO's AS_PATH
>>> was seen as shorter than anything Telstra already had.
>>>
>>> An earlier posted look at routes(below), thanks Gavin, shows all
>>> routes from Telstra taking hops to DODO, then Optus or PIPE before
>>> moving to the destination. Surely Telstra would have had better
>>> routes than pushing all traffic 2 hops out of it's way.
>>>
>>> AS_PATH does not explain how Telstra accepted these as the active
>>> routes. Even if all routes were accepted, Telstra still has better
>>> routes.
>>>
>>> Can anyone explain what BGP Metric was modified/used that pushed
>>> traffic over longer AS_PATHs?
>>>
>>> *> 1.22.161.0/24<http://1.22.161.0/24> 165.228.157.73 100 80 0 1221 38285 7474 7473 55410 45528 i
>>> *> 1.22.162.0/24<http://1.22.162.0/24> 165.228.157.73 100 80 0 1221 38285 7474 7473 55410 45528 i
>>> *> 1.22.163.0/24<http://1.22.163.0/24> 165.228.157.73 100 80 0 1221 38285 7474 7473 55410 45528 i
>>> *> 1.22.167.0/24<http://1.22.167.0/24> 165.228.157.73 100 80 0 1221 38285 7474 7473 6453 4755 45528 i
>>> *> 1.22.168.0/24<http://1.22.168.0/24> 165.228.157.73 100 80 0 1221 38285 7474 7473 6453 4755 45528 i
>>> ..
>>> * 14.201.64.0/24<http://14.201.64.0/24> 165.228.157.73 100 80 0 1221 38285 18398 7545 7545 i
>>>
>>>
>>> Thanks,
>>>
>>> Wade
>>> --
>>> Wade Millican
>>> Technical Consultant Team Lead
>>> Hemisphere Infrastructure Support
>>> Information Technology
>>> *Echo Entertainment Group Limited*
>>>
>>> 2 Edward St
>>> Pyrmont NSW 2009
>>>
>>> T: +61 2 9657 7460<tel:%2B61%202%209657%207460>
>>> M: +61 (0) 400 192 485<tel:%2B61%20%280%29%20400%20192%20485>
>>> _wade.millican at echoent.com.au
>>> _www.echoentertainment.com.au<http://www.echoentertainment.com.au>
>>> From: "Ramsay, Paul"<pramsay at uecomm.com.au
>>> <mailto:pramsay at uecomm.com.au>>
>>> Date: Wed, 22 Feb 2012 22:20:41 -0800
>>> To: "ausnog at ausnog.net<mailto:ausnog at ausnog.net>"
>>> <ausnog at ausnog.net<mailto:ausnog at ausnog.net>>
>>> Subject: Re: [AusNOG] International link issue
>>>
>>> Yes, this reinforces the Rule of Trust. Don’t trust your BGP peers
>>> and ensure your filters are in place, configured correctly and
>>> working, you can’t transfer blame.____
>>>
>>> It can cost you big $$ and pain if you inadvertently turn yourself
>>> into a transit peer because your upstreams may prefer to send
>>> traffic where they can make $$ from.____
>>>
>>> __ __
>>>
>>> *From:*ausnog-bounces at lists.ausnog.net
>>> <mailto:ausnog-bounces at lists.ausnog.net>
>>> [mailto:ausnog-bounces at lists.ausnog.net] *On Behalf Of *Sean K. Finn
>>> *Sent:* Thursday, 23 February 2012 5:09 PM
>>> *To:* 'ausnog at ausnog.net<mailto:'ausnog at ausnog.net>'
>>> *Subject:* Re: [AusNOG] International link issue____
>>>
>>> __ __
>>>
>>> It’s easy to describe for all the media types watching..____
>>>
>>> (And I’m not sure why its not being put out there in Laymans terms).____
>>>
>>> __ __
>>>
>>> From the routes seen at various points, and reported on the WAIX
>>> mailing list earlier..____
>>>
>>> __ __
>>>
>>> __ __
>>>
>>> __ __
>>>
>>> Dodo told Telstra that Dodo was the rest of the Internet.____
>>>
>>> __ __
>>>
>>> Telstra Believed Dodo.____
>>>
>>> __ __
>>>
>>> Telstra entire system tried to use DODO as their ISP instead of
>>> everyone else Telstra is connected to.____
>>>
>>> __ __
>>>
>>> Needless to say this didn’t work, the pipes got Jammed.____
>>>
>>> __ __
>>>
>>> Telstra should have filtered the announcement from Dodo, butdidn’t.____
>>>
>>> __ __
>>>
>>> Filtering is in place as a form of control (which is used instead of
>>> trust).____
>>>
>>> __ __
>>>
>>> Filtering obviously wasn’t in place, or didn’t work, so anything
>>> that Dodo told Telstra about where to find the Internet, Telstra
>>> believed.____
>>>
>>> __ __
>>>
>>> This happens quite often, I’ve heard of this happening on peering
>>> exchanges within Australia, too. Just never at an organizational
>>> level as big as Telstra.____
>>>
>>> __ __
>>>
>>> Over and Out.____
>>>
>>> __ __
>>>
>>> __ __
>>>
>>> This message and its attachments may contain legally privileged or
>>> confidential information. It is for the intended addressee(s) only.____
>>>
>>> If you are not the intended recipient you must not disclose or use
>>> the information contained in it. If you have received this email in
>>> error please notify us immediately by return email and delete the
>>> document.____
>>>
>>> Any views expressed in this message are those of the individual
>>> sender, except where the sender specifies and with authority, states
>>> them to be the views of the Company.____
>>>
>>> Uecomm accepts no liability for any damage caused by this email or
>>> its attachments due to viruses, interference, interception,
>>> corruption or unauthorised access.____
>>>
>>> ------------------------------------------------------------------------
>>> This e-mail message has been scanned for Viruses and Content and
>>> cleared by *NetIQ MailMarshal *
>>> ------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> AusNOG mailing list
>>> AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
>>> http://lists.ausnog.net/mailman/listinfo/ausnog
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> AusNOG mailing list
>>> AusNOG at lists.ausnog.net
>>> http://lists.ausnog.net/mailman/listinfo/ausnog
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> http://lists.ausnog.net/mailman/listinfo/ausnog
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog
>
More information about the AusNOG
mailing list