[AusNOG] International link issue

Mark Doorey SBS.User at netmark.net.au
Fri Feb 24 13:08:16 EST 2012


I was using IPv6 and IPv4 at the time all the IPv6 traffic seemed fine.
Whereas the IPv4 traffic had a meltdown.

Netmark Computer Solutions
If you go online unprotected you are a bloody idiot


Mark Dooréy



On 24/02/2012 12:26 PM, Matthew Moyle-Croft wrote:
> Suspect what happened was:
>
> 1) Dodo starts advertising full table to Telstra
> 2) Telstra don't filter, don't max prefix Dodo so carried this into their BGP Table.
> 3) Telstra started advertising Dodo's advertisements out their transit/peering ports as Dodo's AS is customer one and they don't filter that properly either.
> 4) "International" links go down as Telstra's transit provider and peers (eg. Reach etc) start hitting their max-prefix filters and dropping BGP hence why AS1221 went away.
> 5) Hilarity rains down upon us all.
>
> One thing I didn't see (was behind someone using AS1221 at the time :-( ) was what happened to Telstra's IPv6 table?  They have IPv6 and wondering if AS1221's IPv6 routes disappeared at the same time?
>
> MMC
>
> On 24/02/2012, at 10:28 AM, Andree Toonk wrote:
>
>> That would explain why all of Telstra and many other prefixes (~1400)
>> disappeared from the BGP tables:
>>
>> http://www.bgpmon.net/telstra-feb23-2012.png
>> https://twitter.com/#!/bgpmon/status/172608854855647233/photo/1
>>
>> Though, surely Telstra has more than one international link...
>>
>> Andree
>>
>>
>> .-- My secret spy satellite informs me that at 12-02-23 3:30 PM  Will
>> Tardy wrote:
>>> Telstra claims they had an international link down:
>>>
>>> http://www.zdnet.com.au/telstra-hit-by-nationwide-data-outage-339332310.htm
>>>
>>> If that happened at the same time as DODO incorrectly sending Telstra
>>> the full BGP table, could that explain why Telstra black-holed
>>> all-routes plus pumped all of it's own traffic via dodo?
>>>
>>> On 24 February 2012 10:02, Wade Millican<Wade.Millican at echoent.com.au
>>> <mailto:Wade.Millican at echoent.com.au>>  wrote:
>>>
>>>     Hi All,
>>>
>>>     What I'm yet to understand about this outage is why DODO's AS_PATH
>>>     was seen as shorter than anything Telstra already had.
>>>
>>>     An earlier posted look at routes(below), thanks Gavin, shows all
>>>     routes from Telstra taking hops to DODO, then Optus or PIPE before
>>>     moving to the destination. Surely Telstra would have had better
>>>     routes than pushing all traffic 2 hops out of it's way.
>>>
>>>     AS_PATH does not explain how Telstra accepted these as the active
>>>     routes. Even if all routes were accepted, Telstra still has better
>>>     routes.
>>>
>>>     Can anyone explain what BGP Metric was modified/used that pushed
>>>     traffic over longer AS_PATHs?
>>>
>>>     *>  1.22.161.0/24<http://1.22.161.0/24>     165.228.157.73         100     80      0 1221 38285 7474 7473 55410 45528 i
>>>     *>  1.22.162.0/24<http://1.22.162.0/24>     165.228.157.73         100     80      0 1221 38285 7474 7473 55410 45528 i
>>>     *>  1.22.163.0/24<http://1.22.163.0/24>     165.228.157.73         100     80      0 1221 38285 7474 7473 55410 45528 i
>>>     *>  1.22.167.0/24<http://1.22.167.0/24>     165.228.157.73         100     80      0 1221 38285 7474 7473 6453 4755 45528 i
>>>     *>  1.22.168.0/24<http://1.22.168.0/24>     165.228.157.73         100     80      0 1221 38285 7474 7473 6453 4755 45528 i
>>>     ..
>>>     *  14.201.64.0/24<http://14.201.64.0/24>    165.228.157.73         100     80      0 1221 38285 18398 7545 7545 i
>>>
>>>
>>>     Thanks,
>>>
>>>     Wade
>>>     --
>>>     Wade Millican
>>>     Technical Consultant Team Lead
>>>     Hemisphere Infrastructure Support
>>>     Information Technology
>>>     *Echo Entertainment Group Limited*
>>>
>>>     2 Edward St
>>>     Pyrmont NSW 2009
>>>
>>>     T: +61 2 9657 7460<tel:%2B61%202%209657%207460>
>>>     M: +61 (0) 400 192 485<tel:%2B61%20%280%29%20400%20192%20485>
>>>     _wade.millican at echoent.com.au
>>>     _www.echoentertainment.com.au<http://www.echoentertainment.com.au>
>>>     From: "Ramsay, Paul"<pramsay at uecomm.com.au
>>>     <mailto:pramsay at uecomm.com.au>>
>>>     Date: Wed, 22 Feb 2012 22:20:41 -0800
>>>     To: "ausnog at ausnog.net<mailto:ausnog at ausnog.net>"
>>>     <ausnog at ausnog.net<mailto:ausnog at ausnog.net>>
>>>     Subject: Re: [AusNOG] International link issue
>>>
>>>     Yes, this reinforces the Rule of Trust. Don’t trust your BGP peers
>>>     and ensure your filters are in place, configured correctly and
>>>     working, you can’t transfer blame.____
>>>
>>>     It can cost you big $$ and pain if you inadvertently turn yourself
>>>     into a transit peer because your upstreams may prefer to send
>>>     traffic where they can make $$ from.____
>>>
>>>     __ __
>>>
>>>     *From:*ausnog-bounces at lists.ausnog.net
>>>     <mailto:ausnog-bounces at lists.ausnog.net>
>>>     [mailto:ausnog-bounces at lists.ausnog.net] *On Behalf Of *Sean K. Finn
>>>     *Sent:* Thursday, 23 February 2012 5:09 PM
>>>     *To:* 'ausnog at ausnog.net<mailto:'ausnog at ausnog.net>'
>>>     *Subject:* Re: [AusNOG] International link issue____
>>>
>>>     __ __
>>>
>>>     It’s easy to describe for all the media types watching..____
>>>
>>>     (And I’m not sure why its not being put out there in Laymans terms).____
>>>
>>>     __ __
>>>
>>>     From the routes seen at various points, and reported on the WAIX
>>>     mailing list earlier..____
>>>
>>>     __ __
>>>
>>>     __ __
>>>
>>>     __ __
>>>
>>>     Dodo told Telstra that Dodo was the rest of the Internet.____
>>>
>>>     __ __
>>>
>>>     Telstra Believed Dodo.____
>>>
>>>     __ __
>>>
>>>     Telstra entire system tried to use DODO as their ISP instead of
>>>     everyone else Telstra is connected to.____
>>>
>>>     __ __
>>>
>>>     Needless to say this didn’t work, the pipes got Jammed.____
>>>
>>>     __ __
>>>
>>>     Telstra should have filtered the announcement from Dodo, butdidn’t.____
>>>
>>>     __ __
>>>
>>>     Filtering is in place as a form of control (which is used instead of
>>>     trust).____
>>>
>>>     __ __
>>>
>>>     Filtering obviously wasn’t in place, or didn’t work, so anything
>>>     that Dodo told Telstra about where to find the Internet, Telstra
>>>     believed.____
>>>
>>>     __ __
>>>
>>>     This happens quite often, I’ve heard of this happening on peering
>>>     exchanges within Australia, too. Just never at an organizational
>>>     level as big as Telstra.____
>>>
>>>     __ __
>>>
>>>     Over and Out.____
>>>
>>>     __ __
>>>
>>>     __ __
>>>
>>>     This message and its attachments may contain legally privileged or
>>>     confidential information. It is for the intended addressee(s) only.____
>>>
>>>     If you are not the intended recipient you must not disclose or use
>>>     the information contained in it. If you have received this email in
>>>     error please notify us immediately by return email and delete the
>>>     document.____
>>>
>>>     Any views expressed in this message are those of the individual
>>>     sender, except where the sender specifies and with authority, states
>>>     them to be the views of the Company.____
>>>
>>>     Uecomm accepts no liability for any damage caused by this email or
>>>     its attachments due to viruses, interference, interception,
>>>     corruption or unauthorised access.____
>>>
>>>     ------------------------------------------------------------------------
>>>     This e-mail message has been scanned for Viruses and Content and
>>>     cleared by *NetIQ MailMarshal *
>>>     ------------------------------------------------------------------------
>>>
>>>     _______________________________________________
>>>     AusNOG mailing list
>>>     AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
>>>     http://lists.ausnog.net/mailman/listinfo/ausnog
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> AusNOG mailing list
>>> AusNOG at lists.ausnog.net
>>> http://lists.ausnog.net/mailman/listinfo/ausnog
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> http://lists.ausnog.net/mailman/listinfo/ausnog
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog
>



More information about the AusNOG mailing list