[AusNOG] International link issue

Fri Feb 24 11:29:58 EST 2012

Telstra should have had an Inbound route filter in any case.
(Both an AS Path filter, and a prefix-list filter).

The amount of people I've seen TRYING to advertise 0.0.0.0/0 over an IX, or even leaking private address space into carrier links, is common place. Thankfully most carriers don't trust their clients advertisments by default.

PIPE IX have a fairly complex (but not complicated) portal to use to ensure that your AS PATH LIST and PREFIX LISTS are added to their server manually before adding them to their route filters.Even better, every day there's a mailout with the filters that you can put on your OWN routers to match their routers, just-in-case.

I've seen some providers be extremely strict on even the manual ranges that get added to portals, checking each and every IP assignment to ensure that I'm allowed to advertise the ranges, and AS's, in question. (And in some cases making me jump through silly hurdles to PROVE it).

The very definition of UPSTREAM means it's their responsibility to not trust the DOWNSTREAM.

Fair enough, Dodo had to leak the routes, and Telstra had to accept them, there was a mutual mis-configuration, that part is self evident.

It's easy enough to make the mistake.

S
From: ausnog-bounces at lists.ausnog.net [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Aaron Swayn
Sent: Friday, February 24, 2012 10:20 AM
To: ausnog at ausnog.net
Subject: Re: [AusNOG] International link issue

I think the spin was from Dodo to blame the vendor, but me thinks the fault went down something like this...

-          Dodo engineer troubleshooting BGP advertisement issue

-          Can't figure it out (can't see why, these things do work properly and you can debug them easily enough without having to take risks)

-          Removed outbound route filter from BGP session to Telstra to see if that fixes the problem

And we know the rest...

From: ausnog-bounces at lists.ausnog.net [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Evan Weston
Sent: Friday, 24 February 2012 10:55 AM
To: ausnog at ausnog.net
Subject: Re: [AusNOG] International link issue

Nice spin from Telstra. Blame the hardware vendor, blame the customer but never admit that it was actually *our fault* for not filtering properly.

From: ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net> [mailto:ausnog-bounces at lists.ausnog.net]<mailto:[mailto:ausnog-bounces at lists.ausnog.net]> On Behalf Of Will Tardy
Sent: Friday, 24 February 2012 10:30 AM
To: ausnog at ausnog.net<mailto:ausnog at ausnog.net>
Subject: Re: [AusNOG] International link issue

Telstra claims they had an international link down:

http://www.zdnet.com.au/telstra-hit-by-nationwide-data-outage-339332310.htm

If that happened at the same time as DODO incorrectly sending Telstra the full BGP table, could that explain why Telstra black-holed all-routes plus pumped all of it's own traffic via dodo?
On 24 February 2012 10:02, Wade Millican <Wade.Millican at echoent.com.au<mailto:Wade.Millican at echoent.com.au>> wrote:
Hi All,

What I'm yet to understand about this outage is why DODO's AS_PATH was seen as shorter than anything Telstra already had.

An earlier posted look at routes(below), thanks Gavin, shows all routes from Telstra taking hops to DODO, then Optus or PIPE before moving to the destination. Surely Telstra would have had better routes than pushing all traffic 2 hops out of it's way.

AS_PATH does not explain how Telstra accepted these as the active routes. Even if all routes were accepted, Telstra still has better routes.

Can anyone explain what BGP Metric was modified/used that pushed traffic over longer AS_PATHs?

*> 1.22.161.0/24<http://1.22.161.0/24>    165.228.157.73         100     80      0 1221 38285 7474 7473 55410 45528 i

*> 1.22.162.0/24<http://1.22.162.0/24>    165.228.157.73         100     80      0 1221 38285 7474 7473 55410 45528 i

*> 1.22.163.0/24<http://1.22.163.0/24>    165.228.157.73         100     80      0 1221 38285 7474 7473 55410 45528 i

*> 1.22.167.0/24<http://1.22.167.0/24>    165.228.157.73         100     80      0 1221 38285 7474 7473 6453 4755 45528 i

*> 1.22.168.0/24<http://1.22.168.0/24>    165.228.157.73         100     80      0 1221 38285 7474 7473 6453 4755 45528 i

..

*  14.201.64.0/24<http://14.201.64.0/24>   165.228.157.73         100     80      0 1221 38285 18398 7545 7545 i

Thanks,

Wade
--
Wade Millican
Technical Consultant Team Lead
Hemisphere Infrastructure Support
Information Technology
Echo Entertainment Group Limited

2 Edward St
Pyrmont NSW 2009

T: +61 2 9657 7460<tel:%2B61%202%209657%207460>
M: +61 (0) 400 192 485<tel:%2B61%20%280%29%20400%20192%20485>
wade.millican at echoent.com.au<mailto:wade.millican at echoent.com.au>
www.echoentertainment.com.au<http://www.echoentertainment.com.au>
[cid:image001.png at 01CCF2DF.74951D60]
From: "Ramsay, Paul" <pramsay at uecomm.com.au<mailto:pramsay at uecomm.com.au>>
Date: Wed, 22 Feb 2012 22:20:41 -0800
To: "ausnog at ausnog.net<mailto:ausnog at ausnog.net>" <ausnog at ausnog.net<mailto:ausnog at ausnog.net>>
Subject: Re: [AusNOG] International link issue

Yes, this reinforces the Rule of Trust. Don't trust your BGP peers and ensure your filters are in place, configured correctly and working, you can't transfer blame.
It can cost you big $$ and pain if you inadvertently turn yourself into a transit peer because your upstreams may prefer to send traffic where they can make $$ from.

From: ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net> [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Sean K. Finn
Sent: Thursday, 23 February 2012 5:09 PM
To: 'ausnog at ausnog.net<mailto:'ausnog at ausnog.net>'
Subject: Re: [AusNOG] International link issue

It's easy to describe for all the media types watching..
(And I'm not sure why its not being put out there in Laymans terms).

>From the routes seen at various points, and reported on the WAIX mailing list earlier..

Dodo told Telstra that Dodo was the rest of the Internet.

Telstra Believed Dodo.

Telstra entire system tried to use DODO as their ISP instead of everyone else Telstra is connected to.

Needless to say this didn't work, the pipes got Jammed.

Telstra should have filtered the announcement from Dodo, butdidn't.

Filtering is in place as a form of control (which is used instead of trust).

Filtering obviously wasn't in place, or didn't work, so anything that Dodo told Telstra about where to find the Internet, Telstra believed.

This happens quite often, I've heard of this happening on peering exchanges within Australia, too. Just never at an organizational level as big as Telstra.

Over and Out.

This message and its attachments may contain legally privileged or confidential information. It is for the intended addressee(s) only.
If you are not the intended recipient you must not disclose or use the information contained in it. If you have received this email in error please notify us immediately by return email and delete the document.
Any views expressed in this message are those of the individual sender, except where the sender specifies and with authority, states them to be the views of the Company.
Uecomm accepts no liability for any damage caused by this email or its attachments due to viruses, interference, interception, corruption or unauthorised access.
________________________________
This e-mail message has been scanned for Viruses and Content and cleared by NetIQ MailMarshal
________________________________

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
http://lists.ausnog.net/mailman/listinfo/ausnog

________________________________
Stockland Notice: If this communication has been sent to you by mistake, please delete and notify us. If it has been sent to you by mistake, legal privilege is not waived or lost and you are not entitled to use it in any way. Stockland and its subsidiaries reserve the right to monitor e-mail communication through its networks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20120224/c62c66d2/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 13740 bytes
Desc: image001.png
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20120224/c62c66d2/attachment.png>