[AusNOG] International link issue

Fri Feb 24 12:30:13 EST 2012

Or instead read -

"BGP", by Iljitsch Van Beijnum.
"Internet Routing Architectures, 2nd Edition", by Sam Halabi.
"BGP Design and Implementation", by Randy Zhang and Micah Bartell.

and peruse other BGP related topics and resources at

http://www.bgp4.as/

Another source of BGP design advice are presentation slides from various *nog meetings over the years. For example,

"Tutorial: BGP Communities for Service Providers"
Richard A. Steenbergen, nLayer Communications; Tom Scholl, AT&T Labs
http://www.nanog.org/meetings/nanog40/abstracts.php?pt=MjI4Jm5hbm9nNDA=&nm=nanog40

________________________________

From: ausnog-bounces at lists.ausnog.net [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Aaron Swayn
Sent: Friday, 24 February 2012 12:13 PM
To: Ausnog at ausnog.net
Subject: Re: [AusNOG] International link issue

I've seen Mr BGP, Geoff Huston lurks on these forums from time to time (I think he appeared a few day ago). Might be a good time to suggest his next monthly column on http://www.potaroo.net/ for March to cover ISP BGP peering design. Seems the list has some significant interest in how, where, why and best practices. I believe Geoff has written a few books on ISP's over the many years and could share some valuable insights (and maybe some war stories too from his days at Telstra)

Food for thought.

From: ausnog-bounces at lists.ausnog.net [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Sean K. Finn
Sent: Friday, 24 February 2012 11:58 AM
To: 'Ausnog at ausnog.net'
Subject: Re: [AusNOG] International link issue

I'll be honest here and share an experience.

(All beit an embarrassing one, but I'm willing to admit it, I'm sure there are many others on the list who have done it who may not admit it).

In my early days learning live BGP, (There is no testing with Internet BGP, it's always live, all the time).

There's no putting your feet into the water to learn it, it's head first sink or swim.

When I turned on our second multi homed connection, and started taking full global routing tables, I had no AS path filtering in place, and no Prefix filtering in place. I didn't even know what they were, really, back then.

I was advertising OPTUS's entire global routing table to AAPT, and AAPT's entire global routing table to OPTUS.

I was advertising every piece of private address space known to man (10.0.0.0/8, 192.168, 172, etc etc) into BOTH of them.

Did I know what was going on in the early days of BGP? Yes, I saw the advertisments, but, thankfully, AAPT and OPTUS had seen clients like me before (at that stage, without a clue). Did I know what effect it would have? Heck no.

Did everything work? Yes, perfectly and absolutely, because there were controls in place to prevent any issues.

Thankfully, these days, I know better.

ROUTE-MAP's and AS-PATH filters and prefix lists and policy based routing, community based routing, come later, after you figure out how BGP works in the first place.

Fast forward to now.

We now perform at least one or two new BGP based interconnections per week, and unless *we* have our policies absolutely perfect, weird weird routing can and does occur.

I expect my downstreams to have no idea what AS path filtering or prefix filtering is about. I expect them to advertise 0.0.0.0/0 to me. (and some do J )

Even then, even with appropriate route-filters and AS path filters in place, strangeness can occur.

I've even seen a client's network, that we were YET to physically connect to, but had setup AS path and Prefix list filtering internally and with our upstreams, bring traffic in through Optus, through our network, and out through AAPT to reach its final destination. This occurred because we're a client of both Optus and AAPT and the route through us was seen as the shortest path. (Amazing, I know).

This happened because we were learning their routes from AAPT, it was passing through our filtering, and then being re-advertised through to OPTUS.

There's no standard single way of setting up this filtering, either, every network is different.

In some networks, filters are done manually by the network ops technicians.

Other networks, its policy based, determining where to advertise routes to based on where the routes were learnt from.

In others, like the IX's, there's automated systems that generate policy and route-maps and prefix-lists based on what you punch into them, and push the configs out to the routers that matter at various intervals.

It's so damn easy to make a mistake with any of these systems when bringing on a new peer (Transit  peer, upstream peer, peer peer, or client peer).

Maybe a good topic for this year's AusNOG, on different ways to do AS-Path-Filtering and Prefix-List Filtering.

Does anybody else care to share their experiences?

From: Burt Mascareigne [mailto:burt at prioritycomputer.com.au]
Sent: Friday, February 24, 2012 10:42 AM
To: James Spenceley; Sean K. Finn
Cc: 'ausnog at ausnog.net'
Subject: RE: [AusNOG] International link issue

/Agree

Now let's get the jurnos to understand that and to pass that onto the public,

Ready, Set, Go!

From: ausnog-bounces at lists.ausnog.net [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of James Spenceley
Sent: Friday, 24 February 2012 11:39 AM
To: Sean K. Finn
Cc: 'ausnog at ausnog.net'
Subject: Re: [AusNOG] International link issue

Well said Sean.

If anyone thinks this is Dodo's fault, think again. Customers make configuration errors like this regularly.

I understand Telstra's position in some cases, to prefix filter us is something like 2500 entires with lots of updates, that can be hard to maintain, *at a prefix level*

Telstra not having a generic AS-PATH or even max-prefix put the fault entirely with Telstra.

On 24/02/2012, at 11:29 AM, Sean K. Finn wrote:

Telstra should have had an Inbound route filter in any case.

(Both an AS Path filter, and a prefix-list filter).

The amount of people I've seen TRYING to advertise 0.0.0.0/0 over an IX, or even leaking private address space into carrier links, is common place. Thankfully most carriers don't trust their clients advertisments by default.

PIPE IX have a fairly complex (but not complicated) portal to use to ensure that your AS PATH LIST and PREFIX LISTS are added to their server manually before adding them to their route filters.Even better, every day there's a mailout with the filters that you can put on your OWN routers to match their routers, just-in-case.

I've seen some providers be extremely strict on even the manual ranges that get added to portals, checking each and every IP assignment to ensure that I'm allowed to advertise the ranges, and AS's, in question. (And in some cases making me jump through silly hurdles to PROVE it).

The very definition of UPSTREAM means it's their responsibility to not trust the DOWNSTREAM.

Fair enough, Dodo had to leak the routes, and Telstra had to accept them, there was a mutual mis-configuration, that part is self evident.

It's easy enough to make the mistake.

S

From: ausnog-bounces at lists.ausnog.net [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Aaron Swayn
Sent: Friday, February 24, 2012 10:20 AM
To: ausnog at ausnog.net
Subject: Re: [AusNOG] International link issue

I think the spin was from Dodo to blame the vendor, but me thinks the fault went down something like this...

-          Dodo engineer troubleshooting BGP advertisement issue

-          Can't figure it out (can't see why, these things do work properly and you can debug them easily enough without having to take risks)

-          Removed outbound route filter from BGP session to Telstra to see if that fixes the problem

And we know the rest...

From: ausnog-bounces at lists.ausnog.net [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Evan Weston
Sent: Friday, 24 February 2012 10:55 AM
To: ausnog at ausnog.net
Subject: Re: [AusNOG] International link issue

Nice spin from Telstra. Blame the hardware vendor, blame the customer but never admit that it was actually *our fault* for not filtering properly.

From: ausnog-bounces at lists.ausnog.net [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Will Tardy
Sent: Friday, 24 February 2012 10:30 AM
To: ausnog at ausnog.net
Subject: Re: [AusNOG] International link issue

Telstra claims they had an international link down:

http://www.zdnet.com.au/telstra-hit-by-nationwide-data-outage-339332310.htm

If that happened at the same time as DODO incorrectly sending Telstra the full BGP table, could that explain why Telstra black-holed all-routes plus pumped all of it's own traffic via dodo?

On 24 February 2012 10:02, Wade Millican <Wade.Millican at echoent.com.au> wrote:

Hi All,

What I'm yet to understand about this outage is why DODO's AS_PATH was seen as shorter than anything Telstra already had.

An earlier posted look at routes(below), thanks Gavin, shows all routes from Telstra taking hops to DODO, then Optus or PIPE before moving to the destination. Surely Telstra would have had better routes than pushing all traffic 2 hops out of it's way.

AS_PATH does not explain how Telstra accepted these as the active routes. Even if all routes were accepted, Telstra still has better routes.

Can anyone explain what BGP Metric was modified/used that pushed traffic over longer AS_PATHs?

*> 1.22.161.0/24    165.228.157.73         100     80      0 1221 38285 7474 7473 55410 45528 i
*> 1.22.162.0/24    165.228.157.73         100     80      0 1221 38285 7474 7473 55410 45528 i
*> 1.22.163.0/24    165.228.157.73         100     80      0 1221 38285 7474 7473 55410 45528 i
*> 1.22.167.0/24    165.228.157.73         100     80      0 1221 38285 7474 7473 6453 4755 45528 i
*> 1.22.168.0/24    165.228.157.73         100     80      0 1221 38285 7474 7473 6453 4755 45528 i
..
*  14.201.64.0/24   165.228.157.73         100     80      0 1221 38285 18398 7545 7545 i

Thanks,

Wade

--

Wade Millican
Technical Consultant Team Lead
Hemisphere Infrastructure Support
Information Technology
Echo Entertainment Group Limited

2 Edward St
Pyrmont NSW 2009

T: +61 2 9657 7460 <tel:%2B61%202%209657%207460>
M: +61 (0) 400 192 485 <tel:%2B61%20%280%29%20400%20192%20485>
wade.millican at echoent.com.au
www.echoentertainment.com.au
<image001.png>

From: "Ramsay, Paul" <pramsay at uecomm.com.au>
Date: Wed, 22 Feb 2012 22:20:41 -0800
To: "ausnog at ausnog.net" <ausnog at ausnog.net>
Subject: Re: [AusNOG] International link issue

Yes, this reinforces the Rule of Trust. Don't trust your BGP peers and ensure your filters are in place, configured correctly and working, you can't transfer blame.

It can cost you big $$ and pain if you inadvertently turn yourself into a transit peer because your upstreams may prefer to send traffic where they can make $$ from.

From: ausnog-bounces at lists.ausnog.net [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Sean K. Finn
Sent: Thursday, 23 February 2012 5:09 PM
To: 'ausnog at ausnog.net'
Subject: Re: [AusNOG] International link issue

It's easy to describe for all the media types watching..

(And I'm not sure why its not being put out there in Laymans terms).

>From the routes seen at various points, and reported on the WAIX mailing list earlier..

Dodo told Telstra that Dodo was the rest of the Internet.

Telstra Believed Dodo.

Telstra entire system tried to use DODO as their ISP instead of everyone else Telstra is connected to.

Needless to say this didn't work, the pipes got Jammed.

Telstra should have filtered the announcement from Dodo, butdidn't.

Filtering is in place as a form of control (which is used instead of trust).

Filtering obviously wasn't in place, or didn't work, so anything that Dodo told Telstra about where to find the Internet, Telstra believed.

This happens quite often, I've heard of this happening on peering exchanges within Australia, too. Just never at an organizational level as big as Telstra.

Over and Out.

This message and its attachments may contain legally privileged or confidential information. It is for the intended addressee(s) only.

If you are not the intended recipient you must not disclose or use the information contained in it. If you have received this email in error please notify us immediately by return email and delete the document.

Any views expressed in this message are those of the individual sender, except where the sender specifies and with authority, states them to be the views of the Company.

Uecomm accepts no liability for any damage caused by this email or its attachments due to viruses, interference, interception, corruption or unauthorised access.

________________________________

This e-mail message has been scanned for Viruses and Content and cleared by NetIQ MailMarshal

________________________________

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
http://lists.ausnog.net/mailman/listinfo/ausnog

________________________________

Stockland Notice: If this communication has been sent to you by mistake, please delete and notify us. If it has been sent to you by mistake, legal privilege is not waived or lost and you are not entitled to use it in any way. Stockland and its subsidiaries reserve the right to monitor e-mail communication through its networks.

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
http://lists.ausnog.net/mailman/listinfo/ausnog

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________