[AusNOG] International link issue

Fri Feb 24 12:13:28 EST 2012

I've seen Mr BGP, Geoff Huston lurks on these forums from time to time (I
think he appeared a few day ago). Might be a good time to suggest his next
monthly column on http://www.potaroo.net/ for March to cover ISP BGP peering
design. Seems the list has some significant interest in how, where, why and
best practices. I believe Geoff has written a few books on ISP's over the
many years and could share some valuable insights (and maybe some war
stories too from his days at Telstra)

Food for thought.

From: ausnog-bounces at lists.ausnog.net
[mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Sean K. Finn
Sent: Friday, 24 February 2012 11:58 AM
To: 'Ausnog at ausnog.net'
Subject: Re: [AusNOG] International link issue

I'll be honest here and share an experience.

(All beit an embarrassing one, but I'm willing to admit it, I'm sure there
are many others on the list who have done it who may not admit it).

In my early days learning live BGP, (There is no testing with Internet BGP,
it's always live, all the time).

There's no putting your feet into the water to learn it, it's head first
sink or swim.

When I turned on our second multi homed connection, and started taking full
global routing tables, I had no AS path filtering in place, and no Prefix
filtering in place. I didn't even know what they were, really, back then.

I was advertising OPTUS's entire global routing table to AAPT, and AAPT's
entire global routing table to OPTUS.

I was advertising every piece of private address space known to man
(10.0.0.0/8, 192.168, 172, etc etc) into BOTH of them.

Did I know what was going on in the early days of BGP? Yes, I saw the
advertisments, but, thankfully, AAPT and OPTUS had seen clients like me
before (at that stage, without a clue). Did I know what effect it would
have? Heck no.

Did everything work? Yes, perfectly and absolutely, because there were
controls in place to prevent any issues.

Thankfully, these days, I know better.

ROUTE-MAP's and AS-PATH filters and prefix lists and policy based routing,
community based routing, come later, after you figure out how BGP works in
the first place.

Fast forward to now.

We now perform at least one or two new BGP based interconnections per week,
and unless *we* have our policies absolutely perfect, weird weird routing
can and does occur.

I expect my downstreams to have no idea what AS path filtering or prefix
filtering is about. I expect them to advertise 0.0.0.0/0 to me. (and some do
J )

Even then, even with appropriate route-filters and AS path filters in place,
strangeness can occur.

I've even seen a client's network, that we were YET to physically connect
to, but had setup AS path and Prefix list filtering internally and with our
upstreams, bring traffic in through Optus, through our network, and out
through AAPT to reach its final destination. This occurred because we're a
client of both Optus and AAPT and the route through us was seen as the
shortest path. (Amazing, I know).

This happened because we were learning their routes from AAPT, it was
passing through our filtering, and then being re-advertised through to
OPTUS.

There's no standard single way of setting up this filtering, either, every
network is different.

In some networks, filters are done manually by the network ops technicians.

Other networks, its policy based, determining where to advertise routes to
based on where the routes were learnt from.

In others, like the IX's, there's automated systems that generate policy and
route-maps and prefix-lists based on what you punch into them, and push the
configs out to the routers that matter at various intervals.

It's so damn easy to make a mistake with any of these systems when bringing
on a new peer (Transit  peer, upstream peer, peer peer, or client peer).

Maybe a good topic for this year's AusNOG, on different ways to do
AS-Path-Filtering and Prefix-List Filtering.

Does anybody else care to share their experiences?

From: Burt Mascareigne [mailto:burt at prioritycomputer.com.au] 
Sent: Friday, February 24, 2012 10:42 AM
To: James Spenceley; Sean K. Finn
Cc: 'ausnog at ausnog.net'
Subject: RE: [AusNOG] International link issue

/Agree

Now let's get the jurnos to understand that and to pass that onto the
public,

Ready, Set, Go!

From: ausnog-bounces at lists.ausnog.net
[mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of James Spenceley
Sent: Friday, 24 February 2012 11:39 AM
To: Sean K. Finn
Cc: 'ausnog at ausnog.net'
Subject: Re: [AusNOG] International link issue

Well said Sean.

If anyone thinks this is Dodo's fault, think again. Customers make
configuration errors like this regularly. 

I understand Telstra's position in some cases, to prefix filter us is
something like 2500 entires with lots of updates, that can be hard to
maintain, *at a prefix level*

Telstra not having a generic AS-PATH or even max-prefix put the fault
entirely with Telstra. 

On 24/02/2012, at 11:29 AM, Sean K. Finn wrote:

Telstra should have had an Inbound route filter in any case.

(Both an AS Path filter, and a prefix-list filter).

The amount of people I've seen TRYING to advertise 0.0.0.0/0 over an IX, or
even leaking private address space into carrier links, is common place.
Thankfully most carriers don't trust their clients advertisments by default.

PIPE IX have a fairly complex (but not complicated) portal to use to ensure
that your AS PATH LIST and PREFIX LISTS are added to their server manually
before adding them to their route filters.Even better, every day there's a
mailout with the filters that you can put on your OWN routers to match their
routers, just-in-case.

I've seen some providers be extremely strict on even the manual ranges that
get added to portals, checking each and every IP assignment to ensure that
I'm allowed to advertise the ranges, and AS's, in question. (And in some
cases making me jump through silly hurdles to PROVE it).

The very definition of UPSTREAM means it's their responsibility to not trust
the DOWNSTREAM.

Fair enough, Dodo had to leak the routes, and Telstra had to accept them,
there was a mutual mis-configuration, that part is self evident.

It's easy enough to make the mistake.

S

From: ausnog-bounces at lists.ausnog.net
[mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Aaron Swayn
Sent: Friday, February 24, 2012 10:20 AM
To: ausnog at ausnog.net
Subject: Re: [AusNOG] International link issue

I think the spin was from Dodo to blame the vendor, but me thinks the fault
went down something like this.

-          Dodo engineer troubleshooting BGP advertisement issue

-          Can't figure it out (can't see why, these things do work properly
and you can debug them easily enough without having to take risks)

-          Removed outbound route filter from BGP session to Telstra to see
if that fixes the problem

And we know the rest.

From: ausnog-bounces at lists.ausnog.net
[mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Evan Weston
Sent: Friday, 24 February 2012 10:55 AM
To: ausnog at ausnog.net
Subject: Re: [AusNOG] International link issue

Nice spin from Telstra. Blame the hardware vendor, blame the customer but
never admit that it was actually *our fault* for not filtering properly.

From: ausnog-bounces at lists.ausnog.net
[mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Will Tardy
Sent: Friday, 24 February 2012 10:30 AM
To: ausnog at ausnog.net
Subject: Re: [AusNOG] International link issue

Telstra claims they had an international link down:

http://www.zdnet.com.au/telstra-hit-by-nationwide-data-outage-339332310.htm

If that happened at the same time as DODO incorrectly sending Telstra the
full BGP table, could that explain why Telstra black-holed all-routes plus
pumped all of it's own traffic via dodo?

On 24 February 2012 10:02, Wade Millican <Wade.Millican at echoent.com.au>
wrote:

Hi All,

What I'm yet to understand about this outage is why DODO's AS_PATH was seen
as shorter than anything Telstra already had.

An earlier posted look at routes(below), thanks Gavin, shows all routes from
Telstra taking hops to DODO, then Optus or PIPE before moving to the
destination. Surely Telstra would have had better routes than pushing all
traffic 2 hops out of it's way.

AS_PATH does not explain how Telstra accepted these as the active routes.
Even if all routes were accepted, Telstra still has better routes.

Can anyone explain what BGP Metric was modified/used that pushed traffic
over longer AS_PATHs? 

*> 1.22.161.0/24    165.228.157.73         100     80      0 1221 38285 7474
7473 55410 45528 i
*> 1.22.162.0/24    165.228.157.73         100     80      0 1221 38285 7474
7473 55410 45528 i
*> 1.22.163.0/24    165.228.157.73         100     80      0 1221 38285 7474
7473 55410 45528 i
*> 1.22.167.0/24    165.228.157.73         100     80      0 1221 38285 7474
7473 6453 4755 45528 i
*> 1.22.168.0/24    165.228.157.73         100     80      0 1221 38285 7474
7473 6453 4755 45528 i
..
*  14.201.64.0/24   165.228.157.73         100     80      0 1221 38285
18398 7545 7545 i

Thanks,

Wade

-- 

Wade Millican 
Technical Consultant Team Lead
Hemisphere Infrastructure Support
Information Technology
Echo Entertainment Group Limited 

2 Edward St
Pyrmont NSW 2009 

T: +61 2 9657 7460 <tel:%2B61%202%209657%207460> 
M: +61 (0) 400 192 485 <tel:%2B61%20%280%29%20400%20192%20485> 
wade.millican at echoent.com.au
www.echoentertainment.com.au
<image001.png>

From: "Ramsay, Paul" <pramsay at uecomm.com.au>
Date: Wed, 22 Feb 2012 22:20:41 -0800
To: "ausnog at ausnog.net" <ausnog at ausnog.net>
Subject: Re: [AusNOG] International link issue

Yes, this reinforces the Rule of Trust. Don't trust your BGP peers and
ensure your filters are in place, configured correctly and working, you
can't transfer blame.

It can cost you big $$ and pain if you inadvertently turn yourself into a
transit peer because your upstreams may prefer to send traffic where they
can make $$ from.

From: ausnog-bounces at lists.ausnog.net
[mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Sean K. Finn
Sent: Thursday, 23 February 2012 5:09 PM
To: 'ausnog at ausnog.net'
Subject: Re: [AusNOG] International link issue

It's easy to describe for all the media types watching..

(And I'm not sure why its not being put out there in Laymans terms).

>From the routes seen at various points, and reported on the WAIX mailing
list earlier..

Dodo told Telstra that Dodo was the rest of the Internet.

Telstra Believed Dodo.

Telstra entire system tried to use DODO as their ISP instead of everyone
else Telstra is connected to.

Needless to say this didn't work, the pipes got Jammed.

Telstra should have filtered the announcement from Dodo, butdidn't.

Filtering is in place as a form of control (which is used instead of trust).

Filtering obviously wasn't in place, or didn't work, so anything that Dodo
told Telstra about where to find the Internet, Telstra believed.

This happens quite often, I've heard of this happening on peering exchanges
within Australia, too. Just never at an organizational level as big as
Telstra.

Over and Out.

This message and its attachments may contain legally privileged or
confidential information. It is for the intended addressee(s) only.

If you are not the intended recipient you must not disclose or use the
information contained in it. If you have received this email in error please
notify us immediately by return email and delete the document.

Any views expressed in this message are those of the individual sender,
except where the sender specifies and with authority, states them to be the
views of the Company.

Uecomm accepts no liability for any damage caused by this email or its
attachments due to viruses, interference, interception, corruption or
unauthorised access.

  _____  

This e-mail message has been scanned for Viruses and Content and cleared by
NetIQ MailMarshal

  _____  

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
http://lists.ausnog.net/mailman/listinfo/ausnog

  _____  

Stockland Notice: If this communication has been sent to you by mistake,
please delete and notify us. If it has been sent to you by mistake, legal
privilege is not waived or lost and you are not entitled to use it in any
way. Stockland and its subsidiaries reserve the right to monitor e-mail
communication through its networks.

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
http://lists.ausnog.net/mailman/listinfo/ausnog

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20120224/27531ac9/attachment.html>