[AusNOG] International link issue

Fri Feb 24 12:27:47 EST 2012

Good one Sean!

A long, long time ago in a land far, far away In another life &
different place and working as a BGP greenie I did a typo in the
customer's prefix list name so it did not match the customer BGP
configuration calling for that prefix list. The customer had not put any
filtering on at his end and was advertising the full internet routing
table to me. Result = no filters and full internet routing table
received from customer!

I found out some interesting things about BGP reconvergence that day.
BGP doesn't worry about bandwidth, but is full of bells and whistles
with rich metrics, the higher local pref of the customer took over,
traffic reconverged saturating his link and caused severe packet loss to
everyone else. A nice war story that I still use while facilitating
training today!

A BGP mentor guru once told me...there are three rules about BGP
configuration and they are "know what you are doing" J

Tryon Edwards - Some of the best lessons we ever learn we learn from our
mistakes and failures. The error of the past is the wisdom of the
future.

From: ausnog-bounces at lists.ausnog.net
[mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Sean K. Finn
Sent: Friday, 24 February 2012 11:58 AM
To: 'Ausnog at ausnog.net'
Subject: Re: [AusNOG] International link issue

I'll be honest here and share an experience.

(All beit an embarrassing one, but I'm willing to admit it, I'm sure
there are many others on the list who have done it who may not admit
it).

In my early days learning live BGP, (There is no testing with Internet
BGP, it's always live, all the time).

There's no putting your feet into the water to learn it, it's head first
sink or swim.

When I turned on our second multi homed connection, and started taking
full global routing tables, I had no AS path filtering in place, and no
Prefix filtering in place. I didn't even know what they were, really,
back then.

I was advertising OPTUS's entire global routing table to AAPT, and
AAPT's entire global routing table to OPTUS.

I was advertising every piece of private address space known to man
(10.0.0.0/8, 192.168, 172, etc etc) into BOTH of them.

Did I know what was going on in the early days of BGP? Yes, I saw the
advertisments, but, thankfully, AAPT and OPTUS had seen clients like me
before (at that stage, without a clue). Did I know what effect it would
have? Heck no.

Did everything work? Yes, perfectly and absolutely, because there were
controls in place to prevent any issues.

Thankfully, these days, I know better.

ROUTE-MAP's and AS-PATH filters and prefix lists and policy based
routing, community based routing, come later, after you figure out how
BGP works in the first place.

Fast forward to now.

We now perform at least one or two new BGP based interconnections per
week, and unless *we* have our policies absolutely perfect, weird weird
routing can and does occur.

I expect my downstreams to have no idea what AS path filtering or prefix
filtering is about. I expect them to advertise 0.0.0.0/0 to me. (and
some do J )

Even then, even with appropriate route-filters and AS path filters in
place, strangeness can occur.

I've even seen a client's network, that we were YET to physically
connect to, but had setup AS path and Prefix list filtering internally
and with our upstreams, bring traffic in through Optus, through our
network, and out through AAPT to reach its final destination. This
occurred because we're a client of both Optus and AAPT and the route
through us was seen as the shortest path. (Amazing, I know).

This happened because we were learning their routes from AAPT, it was
passing through our filtering, and then being re-advertised through to
OPTUS.

There's no standard single way of setting up this filtering, either,
every network is different.

In some networks, filters are done manually by the network ops
technicians.

Other networks, its policy based, determining where to advertise routes
to based on where the routes were learnt from.

In others, like the IX's, there's automated systems that generate policy
and route-maps and prefix-lists based on what you punch into them, and
push the configs out to the routers that matter at various intervals.

It's so damn easy to make a mistake with any of these systems when
bringing on a new peer (Transit  peer, upstream peer, peer peer, or
client peer).

Maybe a good topic for this year's AusNOG, on different ways to do
AS-Path-Filtering and Prefix-List Filtering.

Does anybody else care to share their experiences?

From: Burt Mascareigne [mailto:burt at prioritycomputer.com.au] 
Sent: Friday, February 24, 2012 10:42 AM
To: James Spenceley; Sean K. Finn
Cc: 'ausnog at ausnog.net'
Subject: RE: [AusNOG] International link issue

/Agree

Now let's get the jurnos to understand that and to pass that onto the
public,

Ready, Set, Go!

From: ausnog-bounces at lists.ausnog.net
[mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of James Spenceley
Sent: Friday, 24 February 2012 11:39 AM
To: Sean K. Finn
Cc: 'ausnog at ausnog.net'
Subject: Re: [AusNOG] International link issue

Well said Sean.

If anyone thinks this is Dodo's fault, think again. Customers make
configuration errors like this regularly. 

I understand Telstra's position in some cases, to prefix filter us is
something like 2500 entires with lots of updates, that can be hard to
maintain, *at a prefix level*

Telstra not having a generic AS-PATH or even max-prefix put the fault
entirely with Telstra. 

On 24/02/2012, at 11:29 AM, Sean K. Finn wrote:

Telstra should have had an Inbound route filter in any case.

(Both an AS Path filter, and a prefix-list filter).

The amount of people I've seen TRYING to advertise 0.0.0.0/0 over an IX,
or even leaking private address space into carrier links, is common
place. Thankfully most carriers don't trust their clients advertisments
by default.

PIPE IX have a fairly complex (but not complicated) portal to use to
ensure that your AS PATH LIST and PREFIX LISTS are added to their server
manually before adding them to their route filters.Even better, every
day there's a mailout with the filters that you can put on your OWN
routers to match their routers, just-in-case.

I've seen some providers be extremely strict on even the manual ranges
that get added to portals, checking each and every IP assignment to
ensure that I'm allowed to advertise the ranges, and AS's, in question.
(And in some cases making me jump through silly hurdles to PROVE it).

The very definition of UPSTREAM means it's their responsibility to not
trust the DOWNSTREAM.

Fair enough, Dodo had to leak the routes, and Telstra had to accept
them, there was a mutual mis-configuration, that part is self evident.

It's easy enough to make the mistake.

S

From: ausnog-bounces at lists.ausnog.net
[mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Aaron Swayn
Sent: Friday, February 24, 2012 10:20 AM
To: ausnog at ausnog.net
Subject: Re: [AusNOG] International link issue

I think the spin was from Dodo to blame the vendor, but me thinks the
fault went down something like this...

-          Dodo engineer troubleshooting BGP advertisement issue

-          Can't figure it out (can't see why, these things do work
properly and you can debug them easily enough without having to take
risks)

-          Removed outbound route filter from BGP session to Telstra to
see if that fixes the problem

And we know the rest...

From: ausnog-bounces at lists.ausnog.net
[mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Evan Weston
Sent: Friday, 24 February 2012 10:55 AM
To: ausnog at ausnog.net
Subject: Re: [AusNOG] International link issue

Nice spin from Telstra. Blame the hardware vendor, blame the customer
but never admit that it was actually *our fault* for not filtering
properly.

From: ausnog-bounces at lists.ausnog.net
[mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Will Tardy
Sent: Friday, 24 February 2012 10:30 AM
To: ausnog at ausnog.net
Subject: Re: [AusNOG] International link issue

Telstra claims they had an international link down:

http://www.zdnet.com.au/telstra-hit-by-nationwide-data-outage-339332310.
htm

If that happened at the same time as DODO incorrectly sending Telstra
the full BGP table, could that explain why Telstra black-holed
all-routes plus pumped all of it's own traffic via dodo?

On 24 February 2012 10:02, Wade Millican <Wade.Millican at echoent.com.au>
wrote:

Hi All,

What I'm yet to understand about this outage is why DODO's AS_PATH was
seen as shorter than anything Telstra already had.

An earlier posted look at routes(below), thanks Gavin, shows all routes
from Telstra taking hops to DODO, then Optus or PIPE before moving to
the destination. Surely Telstra would have had better routes than
pushing all traffic 2 hops out of it's way.

AS_PATH does not explain how Telstra accepted these as the active
routes. Even if all routes were accepted, Telstra still has better
routes.

Can anyone explain what BGP Metric was modified/used that pushed traffic
over longer AS_PATHs? 

*> 1.22.161.0/24    165.228.157.73         100     80      0 1221 38285
7474 7473 55410 45528 i
*> 1.22.162.0/24    165.228.157.73         100     80      0 1221 38285
7474 7473 55410 45528 i
*> 1.22.163.0/24    165.228.157.73         100     80      0 1221 38285
7474 7473 55410 45528 i
*> 1.22.167.0/24    165.228.157.73         100     80      0 1221 38285
7474 7473 6453 4755 45528 i
*> 1.22.168.0/24    165.228.157.73         100     80      0 1221 38285
7474 7473 6453 4755 45528 i
..
*  14.201.64.0/24   165.228.157.73         100     80      0 1221 38285
18398 7545 7545 i

Thanks,

Wade

-- 

Wade Millican 
Technical Consultant Team Lead
Hemisphere Infrastructure Support
Information Technology
Echo Entertainment Group Limited 

2 Edward St
Pyrmont NSW 2009 

T: +61 2 9657 7460 <tel:%2B61%202%209657%207460> 
M: +61 (0) 400 192 485 <tel:%2B61%20%280%29%20400%20192%20485> 
wade.millican at echoent.com.au
www.echoentertainment.com.au
<image001.png>

From: "Ramsay, Paul" <pramsay at uecomm.com.au>
Date: Wed, 22 Feb 2012 22:20:41 -0800
To: "ausnog at ausnog.net" <ausnog at ausnog.net>
Subject: Re: [AusNOG] International link issue

Yes, this reinforces the Rule of Trust. Don't trust your BGP peers and
ensure your filters are in place, configured correctly and working, you
can't transfer blame.

It can cost you big $$ and pain if you inadvertently turn yourself into
a transit peer because your upstreams may prefer to send traffic where
they can make $$ from.

From: ausnog-bounces at lists.ausnog.net
[mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Sean K. Finn
Sent: Thursday, 23 February 2012 5:09 PM
To: 'ausnog at ausnog.net'
Subject: Re: [AusNOG] International link issue

It's easy to describe for all the media types watching..

(And I'm not sure why its not being put out there in Laymans terms).

>From the routes seen at various points, and reported on the WAIX mailing
list earlier..

Dodo told Telstra that Dodo was the rest of the Internet.

Telstra Believed Dodo.

Telstra entire system tried to use DODO as their ISP instead of everyone
else Telstra is connected to.

Needless to say this didn't work, the pipes got Jammed.

Telstra should have filtered the announcement from Dodo, butdidn't.

Filtering is in place as a form of control (which is used instead of
trust).

Filtering obviously wasn't in place, or didn't work, so anything that
Dodo told Telstra about where to find the Internet, Telstra believed.

This happens quite often, I've heard of this happening on peering
exchanges within Australia, too. Just never at an organizational level
as big as Telstra.

Over and Out.

This message and its attachments may contain legally privileged or
confidential information. It is for the intended addressee(s) only.

If you are not the intended recipient you must not disclose or use the
information contained in it. If you have received this email in error
please notify us immediately by return email and delete the document.

Any views expressed in this message are those of the individual sender,
except where the sender specifies and with authority, states them to be
the views of the Company.

Uecomm accepts no liability for any damage caused by this email or its
attachments due to viruses, interference, interception, corruption or
unauthorised access.

________________________________

This e-mail message has been scanned for Viruses and Content and cleared
by NetIQ MailMarshal

________________________________

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
http://lists.ausnog.net/mailman/listinfo/ausnog

________________________________

Stockland Notice: If this communication has been sent to you by mistake,
please delete and notify us. If it has been sent to you by mistake,
legal privilege is not waived or lost and you are not entitled to use it
in any way. Stockland and its subsidiaries reserve the right to monitor
e-mail communication through its networks.

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
http://lists.ausnog.net/mailman/listinfo/ausnog

This message and its attachments may contain legally privileged or confidential information. It is for the intended addressee(s) only.
If you are not the intended recipient you must not disclose or use the information contained in it. If you have received this email in error please notify us immediately by return email and delete the document.
Any views expressed in this message are those of the individual sender, except where the sender specifies and with authority, states them to be the views of the Company.
Uecomm accepts no liability for any damage caused by this email or its attachments due to viruses, interference, interception, corruption or unauthorised access.

#####################################################################################
This e-mail message has been scanned for Viruses and Content and cleared 
by NetIQ MailMarshal
#####################################################################################
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20120224/60e8065d/attachment.html>