[AusNOG] International link issue

Chris Chaundy chris.chaundy at gmail.com
Fri Feb 24 12:30:39 EST 2012


As others have pointed out, in BGP local preference wins over AS path
length (and another thing people often overlook is that longer
prefixes always win over shorter prefixes, but that just basic IP
routing - nothing to do with BGP!).

The standard hierarchy in local preferences applied by service providers is:
1) preference customers (this makes you money)
2) then preference peers (this costs nothing - or not much)
3) finally preference providers (this costs the bucks - but not as
much as it used to :-)

In general, the level of trust increases in the same order (for better
or worse).  In any case, if you are not filtering on prefixes, you
should at least filter on AS path, and if you are not doing either,
you should be applying a maximum prefix limit (unless you are
expecting the global table from your provider - when routers were
memory-constrained, you probably wanted to apply this anyway in case
of 'accidents' - not such an issue these days).

Happy routing...

Chris

On Fri, Feb 24, 2012 at 12:13 PM, Aaron Swayn <aaron at swayn.com> wrote:
> I’ve seen Mr BGP, Geoff Huston lurks on these forums from time to time (I
> think he appeared a few day ago). Might be a good time to suggest his next
> monthly column on http://www.potaroo.net/ for March to cover ISP BGP peering
> design. Seems the list has some significant interest in how, where, why and
> best practices. I believe Geoff has written a few books on ISP’s over the
> many years and could share some valuable insights (and maybe some war
> stories too from his days at Telstra)
>
>
>
> Food for thought.
>
>
>
> From: ausnog-bounces at lists.ausnog.net
> [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Sean K. Finn
> Sent: Friday, 24 February 2012 11:58 AM
> To: 'Ausnog at ausnog.net'
>
>
> Subject: Re: [AusNOG] International link issue
>
>
>
> I’ll be honest here and share an experience.
>
>
>
> (All beit an embarrassing one, but I’m willing to admit it, I’m sure there
> are many others on the list who have done it who may not admit it).
>
>
>
> In my early days learning live BGP, (There is no testing with Internet BGP,
> it’s always live, all the time).
>
> There’s no putting your feet into the water to learn it, it’s head first
> sink or swim.
>
>
>
> When I turned on our second multi homed connection, and started taking full
> global routing tables, I had no AS path filtering in place, and no Prefix
> filtering in place. I didn’t even know what they were, really, back then.
>
>
>
> I was advertising OPTUS’s entire global routing table to AAPT, and AAPT’s
> entire global routing table to OPTUS.
>
>
>
> I was advertising every piece of private address space known to man
> (10.0.0.0/8, 192.168, 172, etc etc) into BOTH of them.
>
>
>
> Did I know what was going on in the early days of BGP? Yes, I saw the
> advertisments, but, thankfully, AAPT and OPTUS had seen clients like me
> before (at that stage, without a clue). Did I know what effect it would
> have? Heck no.
>
>
>
> Did everything work? Yes, perfectly and absolutely, because there were
> controls in place to prevent any issues.
>
>
>
> Thankfully, these days, I know better.
>
>
>
> ROUTE-MAP’s and AS-PATH filters and prefix lists and policy based routing,
> community based routing, come later, after you figure out how BGP works in
> the first place.
>
>
>
> Fast forward to now.
>
>
>
> We now perform at least one or two new BGP based interconnections per week,
> and unless *we* have our policies absolutely perfect, weird weird routing
> can and does occur.
>
>
>
> I expect my downstreams to have no idea what AS path filtering or prefix
> filtering is about. I expect them to advertise 0.0.0.0/0 to me. (and some do
> J )
>
>
>
> Even then, even with appropriate route-filters and AS path filters in place,
> strangeness can occur.
>
>
>
> I’ve even seen a client’s network, that we were YET to physically connect
> to, but had setup AS path and Prefix list filtering internally and with our
> upstreams, bring traffic in through Optus, through our network, and out
> through AAPT to reach its final destination. This occurred because we’re a
> client of both Optus and AAPT and the route through us was seen as the
> shortest path. (Amazing, I know).
>
>
>
> This happened because we were learning their routes from AAPT, it was
> passing through our filtering, and then being re-advertised through to
> OPTUS.
>
>
>
> There’s no standard single way of setting up this filtering, either, every
> network is different.
>
>
>
> In some networks, filters are done manually by the network ops technicians.
>
>
>
> Other networks, its policy based, determining where to advertise routes to
> based on where the routes were learnt from.
>
>
>
> In others, like the IX’s, there’s automated systems that generate policy and
> route-maps and prefix-lists based on what you punch into them, and push the
> configs out to the routers that matter at various intervals.
>
>
>
> It’s so damn easy to make a mistake with any of these systems when bringing
> on a new peer (Transit  peer, upstream peer, peer peer, or client peer).
>
>
>
> Maybe a good topic for this year’s AusNOG, on different ways to do
> AS-Path-Filtering and Prefix-List Filtering.
>
>
>
> Does anybody else care to share their experiences?
>
>
>
> From: Burt Mascareigne [mailto:burt at prioritycomputer.com.au]
> Sent: Friday, February 24, 2012 10:42 AM
> To: James Spenceley; Sean K. Finn
> Cc: 'ausnog at ausnog.net'
> Subject: RE: [AusNOG] International link issue
>
>
>
> /Agree
>
>
>
> Now let’s get the jurnos to understand that and to pass that onto the
> public,
>
>
>
> Ready, Set, Go!
>
>
>
> From: ausnog-bounces at lists.ausnog.net
> [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of James Spenceley
> Sent: Friday, 24 February 2012 11:39 AM
> To: Sean K. Finn
> Cc: 'ausnog at ausnog.net'
> Subject: Re: [AusNOG] International link issue
>
>
>
> Well said Sean.
>
>
>
> If anyone thinks this is Dodo's fault, think again. Customers make
> configuration errors like this regularly.
>
>
>
> I understand Telstra's position in some cases, to prefix filter us is
> something like 2500 entires with lots of updates, that can be hard to
> maintain, *at a prefix level*
>
>
>
> Telstra not having a generic AS-PATH or even max-prefix put the fault
> entirely with Telstra.
>
>
>
>
>
> On 24/02/2012, at 11:29 AM, Sean K. Finn wrote:
>
>
>
> Telstra should have had an Inbound route filter in any case.
>
> (Both an AS Path filter, and a prefix-list filter).
>
>
>
> The amount of people I’ve seen TRYING to advertise 0.0.0.0/0 over an IX, or
> even leaking private address space into carrier links, is common place.
> Thankfully most carriers don’t trust their clients advertisments by default.
>
>
>
> PIPE IX have a fairly complex (but not complicated) portal to use to ensure
> that your AS PATH LIST and PREFIX LISTS are added to their server manually
> before adding them to their route filters.Even better, every day there’s a
> mailout with the filters that you can put on your OWN routers to match their
> routers, just-in-case.
>
>
>
> I’ve seen some providers be extremely strict on even the manual ranges that
> get added to portals, checking each and every IP assignment to ensure that
> I’m allowed to advertise the ranges, and AS’s, in question. (And in some
> cases making me jump through silly hurdles to PROVE it).
>
>
>
>
>
> The very definition of UPSTREAM means it’s their responsibility to not trust
> the DOWNSTREAM.
>
>
>
>
>
> Fair enough, Dodo had to leak the routes, and Telstra had to accept them,
> there was a mutual mis-configuration, that part is self evident.
>
>
>
> It’s easy enough to make the mistake.
>
>
>
> S
>
> From: ausnog-bounces at lists.ausnog.net [mailto:ausnog-bounces at lists.ausnog.net] On
> Behalf Of Aaron Swayn
> Sent: Friday, February 24, 2012 10:20 AM
> To: ausnog at ausnog.net
> Subject: Re: [AusNOG] International link issue
>
>
>
> I think the spin was from Dodo to blame the vendor, but me thinks the fault
> went down something like this…
>
>
>
> -          Dodo engineer troubleshooting BGP advertisement issue
>
> -          Can’t figure it out (can’t see why, these things do work properly
> and you can debug them easily enough without having to take risks)
>
> -          Removed outbound route filter from BGP session to Telstra to see
> if that fixes the problem
>
>
>
> And we know the rest…
>
>
>
> From: ausnog-bounces at lists.ausnog.net [mailto:ausnog-bounces at lists.ausnog.net] On
> Behalf Of Evan Weston
> Sent: Friday, 24 February 2012 10:55 AM
> To: ausnog at ausnog.net
> Subject: Re: [AusNOG] International link issue
>
>
>
> Nice spin from Telstra. Blame the hardware vendor, blame the customer but
> never admit that it was actually *our fault* for not filtering properly.
>
>
>
> From: ausnog-bounces at lists.ausnog.net [mailto:ausnog-bounces at lists.ausnog.net] On
> Behalf Of Will Tardy
> Sent: Friday, 24 February 2012 10:30 AM
> To: ausnog at ausnog.net
> Subject: Re: [AusNOG] International link issue
>
>
>
> Telstra claims they had an international link down:
>
>
>
> http://www.zdnet.com.au/telstra-hit-by-nationwide-data-outage-339332310.htm
>
>
>
> If that happened at the same time as DODO incorrectly sending Telstra the
> full BGP table, could that explain why Telstra black-holed all-routes plus
> pumped all of it's own traffic via dodo?
>
> On 24 February 2012 10:02, Wade Millican <Wade.Millican at echoent.com.au>
> wrote:
>
> Hi All,
>
>
>
> What I'm yet to understand about this outage is why DODO's AS_PATH was seen
> as shorter than anything Telstra already had.
>
>
>
> An earlier posted look at routes(below), thanks Gavin, shows all routes from
> Telstra taking hops to DODO, then Optus or PIPE before moving to the
> destination. Surely Telstra would have had better routes than pushing all
> traffic 2 hops out of it's way.
>
>
>
> AS_PATH does not explain how Telstra accepted these as the active routes.
> Even if all routes were accepted, Telstra still has better routes.
>
>
>
> Can anyone explain what BGP Metric was modified/used that pushed traffic
> over longer AS_PATHs?
>
>
>
>
>
> *> 1.22.161.0/24    165.228.157.73         100     80      0 1221 38285 7474
> 7473 55410 45528 i
>
> *> 1.22.162.0/24    165.228.157.73         100     80      0 1221 38285 7474
> 7473 55410 45528 i
>
> *> 1.22.163.0/24    165.228.157.73         100     80      0 1221 38285 7474
> 7473 55410 45528 i
>
> *> 1.22.167.0/24    165.228.157.73         100     80      0 1221 38285 7474
> 7473 6453 4755 45528 i
>
> *> 1.22.168.0/24    165.228.157.73         100     80      0 1221 38285 7474
> 7473 6453 4755 45528 i
>
> ..
>
> *  14.201.64.0/24   165.228.157.73         100     80      0 1221 38285
> 18398 7545 7545 i
>
>
>
> Thanks,
>
>
>
> Wade
>
> --
>
> Wade Millican
> Technical Consultant Team Lead
> Hemisphere Infrastructure Support
> Information Technology
> Echo Entertainment Group Limited
>
> 2 Edward St
> Pyrmont NSW 2009
>
> T: +61 2 9657 7460
> M: +61 (0) 400 192 485
> wade.millican at echoent.com.au
> www.echoentertainment.com.au
> <image001.png>
>
> From: "Ramsay, Paul" <pramsay at uecomm.com.au>
> Date: Wed, 22 Feb 2012 22:20:41 -0800
> To: "ausnog at ausnog.net" <ausnog at ausnog.net>
> Subject: Re: [AusNOG] International link issue
>
>
>
> Yes, this reinforces the Rule of Trust. Don’t trust your BGP peers and
> ensure your filters are in place, configured correctly and working, you
> can’t transfer blame.
>
> It can cost you big $$ and pain if you inadvertently turn yourself into a
> transit peer because your upstreams may prefer to send traffic where they
> can make $$ from.
>
>
>
> From: ausnog-bounces at lists.ausnog.net [mailto:ausnog-bounces at lists.ausnog.net] On
> Behalf Of Sean K. Finn
> Sent: Thursday, 23 February 2012 5:09 PM
> To: 'ausnog at ausnog.net'
> Subject: Re: [AusNOG] International link issue
>
>
>
> It’s easy to describe for all the media types watching..
>
> (And I’m not sure why its not being put out there in Laymans terms).
>
>
>
> From the routes seen at various points, and reported on the WAIX mailing
> list earlier..
>
>
>
>
>
>
>
> Dodo told Telstra that Dodo was the rest of the Internet.
>
>
>
> Telstra Believed Dodo.
>
>
>
> Telstra entire system tried to use DODO as their ISP instead of everyone
> else Telstra is connected to.
>
>
>
> Needless to say this didn’t work, the pipes got Jammed.
>
>
>
> Telstra should have filtered the announcement from Dodo, butdidn’t.
>
>
>
> Filtering is in place as a form of control (which is used instead of trust).
>
>
>
> Filtering obviously wasn’t in place, or didn’t work, so anything that Dodo
> told Telstra about where to find the Internet, Telstra believed.
>
>
>
> This happens quite often, I’ve heard of this happening on peering exchanges
> within Australia, too. Just never at an organizational level as big as
> Telstra.
>
>
>
> Over and Out.
>
>
>
>
>
> This message and its attachments may contain legally privileged or
> confidential information. It is for the intended addressee(s) only.
>
> If you are not the intended recipient you must not disclose or use the
> information contained in it. If you have received this email in error please
> notify us immediately by return email and delete the document.
>
> Any views expressed in this message are those of the individual sender,
> except where the sender specifies and with authority, states them to be the
> views of the Company.
>
> Uecomm accepts no liability for any damage caused by this email or its
> attachments due to viruses, interference, interception, corruption or
> unauthorised access.
>
> ________________________________
>
> This e-mail message has been scanned for Viruses and Content and cleared
> by NetIQ MailMarshal
>
> ________________________________
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
>
>
> ________________________________
>
> Stockland Notice: If this communication has been sent to you by mistake,
> please delete and notify us. If it has been sent to you by mistake, legal
> privilege is not waived or lost and you are not entitled to use it in any
> way. Stockland and its subsidiaries reserve the right to monitor e-mail
> communication through its networks.
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog
>



More information about the AusNOG mailing list