[AusNOG] Best practice BGP and wan links

Mark Smith markzzzsmith at gmail.com
Mon Jul 13 14:06:09 EST 2015


So to summarise my last email,

If you think you need scripting and VRRP, you don't know enough about BGP.

Can people please stop thinking about using VRRP? It was never
designed to be used in this scenario (it's a first hop/last redundancy
protocol for devices that *don't* participate in routing protocols -
i.e., usually hosts), and you're just complicating things (creating
*more* possible failure points) trying to use it without any benefit.
You want your BGP sessions to fail when your routers or links fail,
because that is how the upstream network learns that there has been a
topology change, which will then cause it to find another path to you.

Multiple failure recovery methods that are trying to deal with the
same failure can actually make recovery take longer or even fail,
because the recovery methods can interact with each other while trying
to recover, and can therefore be fooled by each other.

On 13 July 2015 at 13:49, Joseph Goldman <joe at apcs.com.au> wrote:
>  You are still ultimately going to need a /29 even with VRRP, unless you use
> a different IP subnet for VRRP and using the on mater / on backup scripting
> built in to RouterOS.
>
>  From my understanding of VRRP implementation, if you want a virtual IP in
> the /30 subnet, you'd need both devices in the /30 then the Virtual IP as a
> /32 on top (on the VRRP interface) within the /30 subnet - if that makes
> sense. My only away around this would be to use say private IP addressing
> just for VRRP communication, and use the scripting to enable/disable the BGP
> communication IP.
>
>  It also adds another protocol and level of complexity for no real benefit -
> if moving to a /29 anyway, you are better having BFD/low timers accross 2
> separate BGP sessions then as one goes down, the next session is already
> established you are just waiting for routes to withdraw from the first
> session (no re-import of routes, or route flap to the upstream provider).
>
>  Even though I rambled a bit, in conclusion: /29, 2 separate BGP sessions, 1
> from each router.
>
> On 13/07/15 13:18, Alex Samad - Yieldbroker wrote:
>>
>> Hi
>>
>> No I think you have over thought it.
>>
>> VRRP was in relation to individual ISP.  for example if I connected to
>> telstra.
>>
>> The wan link goes into 1 stacked switch (yes the cable and the 1 unit of
>> the switch are single points of failure)
>>
>> But then I would have 2 CCR's connected to that WAN ip network.  The
>> question was based around do I ask telstra for another ip and change the wan
>> ip from /30 to /29. or do I try and setup a VRRP setup. I had presumed that
>> I would have to script something to bring down BGP on the CCR that didn't
>> have the VIP.  And yes there would be some downtime as the BGP peer session
>> was rebuilt.
>>
>> but my presumption was with BFD or low BGP timers. the link would be
>> observed as being down very quickly and traffic from telstra would come in
>> the second link i have from them ... to another switch stack and another
>> pair of ccr's.
>>
>> I would then extend this to each of the ISP connections. so 1 VRRP session
>> per ISP connection.
>>
>> Alex
>>
>> ________________________________________
>> From: Mark Smith [markzzzsmith at gmail.com]
>> Sent: Monday, 13 July 2015 1:00 PM
>> To: Alex Samad - Yieldbroker
>> Cc: Benoit Page-Guitard; ausnog at lists.ausnog.net
>> Subject: Re: [AusNOG] Best practice BGP and wan links
>>
>> On 12 July 2015 at 19:09, Alex Samad - Yieldbroker
>> <Alex.Samad at yieldbroker.com> wrote:
>>>
>>> Hi
>>>
>>> Yes more info.  Multiple connections to multiple ISP's.  Currently they
>>> are terminated into switches
>>
>> Presumably two separate switches so that a single switch isn't the
>> single point of failure?
>>
>>> and then L3 terminated into RouterOS VM's.  I am planning on replacing
>>> the VM's with  some MT CCR's. My thought had been to leave the termination
>>> into the switches and then  L3 terminate onto the phy MT boxes.  As I can't
>>> HSRP / stack the routers my only option was VRRP. But BGP VRRP didn't seem
>>> like a good thing,
>>
>> So originally it seemed that you were thinking of doing VRRP with a
>> single upstream provider, which would have meant that your single BGP
>> session would only be active on one of your VRRP routers, and then if
>> VRRP switched to the other router for some reason, the BGP session
>> would go down and then come back up. That would have added BGP
>> neighbor discovery and BGP session initialisation time to your fail
>> over period, which could be a significant time increase.
>>
>> VRRP to two different providers would be both unusual and possibly
>> quite confusing to both of them. One of them would have a BGP neighbor
>> address that doesn't fall within the IP subnet/prefix that they've
>> assigned to their end of the link, which would be different from most
>> if not all of their other customers. The other confusing thing would
>> be that the BGP session for the non-active provider would be down
>> while you're not using that provider for traffic. That would be
>> confusing for them, because providers will consider a BGP session that
>> is up to mean the customer intends to use the link even if there is no
>> traffic flowing over it i.e., a link over which a BGP session is up
>> with no traffic is clearly a backup link. However, if the BGP session
>> is down and stays down for long periods, it would be considered a sign
>> that something is broken rather than an intentional but unusual setup
>> like using VRRP with a single BGP session to two different providers.
>>
>> Another issue would have been if your segment became partitioned such
>> that both of your VRRP routers became active, meaning that you had two
>> active BGP sessions with the same IP address at your end to both of
>> your providers. While odd, at first impression I can't see how that
>> would cause a problem, but to be sure of no issues, you'd have to very
>> thoroughly work through all the possible failure modes of this
>> scenario. I think there is a high likelihood it would cause problems
>> if this happened and you were using the same upstream upstream
>> provider for the two links.
>>
>> Regards,
>> Mark.
>>
>>
>>
>>> better to get the extra IP and have  2 links.
>>>
>>> Interestingly I have BFD running on some of those links and reduced
>>> timers on the BGP session for the other links as some ISP didn't/wouldn't
>>> run BFD..
>>>
>>>
>>> Thanks
>>> Alex
>>>
>>> -----Original Message-----
>>> From: Mark Smith [mailto:markzzzsmith at gmail.com]
>>> Sent: Sunday, 12 July 2015 5:54 PM
>>> To: Alex Samad - Yieldbroker
>>> Cc: Benoit Page-Guitard; ausnog at lists.ausnog.net
>>> Subject: Re: [AusNOG] Best practice BGP and wan links
>>>
>>> On 12 July 2015 at 15:14, Alex Samad - Yieldbroker
>>> <Alex.Samad at yieldbroker.com> wrote:
>>>>
>>>> Yeah that was sort of my thought, I guess I have to start the process of
>>>> asking for the extra IP..
>>>>
>>> More details of your scenario would be better.
>>>
>>> VRRP being an option means that you only have a single link to your
>>> upstream. Since in general links fail more often than devices, the
>>> redundancy value of having two routers at your end and two BGP sessions over
>>> a single link to a single upstream router is a bit questionable, because you
>>> haven't eliminated all single points of failure. You have partial but not
>>> complete redundancy, and you need to consider whether not having complete
>>> redundancy is acceptable to either or both you or your network's users.
>>>
>>>
>>>
>>>> A
>>>>
>>>> -----Original Message-----
>>>> From: Benoit Page-Guitard [mailto:benoit at anchor.net.au]
>>>> Sent: Saturday, 11 July 2015 11:13 PM
>>>> To: Alex Samad - Yieldbroker
>>>> Cc: ausnog at lists.ausnog.net
>>>> Subject: Re: [AusNOG] Best practice BGP and wan links
>>>>
>>>> Hi Alex,
>>>>
>>>> I assume the use case here is having redundant routers at the branch end
>>>> and using VRRP on the WAN link as a signalling mechanism for deciding which
>>>> router should "own" the WAN IP + speak BGP with the upstream router?
>>>>
>>>> If so, I'd definitely opt for an extra WAN IP if you can swing it. It'll
>>>> make the whole failover scenario a lot smoother, and would also have the
>>>> indirect benefit of giving you free load balancing for your
>>>> downstream-facing LAN interfaces.
>>>>
>>>> Regards,
>>>> Benoit
>>>>
>>>> On Sat Jul 11, 2015 at 08:03:10 +0000, Alex Samad - Yieldbroker wrote:
>>>>>
>>>>> What I was looking at doing was setting up bgp over vrrp on some
>>>>> mikrotik boxes, seems like it's possible, but it also seem easier to get an
>>>>> extra WAN ip.
>>>>>
>>>>> Any one doing this ?
>>>>>
>>>> _______________________________________________
>>>> AusNOG mailing list
>>>> AusNOG at lists.ausnog.net
>>>> http://lists.ausnog.net/mailman/listinfo/ausnog
>>
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> http://lists.ausnog.net/mailman/listinfo/ausnog
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog


More information about the AusNOG mailing list