<div dir="ltr">Having had a CCR1009 die, back when it was my 'everything' router, I now have one RouterOS VM per transit provider, plus one more which maintains V6 tunnels to a US box over all three transit providers, as a last-chance router incase all three providers' customer BGP routers die at the same time. I also have two RouterOS VM's as my core routers, sending BGP to the four edge routers, and then doing VRRP to provide routing to all the customer vlans.<div><br></div><div>The last-chance US VM, and one of the 'core router' VM's are in a XenServer all by themselves, the others are on my main cluster, with iSCSI backend, so they're fault tolerant incase of a hardware failure. </div><div><br></div><div>It works (quite well), though at some stage I would like to replace the two 'core' VM's with a pair of CCR1009's, and maybe also the edge VM pointing at my primary upstream. But I don't see the point in doing that until Mikrotik sort out their single-core routing limitation :)</div><div><br></div><div>I did initially play with BGP with VRRP to have two routers talking to one upstream, but it just confused the hell out of the upstream provider router, and generally didn't work well at all - was better to have one router to each upstream. </div></div><div class="gmail_extra"><br><div class="gmail_quote">On 13 July 2015 at 14:26, Alex Samad - Yieldbroker <span dir="ltr"><<a href="mailto:Alex.Samad@yieldbroker.com" target="_blank">Alex.Samad@yieldbroker.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi<br>
<br>
So I am going to have 2 CCR's<br>
If I could easily get VRRP and BGP working I would<br>
If I can get a second BGP session going over the same link I would prefer that than VRRP<br>
<br>
To handle the single comms line, I have multiple of those.<br>
<br>
I was hoping to get a bit of insight into what people do in regards to this.<br>
I think Joseph Goldman gave the best reply.<br>
<span class="im HOEnZb"><br>
Alex<br>
<br>
<br>
<br>
-----Original Message-----<br>
From: Mark Smith [mailto:<a href="mailto:markzzzsmith@gmail.com">markzzzsmith@gmail.com</a>]<br>
</span><div class="HOEnZb"><div class="h5">Sent: Monday, 13 July 2015 2:19 PM<br>
To: Alex Samad - Yieldbroker<br>
Cc: Benoit Page-Guitard; <a href="mailto:ausnog@lists.ausnog.net">ausnog@lists.ausnog.net</a><br>
Subject: Re: [AusNOG] Best practice BGP and wan links<br>
<br>
On 13 July 2015 at 13:52, Alex Samad - Yieldbroker <<a href="mailto:Alex.Samad@yieldbroker.com">Alex.Samad@yieldbroker.com</a>> wrote:<br>
> ???<br>
><br>
> I think you have miss understood.<br>
<br>
<br>
So my original statement,<br>
<br>
>> VRRP being an option means that you only have a single link to your upstream. Since in general links fail more often than devices, the redundancy value of having two routers at your end and two BGP sessions over a single link to a single upstream router is a bit questionable, because you haven't eliminated all single points of failure. You have partial but not complete redundancy, and you need to consider whether not having complete redundancy is acceptable to either or both you or your network's users.<br>
<br>
is the the correct one.<br>
<br>
I don't see a lot of value in having two routers and only one link, when links fail more often than routers do (assuming you have good quality and well maintained routers).<br>
<br>
><br>
> I am looking to handle 1 WAN connection with 2 CCR as the L3 termination point, I current have 1 ROS VM. But I am thinking or replacing them with 2 phy boxes. So if I have issues with 1 the other will take over.<br>
><br>
> I can ask the ISP for another ip and create 2 BGP peer sessions or I can try VRRP and BGP.<br>
><br>
> I current have a /30 for the wan link, if the ISP will not give me a<br>
> /29 so I can have 2 devices including their device on the ip network<br>
> then I am stuck and have to look at the VRRP option<br>
><br>
> I am fairly happy the rest of by BGP setup is okay.<br>
><br>
> Alex<br>
><br>
><br>
> -----Original Message-----<br>
> From: Mark Smith [mailto:<a href="mailto:markzzzsmith@gmail.com">markzzzsmith@gmail.com</a>]<br>
> Sent: Monday, 13 July 2015 1:46 PM<br>
> To: Alex Samad - Yieldbroker<br>
> Cc: Benoit Page-Guitard; <a href="mailto:ausnog@lists.ausnog.net">ausnog@lists.ausnog.net</a><br>
> Subject: Re: [AusNOG] Best practice BGP and wan links<br>
><br>
> On 13 July 2015 at 13:18, Alex Samad - Yieldbroker <<a href="mailto:Alex.Samad@yieldbroker.com">Alex.Samad@yieldbroker.com</a>> wrote:<br>
>> Hi<br>
>><br>
>> No I think you have over thought it.<br>
>><br>
>> VRRP was in relation to individual ISP. for example if I connected to telstra.<br>
>><br>
>> The wan link goes into 1 stacked switch (yes the cable and the 1 unit<br>
>> of the switch are single points of failure)<br>
>><br>
>> But then I would have 2 CCR's connected to that WAN ip network. The question was based around do I ask telstra for another ip and change the wan ip from /30 to /29. or do I try and setup a VRRP setup. I had presumed that I would have to script something to bring down BGP on the CCR that didn't have the VIP. And yes there would be some downtime as the BGP peer session was rebuilt.<br>
>><br>
>> but my presumption was with BFD or low BGP timers. the link would be observed as being down very quickly and traffic from telstra would come in the second link i have from them ... to another switch stack and another pair of ccr's.<br>
>><br>
>> I would then extend this to each of the ISP connections. so 1 VRRP session per ISP connection.<br>
><br>
> If you were thinking you could have or would benefit from doing all of this, then you were way overcomplicating things. It is not necessary at all to use any scripting (which makes the 'scripting' server another possible single point of failure) or VRRP to have BGP completely handle failover between two different providers.<br>
><br>
> I think you might need a better understanding of BGP to achieve what you want to achieve, because you're thinking you need to invent ways to handle failure modes that BGP already deals with. The book, "Internet Routing Architectures" is one of the best on BGP, and your scenario and similar ones are covered in Chapter 7, "Redundancy, Symmetry, and Load Balancing".<br>
><br>
> You need to be confident that BGP is going to handle failures properly, and one of the best ways to do that is to use the tried and tested methods rather than inventing your own.<br>
><br>
> Regards,<br>
> Mark.<br>
><br>
><br>
>><br>
>> Alex<br>
>><br>
>> ________________________________________<br>
>> From: Mark Smith [<a href="mailto:markzzzsmith@gmail.com">markzzzsmith@gmail.com</a>]<br>
>> Sent: Monday, 13 July 2015 1:00 PM<br>
>> To: Alex Samad - Yieldbroker<br>
>> Cc: Benoit Page-Guitard; <a href="mailto:ausnog@lists.ausnog.net">ausnog@lists.ausnog.net</a><br>
>> Subject: Re: [AusNOG] Best practice BGP and wan links<br>
>><br>
>> On 12 July 2015 at 19:09, Alex Samad - Yieldbroker<br>
>> <<a href="mailto:Alex.Samad@yieldbroker.com">Alex.Samad@yieldbroker.com</a>> wrote:<br>
>>> Hi<br>
>>><br>
>>> Yes more info. Multiple connections to multiple ISP's. Currently<br>
>>> they are terminated into switches<br>
>><br>
>> Presumably two separate switches so that a single switch isn't the<br>
>> single point of failure?<br>
>><br>
>>> and then L3 terminated into RouterOS VM's. I am planning on<br>
>>> replacing the VM's with some MT CCR's. My thought had been to leave<br>
>>> the termination into the switches and then L3 terminate onto the<br>
>>> phy MT boxes. As I can't HSRP / stack the routers my only option<br>
>>> was VRRP. But BGP VRRP didn't seem like a good thing,<br>
>><br>
>> So originally it seemed that you were thinking of doing VRRP with a<br>
>> single upstream provider, which would have meant that your single BGP<br>
>> session would only be active on one of your VRRP routers, and then if<br>
>> VRRP switched to the other router for some reason, the BGP session<br>
>> would go down and then come back up. That would have added BGP<br>
>> neighbor discovery and BGP session initialisation time to your fail<br>
>> over period, which could be a significant time increase.<br>
>><br>
>> VRRP to two different providers would be both unusual and possibly<br>
>> quite confusing to both of them. One of them would have a BGP<br>
>> neighbor address that doesn't fall within the IP subnet/prefix that<br>
>> they've assigned to their end of the link, which would be different<br>
>> from most if not all of their other customers. The other confusing<br>
>> thing would be that the BGP session for the non-active provider would<br>
>> be down while you're not using that provider for traffic. That would<br>
>> be confusing for them, because providers will consider a BGP session<br>
>> that is up to mean the customer intends to use the link even if there<br>
>> is no traffic flowing over it i.e., a link over which a BGP session<br>
>> is up with no traffic is clearly a backup link. However, if the BGP<br>
>> session is down and stays down for long periods, it would be<br>
>> considered a sign that something is broken rather than an intentional<br>
>> but unusual setup like using VRRP with a single BGP session to two different providers.<br>
>><br>
>> Another issue would have been if your segment became partitioned such<br>
>> that both of your VRRP routers became active, meaning that you had<br>
>> two active BGP sessions with the same IP address at your end to both<br>
>> of your providers. While odd, at first impression I can't see how<br>
>> that would cause a problem, but to be sure of no issues, you'd have<br>
>> to very thoroughly work through all the possible failure modes of<br>
>> this scenario. I think there is a high likelihood it would cause<br>
>> problems if this happened and you were using the same upstream<br>
>> upstream provider for the two links.<br>
>><br>
>> Regards,<br>
>> Mark.<br>
>><br>
>><br>
>><br>
>>> better to get the extra IP and have 2 links.<br>
>>><br>
>>> Interestingly I have BFD running on some of those links and reduced timers on the BGP session for the other links as some ISP didn't/wouldn't run BFD..<br>
>>><br>
>>><br>
>>> Thanks<br>
>>> Alex<br>
>>><br>
>>> -----Original Message-----<br>
>>> From: Mark Smith [mailto:<a href="mailto:markzzzsmith@gmail.com">markzzzsmith@gmail.com</a>]<br>
>>> Sent: Sunday, 12 July 2015 5:54 PM<br>
>>> To: Alex Samad - Yieldbroker<br>
>>> Cc: Benoit Page-Guitard; <a href="mailto:ausnog@lists.ausnog.net">ausnog@lists.ausnog.net</a><br>
>>> Subject: Re: [AusNOG] Best practice BGP and wan links<br>
>>><br>
>>> On 12 July 2015 at 15:14, Alex Samad - Yieldbroker <<a href="mailto:Alex.Samad@yieldbroker.com">Alex.Samad@yieldbroker.com</a>> wrote:<br>
>>>> Yeah that was sort of my thought, I guess I have to start the process of asking for the extra IP..<br>
>>>><br>
>>><br>
>>> More details of your scenario would be better.<br>
>>><br>
>>> VRRP being an option means that you only have a single link to your upstream. Since in general links fail more often than devices, the redundancy value of having two routers at your end and two BGP sessions over a single link to a single upstream router is a bit questionable, because you haven't eliminated all single points of failure. You have partial but not complete redundancy, and you need to consider whether not having complete redundancy is acceptable to either or both you or your network's users.<br>
>>><br>
>>><br>
>>><br>
>>>> A<br>
>>>><br>
>>>> -----Original Message-----<br>
>>>> From: Benoit Page-Guitard [mailto:<a href="mailto:benoit@anchor.net.au">benoit@anchor.net.au</a>]<br>
>>>> Sent: Saturday, 11 July 2015 11:13 PM<br>
>>>> To: Alex Samad - Yieldbroker<br>
>>>> Cc: <a href="mailto:ausnog@lists.ausnog.net">ausnog@lists.ausnog.net</a><br>
>>>> Subject: Re: [AusNOG] Best practice BGP and wan links<br>
>>>><br>
>>>> Hi Alex,<br>
>>>><br>
>>>> I assume the use case here is having redundant routers at the branch end and using VRRP on the WAN link as a signalling mechanism for deciding which router should "own" the WAN IP + speak BGP with the upstream router?<br>
>>>><br>
>>>> If so, I'd definitely opt for an extra WAN IP if you can swing it. It'll make the whole failover scenario a lot smoother, and would also have the indirect benefit of giving you free load balancing for your downstream-facing LAN interfaces.<br>
>>>><br>
>>>> Regards,<br>
>>>> Benoit<br>
>>>><br>
>>>> On Sat Jul 11, 2015 at 08:03:10 +0000, Alex Samad - Yieldbroker wrote:<br>
>>>>><br>
>>>>>What I was looking at doing was setting up bgp over vrrp on some mikrotik boxes, seems like it's possible, but it also seem easier to get an extra WAN ip.<br>
>>>>><br>
>>>>>Any one doing this ?<br>
>>>>><br>
>>>> _______________________________________________<br>
>>>> AusNOG mailing list<br>
>>>> <a href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a><br>
>>>> <a href="http://lists.ausnog.net/mailman/listinfo/ausnog" rel="noreferrer" target="_blank">http://lists.ausnog.net/mailman/listinfo/ausnog</a><br>
_______________________________________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a><br>
<a href="http://lists.ausnog.net/mailman/listinfo/ausnog" rel="noreferrer" target="_blank">http://lists.ausnog.net/mailman/listinfo/ausnog</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr">
<p>Damien Gardner Jnr<br>VK2TDG. Dip EE. GradIEAust<br><a href="mailto:rendrag@rendrag.net" target="_blank">rendrag@rendrag.net</a> - <span><a href="http://www.rendrag.net/" target="_blank">http://www.rendrag.net/</a><u><br></u></span>--<br>We rode on the winds of the rising storm,<br> We ran to the sounds of thunder.<br>We danced among the lightning bolts,<br> and tore the world asunder</p></div></div>
</div>