<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:624041873;
mso-list-type:hybrid;
mso-list-template-ids:1656262720 726430152 201916419 201916421 201916417 201916419 201916421 201916417 201916419 201916421;}
@list l0:level1
{mso-level-start-at:4;
mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1
{mso-list-id:1792361453;
mso-list-template-ids:1148100424;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style>
</head>
<body lang="EN-AU" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">Hi Guys,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">Thanks David for confirming BFD is the way to go here. Luckily, I have been able to enable BFD on all my transit links so far, so the time to detect peer failure has been quick.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">And thanks Geoff for your detailed reply. From some off-list discussions, I think that I first need to apply some of the configs (like Add-Path) that I mentioned originally and see how
I go from there, and also need to pinpoint with more certainty where the issue is occurring.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">I know that I’ve mentioned primary/secondary transit links, but I actually _<i>am</i>_ announcing all prefixes on all transit links, and I’m only using AS Path prepending to try and
optimise routing for prefixes that are in VIC vs NSW. So it’s not a case of conditionally advertising routes in this case. I did also try advertising more specific prefixes (e.g. /22 at NSW and /24 in VIC) but I found anecdotally that AS path prepending was
faster for the inbound traffic to converge during failover.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">So in a sense, I _<i>am</i>_ talking about MRAI timers, which I totally understand is just not a valid discussion to be having in the context of the general internet and it’s likely
that yes, the outage window I’m seeing when a prefix is announced over a new transit path is totally reasonable. BUT where I start to run into a problem with the outcome is still this way when I have multiple links with a single transit provider. For example:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<ul style="margin-top:0cm" type="disc">
<li class="MsoNormal" style="mso-list:l0 level1 lfo3"><span lang="EN-GB" style="mso-fareast-language:EN-US">I have cross-connect directly between one of my transit edge routers and one of their routers.<o:p></o:p></span></li><li class="MsoNormal" style="mso-list:l0 level1 lfo3"><span lang="EN-GB" style="mso-fareast-language:EN-US">I have another cross-connect directly between another of my transit edge routers and another of their routers (and this is not to mean that I intend
this to be a backup path – I send out traffic active/active). <o:p></o:p></span></li><li class="MsoNormal" style="mso-list:l0 level1 lfo3"><span lang="EN-GB" style="mso-fareast-language:EN-US">Both links are to the same transit provider, in the same POP.<o:p></o:p></span></li><li class="MsoNormal" style="mso-list:l0 level1 lfo3"><span lang="EN-GB" style="mso-fareast-language:EN-US">I am advertising the same prefixes over both links, no AS path prepending, so the announcements are basically identical.<o:p></o:p></span></li><li class="MsoNormal" style="mso-list:l0 level1 lfo3"><span lang="EN-GB" style="mso-fareast-language:EN-US">My transit provider in Sydney uses localpref on their side to designate one session as “primary” and I am not able to change that. But I can and do send
traffic out on both links as equal cost.<o:p></o:p></span></li><li class="MsoNormal" style="mso-list:l0 level1 lfo3"><span lang="EN-GB" style="mso-fareast-language:EN-US">As far as the rest of the internet is concerned my prefixes are still being announced from the same transit provider, so there shouldn’t be a need to
propagate routing changes beyond my directly adjacent peer and their internal network. This is primarily why I am expecting not to see any impact in this scenario.<o:p></o:p></span></li><li class="MsoNormal" style="mso-list:l0 level1 lfo3"><span lang="EN-GB" style="mso-fareast-language:EN-US">Given that I have adjusted my MRAI timer down to 0 with my adjacent transit peers, and have BFD enabled, they should be able to switchover to the alternate
link fairly quickly<o:p></o:p></span></li><li class="MsoNormal" style="mso-list:l0 level1 lfo3"><span lang="EN-GB" style="mso-fareast-language:EN-US">And yet, I see a 20 second outage window even in this scenario when I ping from an external connection into one of my prefixes announced over this transit.<o:p></o:p></span></li></ul>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">That scenario above is mainly what I am concerned about as I didn’t expect much/any service impact in the above scenario, since I would have thought the path over the internet in general
would remain unchanged up till my transit provider’s internal network.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">Regarding what you listed as problem b) totally understand this, and I would expect some kind of delay when re-announcing via another transit since as you say, this has to propagate
through countless upstreams throughout the internet - naturally this will take time. It’s good to hear you say 20-30 seconds is a good number in terms of getting everyone to re-learn routes. That’s really helpful.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">In terms of time it takes to learn a new outbound path, I don’t see this as an issue given the options I have to announce multiple paths over iBGP and use of BFD – this should be possible
to make quick by tuning my internal peer configs.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Thanks everyone for your experiences and insights. Based on some of the replies I got, it seems like it is reasonable to expect that in the scenario described in the bullet points above, it’s possible to see very little
if any forwarding loss. And only once I am forced to advertise via a new transit would I expect to see the 20-30 second window as everyone on the internet learns a new path. I do need to improve my iBGP convergence and actually implement some of the methods
I mentioned originally, and re-evaluate so as to rule out my iBGP convergence time as the issue I’m currently seeing for the scenario in the bullet points above.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Thanks everyone for your help.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><br>
Rhys Hanrahan<br>
Chief Information Officer<br>
Nexus One Pty Ltd<br>
<br>
E: <a href="mailto:support@nexusone.com.au" target="_blank"><span style="color:#0563C1">support@nexusone.com.au</span></a><br>
P: +61 2 9191 0606<br>
W: http://www.nexusone.com.au/<br>
M: PO Box 127, Royal Exchange NSW 1225<br>
A: Level 10 307 Pitt St, Sydney NSW 2000<br>
<br>
<img border="0" width="280" height="73" style="width:2.9166in;height:.7604in" id="Picture_x0020_2" src="cid:image001.jpg@01D3AFCC.94ED3880" alt="ttp://quintus.nexusone.com.au/~rhys/nexus1-email-sig.jpg"></span><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p></o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">AusNOG <ausnog-bounces@lists.ausnog.net> on behalf of David Hughes <david@hughes.com.au><br>
<b>Date: </b>Tuesday, 27 February 2018 at 9:39 am<br>
<b>To: </b>Geoff Huston <gih@apnic.net><br>
<b>Cc: </b>"ausnog@lists.ausnog.net" <ausnog@lists.ausnog.net><br>
<b>Subject: </b>Re: [AusNOG] Best practices on speeding up BGP convergence times<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><a name="_MailOriginalBody"><o:p> </o:p></a></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">On 26 Feb 2018, at 9:52 pm, Geoff Huston <</span><a href="mailto:gih@apnic.net"><span style="mso-bookmark:_MailOriginalBody">gih@apnic.net</span><span style="mso-bookmark:_MailOriginalBody"></span></a><span style="mso-bookmark:_MailOriginalBody">>
wrote:</span></p>
</div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"><br>
a) detecting link down quickly<br>
<br>
You can adjust your BGP session keepalive timers to smaller values and make the session more sensitive to outages as a result. I also thought that these days you can get the interface status to directly map to the session state, but its been a while since
I’ve done this in anger and frankly I have NFC how to do that, even if I used to know! Maybe you are already doing that anyway.</span></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">This is the scenario I was talking about (references below). You can easily have link on a northbound interface even if the peer isn’t there (you hit a layer-2 agg switch on the way for example).
If the peer fails but you still have link on the interface you’ll be blindly forwarding packets to it, even though it’s not there anymore, until the BGP timers expire. That was the point of the lightning talk I gave way-back -then. Default timers aren’t
helpful in this situation.</span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">Fast forward to this decade and you have routing protocols that are “BFD-aware” so you have sub-second link failure detection. That allows the control plane to pull down the peer session and
remove paths to that peer from the FIB. You can only run BFD if your upstream is as well so you know they will dump the prefixes from that peer session as quickly as you will. It makes failing over to a secondary link within the same upstream provider pretty
seamless.</span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">Ref :</span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"></span><a href="http://archive.apnic.net/meetings/21/docs/sigs/routing/routing-pres-hughes-bgp.pdf"><span style="mso-bookmark:_MailOriginalBody">http://archive.apnic.net/meetings/21/docs/sigs/routing/routing-pres-hughes-bgp.pdf</span><span style="mso-bookmark:_MailOriginalBody"></span></a><span style="mso-bookmark:_MailOriginalBody"></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"></span><a href="http://lists.ausnog.net/pipermail/ausnog/2015-January/029486.html"><span style="mso-bookmark:_MailOriginalBody">http://lists.ausnog.net/pipermail/ausnog/2015-January/029486.html</span><span style="mso-bookmark:_MailOriginalBody"></span></a><span style="mso-bookmark:_MailOriginalBody"></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">David</span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">...</span></p>
</div>
</div>
</body>
</html>