<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI Emoji";
panose-1:2 11 5 2 4 2 4 2 2 3;}
@font-face
{font-family:"Lucida Sans Unicode";
panose-1:2 11 6 2 3 5 4 2 2 4;}
@font-face
{font-family:"Gill Sans MT";
panose-1:2 11 5 2 2 1 4 2 2 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.apple-style-span
{mso-style-name:apple-style-span;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Well,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks for your insight there Craig and for sharing so much with this community!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">-DD<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>From:</b> AusNOG <ausnog-bounces@lists.ausnog.net> <b>On Behalf Of
</b>Craig Askings<br>
<b>Sent:</b> Wednesday, 23 September 2020 11:02 PM<br>
<b>To:</b> Ausnog <ausnog@lists.ausnog.net><br>
<b>Subject:</b> Re: [AusNOG] Broadcast Storm - IX NextDC S1<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">I found other channels reach out to other network engineers to be effective so didn’t resort to ausnog. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-bottom:12.0pt">On 23 Sep 2020, at 10:41 pm, David Dunn <<a href="mailto:david@daviddunn.com.au">david@daviddunn.com.au</a>> wrote:<o:p></o:p></p>
</blockquote>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Hi all,<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I’d love others to provide their thoughts regarding what happened during this time as I was quite surprised to see this not be a thread at the actual time of occurrence.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Multiple providers were posting network notifications at the time that there were issues (Micron21, Zettagrid, Vultr) but no one has elaborated as to what the actual cause of the (what appears to be) peering issues were (BGP flapping, etc).<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Are we sure this was just limited to MegaIX Sydney? Or was it a peer doing this over multiple Sydney based IX’s with a common peer? Who was the peer that caused this?<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Looks like you have done your research Karl so please share <span style="font-family:"Segoe UI Emoji",sans-serif">
😊</span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Kind regards,<o:p></o:p></p>
<p class="MsoNormal">David<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>From:</b> AusNOG <<a href="mailto:ausnog-bounces@lists.ausnog.net">ausnog-bounces@lists.ausnog.net</a>>
<b>On Behalf Of </b>Karl Hardisty<br>
<b>Sent:</b> Wednesday, 23 September 2020 4:38 PM<br>
<b>To:</b> Ryan van Klaveren <<a href="mailto:ryan.vanklaveren@comvergence.com.au">ryan.vanklaveren@comvergence.com.au</a>><br>
<b>Cc:</b> Ausnog <<a href="mailto:ausnog@lists.ausnog.net">ausnog@lists.ausnog.net</a>><br>
<b>Subject:</b> Re: [AusNOG] Broadcast Storm - IX NextDC S1<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Hi Ryan,<o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">You’re a little late to the party; there was an ARP storm on Mega-IX Sydney yesterday. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Much hilarity ensued. Particularly if peers had default Juniper ARP policers. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">TL:DR it wasn’t just you. It was most of Au and NZ (sucking at the teat of MegaIX in Sydney). <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="color:black">Kind regards,</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Karl</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
<div>
<p class="MsoNormal"><span class="apple-style-span"><b><span lang="EN-GB" style="font-size:11.5pt;font-family:"Gill Sans MT",sans-serif;color:#C00000">l</span></b></span><span class="apple-style-span"><b><span lang="EN-GB" style="font-size:11.5pt;font-family:"Gill Sans MT",sans-serif;color:#4A442A">E</span></b></span><span class="apple-style-span"><span lang="EN-GB" style="font-size:10.0pt;font-family:"Gill Sans MT",sans-serif;color:#4A442A"> </span></span><a href="mailto:karl@mothership.co.nz"><span lang="EN-GB" style="font-size:10.0pt;font-family:"Gill Sans MT",sans-serif">karl@mothership.co.nz</span></a><span class="apple-style-span"><span lang="EN-GB" style="font-size:10.0pt;font-family:"Gill Sans MT",sans-serif;color:#4A442A"> </span></span><span class="apple-style-span"><b><span lang="EN-GB" style="font-size:11.5pt;font-family:"Gill Sans MT",sans-serif;color:#C00000">l</span></b></span><span class="apple-style-span"><b><span lang="EN-GB" style="font-size:11.5pt;font-family:"Gill Sans MT",sans-serif;color:#4A442A">W </span></b></span><a href="http://mothership.co.nz/"><span lang="EN-GB" style="font-size:10.0pt;font-family:"Gill Sans MT",sans-serif">mothership.co.nz</span></a><span class="apple-style-span"><span lang="EN-GB" style="font-size:10.0pt;font-family:"Gill Sans MT",sans-serif;color:#4A442A"> </span></span><span class="apple-style-span"><b><span lang="EN-GB" style="font-size:11.5pt;font-family:"Gill Sans MT",sans-serif;color:#C00000">l</span></b></span><span class="apple-style-span"><b><span lang="EN-GB" style="font-size:11.5pt;font-family:"Gill Sans MT",sans-serif;color:#4A442A">A</span></b></span><span class="apple-style-span"><span lang="EN-GB" style="font-size:10.0pt;font-family:"Gill Sans MT",sans-serif;color:#4A442A"> PO
Box 99814, Newmarket</span></span><span class="apple-style-span"><span lang="EN-GB" style="font-size:11.5pt;color:#515151"> </span></span><span class="apple-style-span"><b><span lang="EN-GB" style="font-size:11.5pt;font-family:"Gill Sans MT",sans-serif;color:#C00000">l</span></b></span><span class="apple-style-span"><b><span lang="EN-GB" style="font-size:11.5pt;font-family:"Gill Sans MT",sans-serif;color:#4A442A">M </span></b></span><span style="font-size:10.0pt;font-family:"Gill Sans MT",sans-serif;color:#4A442A">021
999 990 </span><span class="apple-style-span"><b><span lang="EN-GB" style="font-size:11.5pt;font-family:"Gill Sans MT",sans-serif;color:#C00000">l</span></b></span><span class="apple-style-span"><b><span style="font-size:11.5pt;font-family:"Gill Sans MT",sans-serif;color:#4A442A">P</span></b></span><span class="apple-style-span"><span style="font-size:10.0pt;font-family:"Gill Sans MT",sans-serif;color:#4A442A"> 09
974 3171</span></span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On 23/09/2020, at 6:32 PM, Ryan van Klaveren <<a href="mailto:ryan.vanklaveren@comvergence.com.au">ryan.vanklaveren@comvergence.com.au</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">Hey all,<br>
<br>
Did anyone else experience any irregularities yesterday (22 Sept) between 10:39am - 11:50am ? (I know that sounds a bit vague) - but this is what we experienced during this window:<o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">* Our nameservers failing to do resolution<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal">* Sydney transit routers had BGP flaps to IX NSW peers<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">* Sydney ASR router generated 143 logs of: PUNT_INJECT-5-DROP_PUNT_CAUSE: punt cause policer drop packet casue 7 (ARP request or response)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Spoke to IX, who said: <br>
<span style="font-size:10.5pt;font-family:"Lucida Sans Unicode",sans-serif;color:#2B2E2F">"Yep we did, yesterday another IX had an outage (I believe a broadcast storm occurred) around the timestamps in your logs, a number of our members that had remote ends
also connected to the other IX were affected by that outage and that is likely the reason for their session to flap towards us</span>" <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Just trying to see if anyone else experienced it, and better yet have any more details as to what may have caused it?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt;font-family:"Arial",sans-serif"><br>
Kind regards,<br>
Ryan van Klaveren<br>
<br>
Main: 1300 550 125<br>
Fax: 1300 550 121</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt;font-family:"Arial",sans-serif"><img border="0" width="208" height="88" style="width:2.1666in;height:.9166in" id="_x0000_i1025" src="http://www.comvergence.com.au/wp-content/themes/comvergence/img/logo.gif"></span><span style="font-size:9.5pt;font-family:"Arial",sans-serif"><br>
</span><a href="http://www.comvergence.com.au/" target="_blank"><span style="font-size:9.5pt;font-family:"Arial",sans-serif">www.comvergence.com.au</span></a><span style="font-size:9.5pt;font-family:"Arial",sans-serif"> <br>
Level 2, 38-40 Garden Street, South Yarra, VIC 3141</span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><strong><span style="font-size:9.5pt;font-family:"Arial",sans-serif">Converging Tomorrow Today</span></strong><span style="font-size:9.5pt;font-family:"Arial",sans-serif"><br>
Confidentiality Note: This e-mail is sent to and intended for use by the named addressees only. It contains confidential information. If you receive this e-mail in error, please telephone Comvergence on 1300 550 125, and then delete this message immediately.
Further, you should not disclose, print, re-transmit, copy, store, or reveal the contents of this message to any third party.</span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal">_______________________________________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a><br>
<a href="http://lists.ausnog.net/mailman/listinfo/ausnog">http://lists.ausnog.net/mailman/listinfo/ausnog</a><o:p></o:p></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<p class="MsoNormal">_______________________________________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a><br>
<a href="http://lists.ausnog.net/mailman/listinfo/ausnog">http://lists.ausnog.net/mailman/listinfo/ausnog</a><o:p></o:p></p>
</div>
</blockquote>
</div>
</body>
</html>