<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.emailquote, li.emailquote, div.emailquote
{mso-style-name:emailquote;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:1.0pt;
border:none;
padding:0cm;
font-size:12.0pt;
font-family:"Times New Roman";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:595.0pt 842.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
-->@-ms-viewport {
width: device-width;
}
</style>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body bgcolor="white" lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US">Hi All,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US">We had a major issue with our Extreme Networks stack in our core network in Equinix SY1, as the service was flapping, it was not reporting as a hard down, therefore
transit was not failing over correctly to other data centres for some customers.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US">When a hard down in a network happens, redundancy kicks in within the core, todays issues we saw OSPF flapping within the switch stack, meaning that the redundant
sites and paths did not consider it down long enough to re route traffic.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US">This subsequently caused a heap more load on the switch stack and caused it to reload, multiple times, after crashing.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US">When it was reloading, people were seeing devices come back as the second data centres were taking over as designed.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US">This is not something that should have happened and we are looking into this, the main goal was to get the stack restored as fast as possible.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US">It has been a very long time since we have had a major outage of this size, and I am disappointed that it has happened and I apologise to anyone who was affected.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US">We will be working with the switch vendors to see what happened and will issue a Post Incident Report to anyone that has a ticket open and linked to this incident.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US">For those not able to get through on the phones, we had too many calls and the system could not cope. We are already implementing a plan for that now. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US"><br>
We put a status message up on status.mysau.com.au and the phones to alleviate he call volume and this did help, but we need to be able to handle larger volumes of calls.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US">If anyone is still affected, please call or email us on 1300 788 862 or <a href="mailto:support@serversaustralia.com.au">support@serversaustralia.com.au</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US">Kind Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US">Nick Pratley<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #FFFFFF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">From: </span>
</b><span style="font-family:Calibri;color:black">AusNOG <ausnog-bounces@lists.ausnog.net> on behalf of Benjamin Ricardo <ben.ricardo@acs.com.au><br>
<b>Date: </b>Monday, 4 September 2017 at 7:40 am<br>
<b>To: </b>Burt Mascareigne <Burt@stormnetwork.com.au>, "ausnog@ausnog.net" <ausnog@lists.ausnog.net><br>
<b>Subject: </b>Re: [AusNOG] VOCUS or Server Australia<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">We've got connection failures to a provider at SA this morning too.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div id="x_composer_signature">
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;color:#364F67">Sent on the go with Vodafone<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<br>
---<br><table cellspacing="0" cellpadding="0" border="0">
<tbody><tr><td style="max-width: 100%; height: auto;"><p style="margin:0"><table style="width: 500px;">
<tbody>
<tr style="height: 227px;">
<td style="width: 502px; height: 227px;" valign="top">
<p><span style="font-family: arial, helvetica, sans-serif;"><a href="http://www.serversaustralia.com.au" style="font-family: arial, helvetica, sans-serif;"><img src="https://d23fetfglg1ija.cloudfront.net/signature_fields/58247fb50e0f756394ef223b/logo-sau-blue.gif" border="0" alt="logo-sau-blue.gif" height="27" width="264"></a></span><br><br> <span style="font-family: arial, helvetica, sans-serif; color: #00ce7d; font-size: 21px;"><strong>Nick Pratley</strong></span><br> <span style="font-family: arial, helvetica, sans-serif; color: #8c8c8c; font-size: 15px;"><strong>Senior Network Operator</strong></span></p>
<p><span style="font-family: arial, helvetica, sans-serif; color: #0769ba; font-size: 14px;">Direct Phone: </span> <span style="font-family: arial, helvetica, sans-serif; color: #3d3d3d; font-size: 14px;">0281158817</span> <br> <span style="font-family: arial, helvetica, sans-serif; color: #0769ba; font-size: 14px;">Phone: </span> <span style="font-family: arial, helvetica, sans-serif; color: #3d3d3d; font-size: 14px;">+61 2 8115 8888</span> <br> <span style="font-family: arial, helvetica, sans-serif; color: #0769ba; font-size: 14px;">Email: </span> <span style="font-family: arial, helvetica, sans-serif; color: #3d3d3d; font-size: 14px;">nick.pratley@serversaustralia.com.au</span></p>
<p><span style="font-family: arial, helvetica, sans-serif; color: #3d3d3d; font-size: 14px;">11/6 Reliance Drive</span> <br> <span style="font-family: arial, helvetica, sans-serif; color: #3d3d3d; font-size: 14px;">Tuggerah NSW 2259</span> <br> <span style="font-family: arial, helvetica, sans-serif; color: #3d3d3d; font-size: 14px;"><a href="http://www.serversaustralia.com.au" target="
_blank" style="font-family: arial, helvetica, sans-serif; color: #3d3d3d; font-size: 14px;">www.serversaustralia.com</a></span></p>
</td>
<td style="width: 95px; height: 227px;" valign="top"><img src="http://info.serversaustralia.com.au/hubfs/Brand-2017/strip.gif" width="95" height="221"></td>
</tr>
<tr style="height: 17px;">
<td style="height: 17px;" colspan="2" valign="top"><span style="font-family: arial, helvetica, sans-serif; color: #8c8c8c; font-size: 10px;">Notice: This message may contain private and confidential information intended only for the recipients. If you have received this message in error please delete immediately and notify the sender. Any distribution or reproduction of this message is prohibited. The views & opinions expressed in this email are NOT necessarily those of Servers Australia.</span></td>
</tr>
</tbody>
<br>
</table><a href="http://signature.serversaustralia.com.au/uc/582afac90e0f7519c33643a0?recipient=YXVzbm9nQGxpc3RzLmF1c25vZy5uZXQ=" class="sigstr-cta-link" style="outline:0 !important; text-decoration:none;"><img src="http://signature.serversaustralia.com.au/uc/582afac90e0f7519c33643a0/img?recipient=YXVzbm9nQGxpc3RzLmF1c25vZy5uZXQ=" class="sigstr-cta" border="0" alt="What does Servers Australia do?" style='color:blue;font-family:"Helvetica";font-size:12px;' width="526" height="70"></a>
</p></td></tr></tbody></table>----- Original message --------<br>
From: Burt Mascareigne <Burt@stormnetwork.com.au> <br>
Date: 4/09/2017 7:37 AM (GMT+10:00) <br>
To: ausnog <ausnog@lists.ausnog.net> <br>
Subject: [AusNOG] VOCUS or Server Australia <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt">Hi Guys<br>
<br>
Is anyone using VOCUS down? Apart from Sea-Me-We 3 I see nothing on their Status page.
<br>
<br>
VOCUS NOC mentioned Kent St is down. A bit vague, anyone know more?<br>
<br>
<br>
<br>
Regards,<br>
<br>
<br>
Burt Mascareigne<br>
<br>
_______________________________________________<br>
AusNOG mailing list<br>
AusNOG@lists.ausnog.net<br>
<a href="http://lists.ausnog.net/mailman/listinfo/ausnog">http://lists.ausnog.net/mailman/listinfo/ausnog</a><o:p></o:p></span></p>
</div>
</div>
</body>
</html>