<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap:break-word; color:rgb(0,0,0); font-size:14px; font-family:Calibri,sans-serif">
<div>Welcome Art. Thanks for getting on and providing some feedback. I think it's good for AAPT and the Ausnog list that you've joined. I'm sure people will appreciate you coming on-list and take it a bit easy on you (at first:))</div>
<div><br>
</div>
<div>Cheers</div>
<div><br>
</div>
<div>[b]</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; border-bottom:medium none; border-left:medium none; padding-bottom:0in; padding-left:0in; padding-right:0in; border-top:#b5c4df 1pt solid; border-right:medium none; padding-top:3pt">
<span style="font-weight:bold">From: </span>Art Cartwright <<a href="mailto:art.cartwright@aapt.com.au">art.cartwright@aapt.com.au</a>><br>
<span style="font-weight:bold">Date: </span>Tuesday, 3 July 2012 8:33 AM<br>
<span style="font-weight:bold">To: </span>"<a href="mailto:ausnog@lists.ausnog.net">ausnog@lists.ausnog.net</a>" <<a href="mailto:ausnog@lists.ausnog.net">ausnog@lists.ausnog.net</a>><br>
<span style="font-weight:bold">Subject: </span>[AusNOG] AAPT Ethernet outage<br>
</div>
<div><br>
</div>
<div><style>
<!--
@font-face
{font-family:Calibri}
@font-face
{font-family:Tahoma}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif"}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif"}
span.EmailStyle17
{font-family:"Calibri","sans-serif";
color:windowtext}
span.BalloonTextChar
{font-family:"Tahoma","sans-serif"}
.MsoChpDefault
{font-family:"Calibri","sans-serif"}
@page WordSection1
{margin:72.0pt 72.0pt 72.0pt 72.0pt}
div.WordSection1
{}
-->
</style>
<div lang="EN-AU">
<div class="WordSection1">
<p class="MsoNormal">Hi, my name is Art and I run network operations at AAPT. I am new to the forum and I wanted to give everyone an update on the event that happened on Saturday in the AAPT network.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">On Saturday between the times of 12h00 and 14h30 AAPT experienced a large number of Ethernet switches (in both NSW and VIC) stop passing traffic and become unreachable.
</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">We know now that the problem was caused by a vendor’s equipment i<strong><span style="font-size:12pt; font-weight:normal; font-family:Calibri,sans-serif">ncorrectly handling of the “Leap Second Insertion” by
</span></strong><strong><span style="font-size:12pt; font-family:Calibri,sans-serif">NTP</span></strong><strong><span style="font-size:12pt; font-weight:normal; font-family:Calibri,sans-serif">.</span></strong></p>
<p class="MsoNormal"><strong><span style="font-size:12pt; font-weight:normal; font-family:Calibri,sans-serif"> </span></strong></p>
<p class="MsoNormal"><strong><span style="font-size:12pt; font-weight:normal; font-family:Calibri,sans-serif">At 15h00 we mobilized our on-call Field Operations staff who needed access various PoP’s in the CBD to power cycle the switches. We power cycled the
first switch at 16h20 and </span></strong><span style="color:black">all services on the affected switch were restored immediately.</span></p>
<p class="MsoNormal"><span style="color:black"> </span></p>
<p class="MsoNormal"><span style="color:black">We then mobilised more field operations staff as we knew we had to reboot all devices manually.</span></p>
<p class="MsoNormal"><span style="color:black"> </span></p>
<p class="MsoNormal"><strong><span style="font-size:12pt; font-weight:normal; font-family:Calibri,sans-serif">By 19h22 the m</span></strong><span style="color:black">ajority of customer services were confirmed restored and by 01h15 99% of customer services
were restored except for three sites where we had issues with site access. </span>
</p>
<p class="MsoNormal"><span style="color:black"> </span></p>
<p class="MsoNormal"><span style="color:black">The vendor was able to simulate the issue in their lab in the early hours of Sunday morning and isolated it to the NTP “leap second insertion”.</span></p>
<p class="MsoNormal"><span style="color:black"> </span></p>
<p class="MsoNormal">I accept that during the event we did a poor job communicating with customers and the broader community at to what was happening. Our updates were infrequent and at times incorrect. This is something that we are looking at improving.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">I would welcome any suggestions as to the communication channels we should investigate.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Thanks</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Art</p>
<p class="MsoNormal"> </p>
</div>
<pre>This communication, including any attachments, is confidential. If you are not the intended
recipient, you should not read it - please contact me immediately, destroy it, and do not
copy or use any part of this communication or disclose anything about it.
</pre>
</div>
</div>
</span><span style="font-size:12px"><br>
<br>
The information contained in this email and any attachments may be confidential. This email and any attachments are also subject to copyright. No part of them may be reproduced, adapted or transmitted without the written permission of the copyright owner. If
you are not the intended recipient, any use, interference with, disclosure or copying of this information is unauthorised and prohibited. If you have received this email in error, please immediately advise the sender by return email and delete the message
from your system. All email communications to and from NEXTDC Limited are recorded for the purposes of archival and storage.</span>
</body>
</html>