<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"><meta name="Generator" content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";
mso-fareast-language:EN-US;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang="EN-AU" link="blue" vlink="purple"><div class="WordSection1"><p class="MsoNormal">Hi, my name is Art and I run network operations at AAPT. I am new to the forum and I wanted to give everyone an update on the event that happened on Saturday in the AAPT network.</p>
<p class="MsoNormal"> </p><p class="MsoNormal">On Saturday between the times of 12h00 and 14h30 AAPT experienced a large number of Ethernet switches (in both NSW and VIC) stop passing traffic and become unreachable. </p><p class="MsoNormal">
</p><p class="MsoNormal">We know now that the problem was caused by a vendor’s equipment i<strong><span style="font-size:12.0pt;font-family:"Calibri","sans-serif";font-weight:normal">ncorrectly handling of the “Leap Second Insertion” by </span></strong><strong><span style="font-size:12.0pt;font-family:"Calibri","sans-serif"">NTP</span></strong><strong><span style="font-size:12.0pt;font-family:"Calibri","sans-serif";font-weight:normal">.</span></strong></p>
<p class="MsoNormal"><strong><span style="font-size:12.0pt;font-family:"Calibri","sans-serif";font-weight:normal"> </span></strong></p><p class="MsoNormal"><strong><span style="font-size:12.0pt;font-family:"Calibri","sans-serif";font-weight:normal">At 15h00 we mobilized our on-call Field Operations staff who needed access various PoP’s in the CBD to power cycle the switches. We power cycled the first switch at 16h20 and </span></strong><span style="color:black">all services on the affected switch were restored immediately.</span></p>
<p class="MsoNormal"><span style="color:black"> </span></p><p class="MsoNormal"><span style="color:black">We then mobilised more field operations staff as we knew we had to reboot all devices manually.</span></p><p class="MsoNormal">
<span style="color:black"> </span></p><p class="MsoNormal"><strong><span style="font-size:12.0pt;font-family:"Calibri","sans-serif";font-weight:normal">By 19h22 the m</span></strong><span style="color:black">ajority of customer services were confirmed restored and by 01h15 99% of customer services were restored except for three sites where we had issues with site access. </span></p>
<p class="MsoNormal"><span style="color:black"> </span></p><p class="MsoNormal"><span style="color:black">The vendor was able to simulate the issue in their lab in the early hours of Sunday morning and isolated it to the NTP “leap second insertion”.</span></p>
<p class="MsoNormal"><span style="color:black"> </span></p><p class="MsoNormal">I accept that during the event we did a poor job communicating with customers and the broader community at to what was happening. Our updates were infrequent and at times incorrect. This is something that we are looking at improving.</p>
<p class="MsoNormal"> </p><p class="MsoNormal">I would welcome any suggestions as to the communication channels we should investigate.</p><p class="MsoNormal"> </p><p class="MsoNormal">Thanks</p><p class="MsoNormal"> </p>
<p class="MsoNormal">Art</p><p class="MsoNormal"> </p></div></body></html>
<pre>This communication, including any attachments, is confidential. If you are not the intended
recipient, you should not read it - please contact me immediately, destroy it, and do not
copy or use any part of this communication or disclose anything about it.