[AusNOG] AAPT Ethernet outage

[Bevan Slattery]

Welcome Art.  Thanks for getting on and providing some feedback.  I think it's good for AAPT and the Ausnog list that you've joined.  I'm sure people will appreciate you coming on-list and take it a bit easy on you (at first:))

Cheers

[b]

From: Art Cartwright <art.cartwright at aapt.com.au<mailto:art.cartwright at aapt.com.au>>
Date: Tuesday, 3 July 2012 8:33 AM
To: "ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>" <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: [AusNOG] AAPT Ethernet outage

Hi, my name is Art and I run network operations at AAPT. I am new to the forum and I wanted to give everyone an update on the event that happened on Saturday in the AAPT network.

On Saturday between the times of 12h00 and 14h30 AAPT experienced a large number of Ethernet switches (in both NSW and VIC) stop passing traffic and become unreachable.

We know now that the problem was caused by a vendor’s equipment incorrectly handling of the “Leap Second Insertion” by NTP.

At 15h00 we mobilized our on-call Field Operations staff who needed access various PoP’s in the CBD to power cycle the switches. We power cycled the first switch at 16h20 and all services on the affected switch were restored immediately.

We then mobilised more field operations staff as we knew we had to reboot all devices manually.

By 19h22 the majority of customer services were confirmed restored and by 01h15 99% of customer services were restored except for three sites where we had issues with site access.

The vendor was able to simulate the issue in their lab in the early hours of Sunday morning and isolated it to the NTP “leap second insertion”.

I accept that during the event we did a poor job communicating with customers and the broader community at to what was happening. Our updates were infrequent and at times incorrect. This is something that we are looking at improving.

I would welcome any suggestions as to the communication channels we should investigate.

Thanks

Art


This communication, including any attachments, is confidential. If you are not the intended
recipient, you should not read it - please contact me immediately, destroy it, and do not
copy or use any part of this communication or disclose anything about it.




The information contained in this email and any attachments may be confidential. This email and any attachments are also subject to copyright. No part of them may be reproduced, adapted or transmitted without the written permission of the copyright owner. If you are not the intended recipient, any use, interference with, disclosure or copying of this information is unauthorised and prohibited. If you have received this email in error, please immediately advise the sender by return email and delete the message from your system. All email communications to and from NEXTDC Limited are recorded for the purposes of archival and storage.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20120702/58f9c28b/attachment.html>