[AusNOG] AAPT Ethernet outage

[Art Cartwright]

Hi, my name is Art and I run network operations at AAPT. I am new to the
forum and I wanted to give everyone an update on the event that happened on
Saturday in the AAPT network.



On Saturday between the times of 12h00 and 14h30 AAPT experienced a large
number of Ethernet switches (in both NSW and VIC) stop passing traffic and
become unreachable.



We know now that the problem was caused by a vendor’s equipment i*ncorrectly
handling of the “Leap Second Insertion” by **NTP**.*

* *

*At 15h00 we mobilized our on-call Field Operations staff who needed access
various PoP’s in the CBD to power cycle the switches. We power cycled the
first switch at 16h20 and *all services on the affected switch were
restored immediately.



We then mobilised more field operations staff as we knew we had to reboot
all devices manually.



*By 19h22 the m*ajority of customer services were confirmed restored and by
01h15 99% of customer services were restored except for three sites where
we had issues with site access.



The vendor was able to simulate the issue in their lab in the early hours
of Sunday morning and isolated it to the NTP “leap second insertion”.



I accept that during the event we did a poor job communicating with
customers and the broader community at to what was happening. Our updates
were infrequent and at times incorrect. This is something that we are
looking at improving.



I would welcome any suggestions as to the communication channels we should
investigate.



Thanks



Art

This communication, including any attachments, is confidential. If you are not the intended
recipient, you should not read it - please contact me immediately, destroy it, and do not
copy or use any part of this communication or disclose anything about it.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20120703/80b9c6d5/attachment.html>