[AusNOG] AAPT Ethernet outage

[Joshua D'Alton]

Welcome Art, suggestions:

status.aapt.com.au   preferably on a non-aapt service, ie cloudflare,
perhaps look how iinet does it (by no means perfect but certainly better
than bigpond for example).
twitter @aaptnoc or something
this list :)

On Tue, Jul 3, 2012 at 8:33 AM, Art Cartwright
<art.cartwright at aapt.com.au>wrote:

> Hi, my name is Art and I run network operations at AAPT. I am new to the
> forum and I wanted to give everyone an update on the event that happened on
> Saturday in the AAPT network.
>
>
>
> On Saturday between the times of 12h00 and 14h30 AAPT experienced a large
> number of Ethernet switches (in both NSW and VIC) stop passing traffic and
> become unreachable.
>
>
>
> We know now that the problem was caused by a vendor’s equipment i*ncorrectly
> handling of the “Leap Second Insertion” by **NTP**.*
>
> * *
>
> *At 15h00 we mobilized our on-call Field Operations staff who needed
> access various PoP’s in the CBD to power cycle the switches. We power
> cycled the first switch at 16h20 and *all services on the affected switch
> were restored immediately.
>
>
>
> We then mobilised more field operations staff as we knew we had to reboot
> all devices manually.
>
>
>
> *By 19h22 the m*ajority of customer services were confirmed restored and
> by 01h15 99% of customer services were restored except for three sites
> where we had issues with site access.
>
>
>
> The vendor was able to simulate the issue in their lab in the early hours
> of Sunday morning and isolated it to the NTP “leap second insertion”.
>
>
>
> I accept that during the event we did a poor job communicating with
> customers and the broader community at to what was happening. Our updates
> were infrequent and at times incorrect. This is something that we are
> looking at improving.
>
>
>
> I would welcome any suggestions as to the communication channels we should
> investigate.
>
>
>
> Thanks
>
>
>
> Art
>
>
>
> This communication, including any attachments, is confidential. If you are not the intended
> recipient, you should not read it - please contact me immediately, destroy it, and do not
> copy or use any part of this communication or disclose anything about it.
>
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20120703/9d40009e/attachment.html>