[AusNOG] VOCUS or Server Australia

[Nick Pratley]

Hi All,

We had a major issue with our Extreme Networks stack in our core network in Equinix SY1, as the service was flapping, it was not reporting as a hard down, therefore transit was not failing over correctly to other data centres for some customers.

When a hard down in a network happens, redundancy kicks in within the core, todays issues we saw OSPF flapping within the switch stack, meaning that the redundant sites and paths did not consider it down long enough to re route traffic.

This subsequently caused a heap more load on the switch stack and caused it to reload, multiple times, after crashing.

When it was reloading, people were seeing devices come back as the second data centres were taking over as designed.

This is not something that should have happened and we are looking into this, the main goal was to get the stack restored as fast as possible.

It has been a very long time since we have had a major outage of this size, and I am disappointed that it has happened and I apologise to anyone who was affected.

We will be working with the switch vendors to see what happened and will issue a Post Incident Report to anyone that has a ticket open and linked to this incident.

For those not able to get through on the phones, we had too many calls and the system could not cope. We are already implementing a plan for that now.

We put a status message up on status.mysau.com.au and the phones to alleviate he call volume and this did help, but we need to be able to handle larger volumes of calls.

If anyone is still affected, please call or email us on 1300 788 862 or support at serversaustralia.com.au<mailto:support at serversaustralia.com.au>

Kind Regards,
Nick Pratley


From: AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Benjamin Ricardo <ben.ricardo at acs.com.au>
Date: Monday, 4 September 2017 at 7:40 am
To: Burt Mascareigne <Burt at stormnetwork.com.au>, "ausnog at ausnog.net" <ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] VOCUS or Server Australia

We've got connection failures to a provider at SA this morning too.




Sent on the go with Vodafone


-------- Original message --------
From: Burt Mascareigne <Burt at stormnetwork.com.au>
Date: 4/09/2017 7:37 AM (GMT+10:00)
To: ausnog <ausnog at lists.ausnog.net>
Subject: [AusNOG] VOCUS or Server Australia
Hi Guys

Is anyone using VOCUS down? Apart from Sea-Me-We 3 I see nothing on their Status page.

VOCUS NOC mentioned Kent St is down.  A bit vague, anyone know more?



Regards,


Burt Mascareigne

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
http://lists.ausnog.net/mailman/listinfo/ausnog



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20170904/defbd98e/attachment.html>