[AusNOG] Vocus Datacentre outage.

John Edwards jaedwards at gmail.com
Tue Mar 14 12:49:33 AEDT 2023 

Don't rule out a firmware setting that lay dormant or was introduced
through overzealous or simultaneous patching in a maintenance window.

This was how SA ended up with a total blackout a few years back - a default
setting in a particular model of wind turbine shut it down in response to a
previously unseen power anomaly - caused by a power outage elsewhere on the
grid.

John


On Tue, 14 Mar 2023 at 11:48, Matt Perkins <matt at spectrum.com.au> wrote:

> The questions left to be answered for Vocus is why did it take 12 hours to
> restore one rail. Where was the static bypass and failing that where was
> the maintenance bypass.  Im guessing that's what it's on maintenance bypass
> now and the slow action may have something to do with the public holiday in
> Vic..
>
>
> I shouldn't speculate. But it is fun.
>
>
>  Something seriously let go in that UPS. If i had to put my electrical
> engineers hat on I would say their load capacitor maintenance was not done
> and let go. Or someone was working upstream and broke the natural/earth
> bond. But there's a lot of parts in a system like that , most modern DC UPS
> setups are fairly robust.  I once had 3 strings of batteries catch fire and
> melt down and the UPS didn't miss a beat other then spewing a lot of
> alarms. We moved strings around on the fly and nobody even noticed.
>
>
> The other things that seems odd is why did an A failure trip B.  Sounds
> like a single supply is at surge/break capacity. Also a worry.
>
>
> But in the end we all have redundancy around 530 Collins right ? Right ?
>
>
> Matt
>
>
>
> On 14/3/2023 11:50 am, Christopher Hawker wrote:
>
> SY1 apparently experienced a full power outage back in July 2009,
> according to AusNOG Archives. Some reported full outages, some partial. Was
> not a DC fault though, looks like a much larger grid issue where a fault at
> the Bayswater Power Station in the Hunter Valley (which affected generators
> in QLD and VIC) caused about 1000MW to be dropped from the grid, resulting
> in rolling blackouts across QLD, NSW, TAS, SA and VIC.
>
> https://lists.ausnog.net/pipermail/ausnog/2009-July/003222.html
>
> CH.
> ------------------------------
> *From:* AusNOG <ausnog-bounces at lists.ausnog.net>
> <ausnog-bounces at lists.ausnog.net> on behalf of Nathan Brookfield
> <Nathan.Brookfield at iperium.com.au> <Nathan.Brookfield at iperium.com.au>
> *Sent:* Tuesday, March 14, 2023 11:40 AM
> *To:* Matt Perkins <matt at spectrum.com.au> <matt at spectrum.com.au>;
> ausnog at lists.ausnog.net <ausnog at lists.ausnog.net>
> <ausnog at lists.ausnog.net>
> *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>
>
> Have 100% had A and B fail separately at SY1 but never both, thankfully!!!
>
>
>
> Could have been part of the facility though so don’t quote me.
>
>
>
> UK on the other hand, fail!
>
>
>
> *From:* Matt Perkins <matt at spectrum.com.au> <matt at spectrum.com.au>
> *Sent:* Tuesday, 14 March 2023 11:30 AM
> *To:* Nathan Brookfield <Nathan.Brookfield at iperium.com.au>
> <Nathan.Brookfield at iperium.com.au>; ausnog at lists.ausnog.net
> *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>
>
>
> SY1 had A or B fail a few years back if my memory serves me correctly.
> But I was referring  London and Manchester which both had total outages in
> recent years.
>
> Matt.
>
>
>
> On 14/3/2023 11:26 am, Nathan Brookfield wrote:
>
> I’ve never had an A+B power rail failure at Equinix in 18 years I’ve been
> in there facilities, TOUCH WOOD!!!
>
>
>
>
> *Nathan Brookfield | VK2NAB *General Manager
>
> *p*: 1300 592 330 | *m*: 0412 266 008
> *e*: Nathan.brookfield at iperium.com.au | *w*: iperium.com.au
>
> Suite 4.02, 189 Kent Street Sydney NSW 2000
>
>
> *Your Telco Team*
>
> DISCLAIMER: This document is intended solely for the named addressee. This
> electronic communication, which includes any files or attachments thereto,
> contains proprietary or confidential information and may be privileged and
> otherwise protected under copyright or other applicable intellectual
> property laws. The use, disclosure, copying or distribution of any of the
> information contained in this document, by any person other than the
> addressee, is strictly prohibited. If you received this document in error,
> please contact the sender immediately and delete all the material from any
> computer. Confidentiality and legal privilege are not waived or lost by
> reason of mistaken delivery to you. Any views or opinions presented are
> solely those of the author and do not necessarily represent those of
> Iperium.
>
> WARNING: Computer viruses can be transmitted via email. The recipient
> should check this email and any attachments for the presence of viruses.
> Iperium accepts no liability for any damage caused by any virus transmitted
> by this email.
>
> *From:* AusNOG <ausnog-bounces at lists.ausnog.net>
> <ausnog-bounces at lists.ausnog.net> *On Behalf Of *Matt Perkins
> *Sent:* Tuesday, 14 March 2023 11:18 AM
> *To:* ausnog at lists.ausnog.net
> *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>
>
>
> Yea because Eqinix has never had a power outage. :)
>
> I think it's a little premature to judge at the moment. Doesn't sound good
> based on the current info but we have all had bad days im sure.
>
> Matt
>
>
>
> On 14/3/2023 11:06 am, Bradley Amm wrote:
>
> Friends like let friends put gear in a Vocus dtacenter. Only NextDC or
> Equinix :)
>
> Sent from my iPhone
>
>
>
>
> On 14 Mar 2023, at 7:08 am, DaZZa <dazzagibbs at gmail.com>
> <dazzagibbs at gmail.com> wrote:
>
> 
>
> Doesn't say a whole lot about Vocus' data center design methodology, nor
> their periodic contingency testing.
>
>
>
> They're apparently replacing the entire UPS on Friday - guess we can all
> expect more outages then when they stuff that up.
>
>
>
> D
>
>
>
> On Mon, 13 Mar 2023 at 18:30, Chad Kelly <chad at cpkws.com.au> wrote:
>
> Thought I would create a new thread on this, as the issue wasn’t related
> to AussieBB.
>
> It is looking like this Melbourne outage is similar to the Sydney outage
> back in 2018, in that it appears to have been yet again caused by an issue
> with their UPS systems yet again.
>
> I’ll leave this hear
> https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/
> and this article on today’s outage as well.
>
>
> https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987
>
>
>
> Regards Chad.
>
>
>
>
>
>
>
> Chad Kelly
>
> Manager
>
> CPK Web Services
>
> Phone 03 52730246
>
> Web https://www.cpkws.com.au
>
> <image001.jpg>
>
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
> --
>
> veg·e·tar·i·an:
> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
>
> _______________________________________________
>
> AusNOG mailing list
>
> AusNOG at lists.ausnog.net
>
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
> --
>
> ** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL
>
>
>
> /* Matt Perkins
>
>         Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.
>
>         Office 1300 133 299     matt at spectrum.com.au
>
>         ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000
>
>
>
> */
>
> --
>
> ** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL
>
>
>
> /* Matt Perkins
>
>         Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.
>
>         Office 1300 133 299     matt at spectrum.com.au
>
>         ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000
>
>
>
> */
>
> --
> ** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL
>
> /* Matt Perkins
>         Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.
>         Office 1300 133 299     matt at spectrum.com.au
>         ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000
>
> */
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230314/ed80a3f2/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 4595 bytes
Desc: not available
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230314/ed80a3f2/attachment.png>

More information about the AusNOG mailing list