[AusNOG] Aussie Broadband Outage

Mon Mar 13 10:42:49 AEDT 2023

Reminds me of Vocus Sydney way back when. Doesn't sound brilliantly 
engineered - very curious for the explanation down the track to see if it 
could've been avoided.

Re: single feed, transfer switches have their place for single corded kit, 
have no doubt!

Cheers,

Luke Thompson
Chief Technical Officer

The Network Crew Pty Ltd
https://thenetworkcrew.com.au
On 13 March 2023 10:39:58 am James Braunegg <james.braunegg at micron21.com> 
wrote:
> Dear Luke
>
>
>
> This is more than likely related to a power issue at 530 Collins Street we 
> lost at 8:45am this morning supply A and then around 8:55am we also lost 
> supply B for all services running at this location.
>
>
>
> Power retuned around 9:45am for supply B however supply A was still dead… 
> then at 10:20am both supply A and B both went down again….
>
>
>
> Currently it looks like supply B is back online as of 10:33am… but supply A 
> is still offline so anyone using single power equipment on supply A only 
> will be having issues !
>
>
>
> I have raised a ticket with the vendor however no reply as of yet…. Will 
> have to wait for the PIR to hear what happened.
>
>
>
>
> Kindest Regards
>
>
>
>
>
>
>
> James Braunegg
>
>
>
> 	
>
> 	
>
> 1300 769 972 / 0488 997 207
>
>
>
>
> james.braunegg at micron21.com
>
>
>
>
> www.ddosprotection.com.au
>
> 	
>
>
>
>
> 	
>
> 	
>
> Follow us on m21status.com for important service and system updates.
>
>
>
>
> This message is intended for the addressee named above. It may contain 
> privileged or confidential information. If you are not the intended 
> recipient of this message you must not use, copy, distribute or disclose it 
> to anyone other than the addressee. If you have received this message in 
> error please return the message to the sender by replying to it and then 
> delete the message from your computer.
>
>
> From: AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Luke Thompson 
> <luke.t at tncrew.com.au>
> Sent: Monday, 13 March 2023 10:37:03 AM
> To: Damien Gardner Jnr; Chris Wright
> Cc: ausnog at ausnog.net
> Subject: Re: [AusNOG] Aussie Broadband Outage
> Given the commonality of outage type, it's smelling like civil or grid 
> problems isn't it?
>
>
> Vocus and Equinix losing power over both rails, optical links dropping, etc 
> sounds like a major problem in the earth or at a dist/sub station. Curious 
> to learn the root cause.
>
>
> A team somewhere is having a bad Monday morning, that's for sure. Good luck 
> to all.
>
> Luke Thompson
> Chief Technical Officer
>
>
> The Network Crew Pty Ltd
> https://thenetworkcrew.com.au
>
> On 13 March 2023 10:31:35 am Damien Gardner Jnr <rendrag at rendrag.net> wrote:
>> I would assume it’s to do with VDC-MEL03 having lost power around 9:05am.  
>> A few folks reported getting one rail back temporarily, and then lost power 
>> again.  We’ve not seen power restore at all so far.
>>
>>
>> —DG
>>
>>
>> On Mon, 13 Mar 2023 at 10:28 am, Chris Wright <chris at securityshift.com> wrote:
>>
>> I also had outages in Melbourne, (Equinix ME2) around the same time (~9:00 
>> - > ~9:45am).
>>
>> Still got a link down with Vocus and still having links go up and down with 
>> Swoop.
>>
>> Something more significant is happening somewhere - would be great if 
>> anyone can shed some light on what's happening so we all don't keep chasing 
>> our tails.
>>
>> Thanks
>>
>> c.
>>
>> -----Original Message-----
>> From: AusNOG <ausnog-bounces at lists.ausnog.net> On Behalf Of DaZZa
>> Sent: Monday, 13 March 2023 10:21
>> To: Luke Thompson <luke.t at tncrew.com.au>
>> Cc: ausnog at ausnog.net <ausnog at lists.ausnog.net>
>> Subject: Re: [AusNOG] Aussie Broadband Outage
>>
>> [You don't often get email from dazzagibbs at gmail.com. Learn why this is 
>> important at https://aka.ms/LearnAboutSenderIdentification ]
>>
>> This email came from an external source. Please be cautious with links and 
>> content.
>>
>> Interesting.
>>
>> I believe Aussie is based in Melbourne - I had a major outage on several 
>> Vocus services in Melbourne between about 09:10 and 09:40 this morning.
>>
>> Wonder if they're in a shared DC and took a hit at the same time as Aussie 
>> did...
>>
>> D
>>
>> On Mon, 13 Mar 2023 at 10:10, Luke Thompson <luke.t at tncrew.com.au> wrote:
>>>
>>> James on Whirlpool has just shared what you likely saw tweeted before:
>>>
>>> "We are currently experiencing issues with one of our data centres, which 
>>> is impacting some of our services including MyAussie, VoIP, and our 
>>> Customer Service phone lines / Live Chat"
>>>
>>> Sounds like there's no redundancy for Carbon, My Aussie, VoIP, Chat, etc. Wild.
>>>
>>> Luke Thompson
>>> Chief Technical Officer
>>>
>>> The Network Crew Pty Ltd
>>> https://thenetworkcrew.com.au
>>>
>>> On 13 March 2023 10:07:57 am Troy <troy at troykelly.com> wrote:
>>>>
>>>> Is anybody from Aussie able to share what's going on?
>>>>
>>>> Looks like there's a major outage with services (no Carbon, phones,
>>>> etc)
>>>>
>>>> There was? a tweet for about 40 seconds saying that there was an issue with 
>>>> a data centre, but it was deleted.
>>>>
>>>> Regards, Troy
>>>> Brevity is the elixir of life.
>>>> Father Hector McGrath, Pixie 2020
>>>
>>>
>>> _______________________________________________
>>> AusNOG mailing list
>>> AusNOG at lists.ausnog.net
>>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
>>
>>
>> --
>> veg*e*tar*i*an:
>> Ancient tribal slang for the village idiot who can't hunt, fish or ride 
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
>> --
>>
>>
>> Damien Gardner Jnr
>> VK2TDG. Dip EE. GradIEAust
>> rendrag at rendrag.net -  http://www.rendrag.net/
>> --
>> We rode on the winds of the rising storm,
>> We ran to the sounds of thunder.
>> We danced among the lightning bolts,
>> and tore the world asunder
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230313/29180f59/attachment.htm>