[AusNOG] Aussie Broadband Outage
James Braunegg
james.braunegg at micron21.com
Mon Mar 13 10:52:15 AEDT 2023
Dear All
Anyone who wants the official Vocus incident number it is INC30876585.
Kindest Regards
James Braunegg
[cid:image001.png at 01D95599.DE4AEDB0]
1300 769 972<tel:1300%20769%20972> / 0488 997 207<tel:1300%20769%20972>
james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>
www.ddosprotection.com.au <https://www.ddosprotection.com.au/>
[cid:image002.png at 01D95599.DE4AEDB0]<http://www.micron21.com/>
[cid:image003.png at 01D95599.DE4AEDB0]<https://www.facebook.com/micron21/>
[cid:image004.png at 01D95599.DE4AEDB0]<https://twitter.com/micron21>
Follow us on m21status.com<https://m21status.com> for important service and system updates.
This message is intended for the addressee named above. It may contain privileged or confidential information. If you are not the intended recipient of this message you must not use, copy, distribute or disclose it to anyone other than the addressee. If you have received this message in error please return the message to the sender by replying to it and then delete the message from your computer.
From: Chris Wright <chris at securityshift.com>
Sent: Monday, 13 March 2023 10:47 AM
To: Luke Thompson <luke.t at tncrew.com.au>; James Braunegg <james.braunegg at micron21.com>; Damien Gardner Jnr <rendrag at rendrag.net>
Cc: ausnog at ausnog.net <ausnog at lists.ausnog.net>
Subject: RE: [AusNOG] Aussie Broadband Outage
Looks like somethings in vocus own network might be "singe power equipment" :(
traceroute to 91.204.80.53 (91.204.80.53), 30 hops max, 60 byte packets
1 144.48.37.193 (144.48.37.193) 0.654 ms 0.727 ms 0.781 ms
2 144.48.37.45 (144.48.37.45) 0.797 ms 0.976 ms 1.089 ms
3 core-a.me1.vic.hostuniversal.com.au (103.216.222.1) 0.158 ms 0.183 ms 0.216 ms
4 4826.mel.equinix.com (183.177.61.1) 0.785 ms 0.955 ms 1.193 ms
5 be150.cor02.mel11.vic.vocus.network (114.31.196.56) 0.632 ms 0.901 ms 0.753 ms
6 be151.bdr02.mel11.vic.vocus.network (114.31.196.53) 1.367 ms be150.bdr02.mel11.vic.vocus.network (114.31.196.55) 1.281 ms be151.bdr02.mel11.vic.vocus.network (114.31.196.53) 1.501 ms
7 be150.cor02.mel07.vic.vocus.network (114.31.196.54) 0.612 ms be151.cor02.mel11.vic.vocus.network (114.31.196.52) 0.952 ms be150.cor02.mel07.vic.vocus.network (114.31.196.54) 0.657 ms
8 be150.bdr03.mel07.vic.vocus.network (114.31.196.57) 1.196 ms 1.407 ms 1.579 ms
9 be150.cor02.mel11.vic.vocus.network (114.31.196.56) 1.036 ms 1.064 ms be151.cor02.mel07.vic.vocus.network (114.31.196.50) 1.120 ms
10 be150.bdr02.mel11.vic.vocus.network (114.31.196.55) 1.095 ms 1.419 ms be151.bdr02.mel11.vic.vocus.network (114.31.196.53) 1.505 ms
11 be151.cor02.mel11.vic.vocus.network (114.31.196.52) 0.883 ms be150.cor02.mel07.vic.vocus.network (114.31.196.54) 0.847 ms be151.cor02.mel11.vic.vocus.network (114.31.196.52) 1.040 ms
12 be151.bdr03.mel07.vic.vocus.network (114.31.196.51) 0.829 ms be150.bdr03.mel07.vic.vocus.network (114.31.196.57) 1.769 ms 1.602 ms
13 be150.cor02.mel11.vic.vocus.network (114.31.196.56) 1.225 ms be151.cor02.mel07.vic.vocus.network (114.31.196.50) 0.976 ms 1.007 ms
14 be150.bdr02.mel11.vic.vocus.network (114.31.196.55) 0.971 ms be151.bdr02.mel11.vic.vocus.network (114.31.196.53) 1.396 ms 1.460 ms
15 be151.cor02.mel11.vic.vocus.network (114.31.196.52) 1.208 ms 1.245 ms be150.cor02.mel07.vic.vocus.network (114.31.196.54) 1.222 ms
16 be150.bdr03.mel07.vic.vocus.network (114.31.196.57) 1.808 ms be151.bdr03.mel07.vic.vocus.network (114.31.196.51) 4.102 ms be150.bdr03.mel07.vic.vocus.network (114.31.196.57) 1.969 ms
17 be151.cor02.mel07.vic.vocus.network (114.31.196.50) 1.272 ms be150.cor02.mel11.vic.vocus.network (114.31.196.56) 1.318 ms 1.298 ms
18 be150.bdr02.mel11.vic.vocus.network (114.31.196.55) 1.223 ms be151.bdr02.mel11.vic.vocus.network (114.31.196.53) 1.299 ms 1.288 ms
19 be151.cor02.mel11.vic.vocus.network (114.31.196.52) 1.133 ms 1.144 ms 1.288 ms
20 be150.bdr03.mel07.vic.vocus.network (114.31.196.57) 2.011 ms 5.261 ms 5.254 ms
21 be150.cor02.mel11.vic.vocus.network (114.31.196.56) 1.326 ms be151.cor02.mel07.vic.vocus.network (114.31.196.50) 1.503 ms 1.529 ms
22 be150.bdr02.mel11.vic.vocus.network (114.31.196.55) 1.400 ms 1.296 ms be151.bdr02.mel11.vic.vocus.network (114.31.196.53) 1.490 ms
23 be150.cor02.mel07.vic.vocus.network (114.31.196.54) 1.615 ms 1.454 ms 1.629 ms
24 be150.bdr03.mel07.vic.vocus.network (114.31.196.57) 1.885 ms be151.bdr03.mel07.vic.vocus.network (114.31.196.51) 1.293 ms 1.377 ms
25 be151.cor02.mel07.vic.vocus.network (114.31.196.50) 1.364 ms be150.cor02.mel11.vic.vocus.network (114.31.196.56) 1.625 ms be151.cor02.mel07.vic.vocus.network (114.31.196.50) 1.792 ms
26 be151.bdr02.mel11.vic.vocus.network (114.31.196.53) 1.685 ms be150.bdr02.mel11.vic.vocus.network (114.31.196.55) 1.644 ms be151.bdr02.mel11.vic.vocus.network (114.31.196.53) 1.437 ms
27 be150.cor02.mel07.vic.vocus.network (114.31.196.54) 1.654 ms * *
28 be151.bdr03.mel07.vic.vocus.network (114.31.196.51) 1.342 ms 4.852 ms be150.bdr03.mel07.vic.vocus.network (114.31.196.57) 1.659 ms
29 be151.cor02.mel07.vic.vocus.network (114.31.196.50) 1.629 ms 1.639 ms be150.cor02.mel11.vic.vocus.network (114.31.196.56) 1.459 ms
30 be151.bdr02.mel11.vic.vocus.network (114.31.196.53) 1.780 ms 1.867 ms 1.553 ms
Thanks
c.
From: Luke Thompson <luke.t at tncrew.com.au<mailto:luke.t at tncrew.com.au>>
Sent: Monday, 13 March 2023 10:43
To: James Braunegg <james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>>; Damien Gardner Jnr <rendrag at rendrag.net<mailto:rendrag at rendrag.net>>; Chris Wright <chris at securityshift.com<mailto:chris at securityshift.com>>
Cc: ausnog at ausnog.net<mailto:ausnog at ausnog.net> <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: Re: [AusNOG] Aussie Broadband Outage
This email came from an external source. Please be cautious with links and content.
Reminds me of Vocus Sydney way back when. Doesn't sound brilliantly engineered - very curious for the explanation down the track to see if it could've been avoided.
Re: single feed, transfer switches have their place for single corded kit, have no doubt!
Cheers,
Luke Thompson
Chief Technical Officer
The Network Crew Pty Ltd
https://thenetworkcrew.com.au
On 13 March 2023 10:39:58 am James Braunegg <james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>> wrote:
Dear Luke
This is more than likely related to a power issue at 530 Collins Street we lost at 8:45am this morning supply A and then around 8:55am we also lost supply B for all services running at this location.
Power retuned around 9:45am for supply B however supply A was still dead... then at 10:20am both supply A and B both went down again....
Currently it looks like supply B is back online as of 10:33am... but supply A is still offline so anyone using single power equipment on supply A only will be having issues !
I have raised a ticket with the vendor however no reply as of yet.... Will have to wait for the PIR to hear what happened.
Kindest Regards
James Braunegg
1300 769 972<tel:1300%20769%20972> / 0488 997 207<tel:1300%20769%20972>
james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>
www.ddosprotection.com.au <https://www.ddosprotection.com.au/>
Follow us on m21status.com for important service and system updates.<https://www.facebook.com/micron21/>
This message is intended for the addressee named above. It may contain privileged or confidential information. If you are not the intended recipient of this message you must not use, copy, distribute or disclose it to anyone other than the addressee. If you have received this message in error please return the message to the sender by replying to it and then delete the message from your computer.<https://www.facebook.com/micron21/>
<https://www.facebook.com/micron21/>
<https://www.facebook.com/micron21/>
________________________________
From: AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Luke Thompson <luke.t at tncrew.com.au>
Sent: Monday, 13 March 2023 10:37:03 AM
To: Damien Gardner Jnr; Chris Wright
Cc: ausnog at ausnog.net
Subject: Re: [AusNOG] Aussie Broadband Outage <https://www.facebook.com/micron21/>
<https://www.facebook.com/micron21/>
Given the commonality of outage type, it's smelling like civil or grid problems isn't it?<https://www.facebook.com/micron21/>
Vocus and Equinix losing power over both rails, optical links dropping, etc sounds like a major problem in the earth or at a dist/sub station. Curious to learn the root cause. <https://www.facebook.com/micron21/>
A team somewhere is having a bad Monday morning, that's for sure. Good luck to all. <https://www.facebook.com/micron21/>
Luke Thompson<https://www.facebook.com/micron21/>
Chief Technical Officer<https://www.facebook.com/micron21/>
The Network Crew Pty Ltd<https://www.facebook.com/micron21/>
https://thenetworkcrew.com.au<https://www.facebook.com/micron21/>
On 13 March 2023 10:31:35 am Damien Gardner Jnr <rendrag at rendrag.net> wrote:<https://www.facebook.com/micron21/>
I would assume it's to do with VDC-MEL03 having lost power around 9:05am. A few folks reported getting one rail back temporarily, and then lost power again. We've not seen power restore at all so far.<https://www.facebook.com/micron21/>
-DG<https://www.facebook.com/micron21/>
On Mon, 13 Mar 2023 at 10:28 am, Chris Wright <chris at securityshift.com> wrote:<https://www.facebook.com/micron21/>
I also had outages in Melbourne, (Equinix ME2) around the same time (~9:00 - > ~9:45am).
Still got a link down with Vocus and still having links go up and down with Swoop.
Something more significant is happening somewhere - would be great if anyone can shed some light on what's happening so we all don't keep chasing our tails.
Thanks
c.
-----Original Message-----
From: AusNOG <ausnog-bounces at lists.ausnog.net> On Behalf Of DaZZa
Sent: Monday, 13 March 2023 10:21
To: Luke Thompson <luke.t at tncrew.com.au>
Cc: ausnog at ausnog.net <ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] Aussie Broadband Outage
[You don't often get email from dazzagibbs at gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
This email came from an external source. Please be cautious with links and content.
Interesting.
I believe Aussie is based in Melbourne - I had a major outage on several Vocus services in Melbourne between about 09:10 and 09:40 this morning.
Wonder if they're in a shared DC and took a hit at the same time as Aussie did...
D
On Mon, 13 Mar 2023 at 10:10, Luke Thompson <luke.t at tncrew.com.au> wrote:
>
> James on Whirlpool has just shared what you likely saw tweeted before:
>
> "We are currently experiencing issues with one of our data centres, which is impacting some of our services including MyAussie, VoIP, and our Customer Service phone lines / Live Chat"
>
> Sounds like there's no redundancy for Carbon, My Aussie, VoIP, Chat, etc. Wild.
>
> Luke Thompson
> Chief Technical Officer
>
> The Network Crew Pty Ltd
> https://thenetworkcrew.com.au
>
> On 13 March 2023 10:07:57 am Troy <troy at troykelly.com> wrote:
>>
>> Is anybody from Aussie able to share what's going on?
>>
>> Looks like there's a major outage with services (no Carbon, phones,
>> etc)
>>
>> There was? a tweet for about 40 seconds saying that there was an issue with a data centre, but it was deleted.
>>
>> Regards, Troy
>> Brevity is the elixir of life.
>> Father Hector McGrath, Pixie 2020
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
--
veg*e*tar*i*an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride _______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog
_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog<https://www.facebook.com/micron21/>
-- <https://www.facebook.com/micron21/>
Damien Gardner Jnr
VK2TDG. Dip EE. GradIEAust
rendrag at rendrag.net - http://www.rendrag.net/
--
We rode on the winds of the rising storm,
We ran to the sounds of thunder.
We danced among the lightning bolts,
and tore the world asunder<https://www.facebook.com/micron21/>
_______________________________________________<https://www.facebook.com/micron21/>
AusNOG mailing list<https://www.facebook.com/micron21/>
AusNOG at lists.ausnog.net<https://www.facebook.com/micron21/>
https://lists.ausnog.net/mailman/listinfo/ausnog<https://www.facebook.com/micron21/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230312/65c3bf58/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 1047 bytes
Desc: image001.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230312/65c3bf58/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 4219 bytes
Desc: image002.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230312/65c3bf58/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 1137 bytes
Desc: image003.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230312/65c3bf58/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 1246 bytes
Desc: image004.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230312/65c3bf58/attachment-0007.png>
More information about the AusNOG
mailing list