From xrobau at gmail.com  Wed Mar  1 09:25:41 2023
From: xrobau at gmail.com (Rob Thomas)
Date: Wed, 1 Mar 2023 08:25:41 +1000
Subject: [AusNOG] Reminder to vote in the APNIC Elections!
Message-ID: <CAC5zRjPQJDP4eP4wzpqki1vfjkhd-jDqgWybTNCyAiD4MnmUnw@mail.gmail.com>

As the time is getting close to the end of the voting period, I thought it
would be worthwhile to remind everyone to go and submit their votes!

I've done my best to keep quiet on the potential issues in this election,
apart from my official statement  (See my nomination page
https://2023.apricot.net/elections/nominations/robthomas/ for information -
Note that closed captions are available on the video for those with hearing
impairments), but I think the other posts have explained it well enough.

So please, take the time to log into your my.apnic account, cast your
votes, and work towards making APNIC work for us!

For those who don't have a spare Round Tuit, and don't want to be bothered
doing it yourself, feel free to add 'rob at qldvoip.com.au' as a contact and
ONLY assign me 'Voting' rights. However, I really REALLY encourage you to
look into the candidates yourself.

--Rob
+61-402-077-155
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230301/2bc5200a/attachment.htm>

From John.Mah at ibm.com  Fri Mar  3 15:26:30 2023
From: John.Mah at ibm.com (John Mah)
Date: Fri, 3 Mar 2023 04:26:30 +0000
Subject: [AusNOG] AusNOG 2023 Events
Message-ID: <DM6PR15MB40061DEA58DD7C4FBEEB789A9AB39@DM6PR15MB4006.namprd15.prod.outlook.com>

Dear Sir/Ms,

                I am interested to participate and speak this upcoming event for 2023.
                Please advise how and when I can submit for paper.
                Looking forward hearing from you soon.

Thanks.


--
Regards,
John Mah
e: John.Mah at ibm.com<mailto:John.Mah at ibm.com>
m: +6012-3004118
IBM Software Networking - SevOne
[signature_216140385]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230303/e0c7e25b/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 16465 bytes
Desc: image001.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230303/e0c7e25b/attachment-0001.png>

From xrobau at gmail.com  Sun Mar  5 12:54:00 2023
From: xrobau at gmail.com (Rob Thomas)
Date: Sun, 5 Mar 2023 11:54:00 +1000
Subject: [AusNOG] Everything Open Conf AusNog hallway track?
Message-ID: <CAC5zRjMtXdRYNDt6Hsg_bHcXwkFtx53xumztjTmdgRei_Gxngg@mail.gmail.com>

I'm going to be in Melbourne for Everything Open next week, and if anyone
else is planning to come please email me, and we'll do an unofficial ausnog
hallway track!

I'm also staying at the Casino next door, and you will know if you're near
me by all the 'xrobau' SSIDs you'll see 8)

--Rob

(For those that missed it, EverythingOpen is replacing LCA for this year,
and is in Melbourne - https://2023.everythingopen.au/ - but is basically
the same group of people)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230305/d14c5580/attachment.htm>

From bevan at slattery.net.au  Mon Mar  6 12:15:29 2023
From: bevan at slattery.net.au (Bevan Slattery)
Date: Mon, 6 Mar 2023 01:15:29 +0000
Subject: [AusNOG] Everything Open Conf AusNog hallway track?
In-Reply-To: <CAC5zRjMtXdRYNDt6Hsg_bHcXwkFtx53xumztjTmdgRei_Gxngg@mail.gmail.com>
References: <CAC5zRjMtXdRYNDt6Hsg_bHcXwkFtx53xumztjTmdgRei_Gxngg@mail.gmail.com>
Message-ID: <SYBP282MB4215C3D173B18B3DEE615B34FFB69@SYBP282MB4215.AUSP282.PROD.OUTLOOK.COM>

If you find yourself staggering the hallways at the Casino and looking for the SSID ?xrobau? then surely you?d be saying to yourself ?It must be 3am and I feel lonely.?
________________________________
From: AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Rob Thomas <xrobau at gmail.com>
Sent: Sunday, March 5, 2023 5:55 am
To: <ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
Subject: [AusNOG] Everything Open Conf AusNog hallway track?

I'm going to be in Melbourne for Everything Open next week, and if anyone else is planning to come please email me, and we'll do an unofficial ausnog hallway track!

I'm also staying at the Casino next door, and you will know if you're near me by all the 'xrobau' SSIDs you'll see 8)

--Rob

(For those that missed it, EverythingOpen is replacing LCA for this year, and is in Melbourne - https://2023.everythingopen.au/ - but is basically the same group of people)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230306/fdd3a62d/attachment.htm>

From ash at url.net.au  Wed Mar  8 13:53:16 2023
From: ash at url.net.au (Ashley Breeden)
Date: Wed, 8 Mar 2023 13:53:16 +1100
Subject: [AusNOG] Kayo, Binge Incorrect GEO IP Blocking
Message-ID: <CAAtQq0D0DWXBmCtUHwFR8xbHuMwRmEAh0T=B1oMSyLytZeezUA@mail.gmail.com>

Afternoon all,

We are having an issue with one of our IP blocks being blocked by Kayo and
Binge with the following:

Access Denied
You don't have permission to access "http://auth.streamotion.com.au/" on
this server.
Reference #18.4df00117.1678240837.d903c0

In the past, the front-of-house support has been extremely helpful and
resolved a GEO IP issue very quickly.  Unfortunately, this is no longer the
case.

If anyone is on the list from Streammotion/Foxtel or knows someone directly
can you please reach out to me so we can resolve this issue.

Regards,

Ashley Breeden

URL Networks Pty Ltd
Email: ash at url.net.au
WWW: http://www.url.net.au
Phone: 1300 33 11 78 | +61 3 9008 5900
Twitter: @urlnetworks <https://twitter.com/urlnetworks>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230308/bb9203ff/attachment.htm>

From luke at iggleden.com  Wed Mar  8 14:21:44 2023
From: luke at iggleden.com (Luke Iggleden)
Date: Wed, 8 Mar 2023 14:21:44 +1100
Subject: [AusNOG] Kayo, Binge Incorrect GEO IP Blocking
In-Reply-To: <CAAtQq0D0DWXBmCtUHwFR8xbHuMwRmEAh0T=B1oMSyLytZeezUA@mail.gmail.com>
References: <CAAtQq0D0DWXBmCtUHwFR8xbHuMwRmEAh0T=B1oMSyLytZeezUA@mail.gmail.com>
Message-ID: <3a2d36af-d075-1737-9778-48422b703290@iggleden.com>

Hi All,

I also reached out to them last week and had no response.

If anyone is on list please contact me too!


Cheers,

Luke Iggleden



On 8/3/2023 1:53 pm, Ashley Breeden wrote:
> Afternoon all,
>
> We are having an issue with one of our IP blocks being blocked by Kayo 
> and Binge with the following:
>
> Access Denied
> You don't have permission to access "http://auth.streamotion.com.au/" 
> on this server.
> Reference #18.4df00117.1678240837.d903c0
>
> In the past, the front-of-house support has been extremely helpful and 
> resolved a GEO IP issue very quickly. Unfortunately, this is no longer 
> the case.
>
> If anyone is on the list from Streammotion/Foxtel or knows someone 
> directly can you please reach out to me so we can resolve this issue.
>
> Regards,
>
>
> Ashley Breeden
>
>
> URL Networks Pty Ltd
> Email: ash at url.net.au
> WWW: http://www.url.net.au
> Phone: 1300 33 11 78?|?+61 3 9008 5900
> Twitter: @urlnetworks <https://twitter.com/urlnetworks>
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230308/dc09fe4f/attachment.htm>

From pwilson at apnic.net  Wed Mar  8 14:43:39 2023
From: pwilson at apnic.net (Paul Wilson)
Date: Wed, 08 Mar 2023 13:43:39 +1000
Subject: [AusNOG] Kayo, Binge Incorrect GEO IP Blocking
In-Reply-To: <CAAtQq0D0DWXBmCtUHwFR8xbHuMwRmEAh0T=B1oMSyLytZeezUA@mail.gmail.com>
References: <CAAtQq0D0DWXBmCtUHwFR8xbHuMwRmEAh0T=B1oMSyLytZeezUA@mail.gmail.com>
Message-ID: <7BD15FC4-DF92-4D93-B351-614D5FFA0863@apnic.net>

Hi Ashley,

APNIC services folk have experience dealing with geolocation errors, 
through various providers, so I?ve pinged them on your behalf.

If you have more info, like the geoloc provider that?s being used, 
please could you share that?

All the best

Paul.

________________________________________________________________________
Paul Wilson, Director-General, APNIC                        dg at apnic.net
http://www.apnic.net                                            @apnicdg

On 8 Mar 2023, at 12:53, Ashley Breeden wrote:

> Afternoon all,
>
> We are having an issue with one of our IP blocks being blocked by Kayo 
> and
> Binge with the following:
>
> Access Denied
> You don't have permission to access "http://auth.streamotion.com.au/" 
> on
> this server.
> Reference #18.4df00117.1678240837.d903c0
>
> In the past, the front-of-house support has been extremely helpful and
> resolved a GEO IP issue very quickly.  Unfortunately, this is no 
> longer the
> case.
>
> If anyone is on the list from Streammotion/Foxtel or knows someone 
> directly
> can you please reach out to me so we can resolve this issue.
>
> Regards,
>
> Ashley Breeden
>
> URL Networks Pty Ltd
> Email: ash at url.net.au
> WWW: http://www.url.net.au
> Phone: 1300 33 11 78 | +61 3 9008 5900
> Twitter: @urlnetworks <https://twitter.com/urlnetworks>

> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230308/da20b685/attachment.htm>

From vivek at apnic.net  Wed Mar  8 15:15:04 2023
From: vivek at apnic.net (Vivek Nigam)
Date: Wed, 8 Mar 2023 04:15:04 +0000
Subject: [AusNOG] Kayo, Binge Incorrect GEO IP Blocking
In-Reply-To: <7BD15FC4-DF92-4D93-B351-614D5FFA0863@apnic.net>
References: <CAAtQq0D0DWXBmCtUHwFR8xbHuMwRmEAh0T=B1oMSyLytZeezUA@mail.gmail.com>
 <7BD15FC4-DF92-4D93-B351-614D5FFA0863@apnic.net>
Message-ID: <SYBP282MB01863EAE9CEA2278A28C71E4B9B49@SYBP282MB0186.AUSP282.PROD.OUTLOOK.COM>

Hi Ashley, Luke,

You can try reaching out to casops.casops at foxtel.com.au<mailto:casops.casops at foxtel.com.au> to correct your IP Geolocation with Foxtel.

For more information on how APNIC can assist you with correcting your IP geolocation, please see

https://www.apnic.net/geolocation

Thanks
Vivek

From: AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Paul Wilson <pwilson at apnic.net>
Date: Wednesday, 8 March 2023 at 1:43 pm
To: Ashley Breeden <ash at url.net.au>
Cc: ausnog at ausnog.net <ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] Kayo, Binge Incorrect GEO IP Blocking

Hi Ashley,

APNIC services folk have experience dealing with geolocation errors, through various providers, so I?ve pinged them on your behalf.

If you have more info, like the geoloc provider that?s being used, please could you share that?

All the best

Paul.

________________________________

Paul Wilson, Director-General, APNIC dg at apnic.net<mailto:dg at apnic.net>
http://www.apnic.net @apnicdg

On 8 Mar 2023, at 12:53, Ashley Breeden wrote:
Afternoon all,

We are having an issue with one of our IP blocks being blocked by Kayo and Binge with the following:

Access Denied
You don't have permission to access "http://auth.streamotion.com.au/" on this server.
Reference #18.4df00117.1678240837.d903c0

In the past, the front-of-house support has been extremely helpful and resolved a GEO IP issue very quickly.  Unfortunately, this is no longer the case.

If anyone is on the list from Streammotion/Foxtel or knows someone directly can you please reach out to me so we can resolve this issue.


Regards,


Ashley Breeden


URL Networks Pty Ltd
Email: ash at url.net.au<mailto:ash at url.net.au>
WWW: http://www.url.net.au
Phone: 1300 33 11 78 | +61 3 9008 5900
Twitter: @urlnetworks<https://twitter.com/urlnetworks>

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net

https://lists.ausnog.net/mailman/listinfo/ausnog
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230308/c3b993cd/attachment.htm>

From markzzzsmith at gmail.com  Wed Mar  8 17:45:52 2023
From: markzzzsmith at gmail.com (Mark Smith)
Date: Wed, 8 Mar 2023 17:45:52 +1100
Subject: [AusNOG] Everything Open Conf AusNog hallway track?
In-Reply-To: <SYBP282MB4215C3D173B18B3DEE615B34FFB69@SYBP282MB4215.AUSP282.PROD.OUTLOOK.COM>
References: <CAC5zRjMtXdRYNDt6Hsg_bHcXwkFtx53xumztjTmdgRei_Gxngg@mail.gmail.com>
 <SYBP282MB4215C3D173B18B3DEE615B34FFB69@SYBP282MB4215.AUSP282.PROD.OUTLOOK.COM>
Message-ID: <CAO42Z2w509G7NsMa9CsoRZMM7x_hoZjfAEvnauenj85O_UuAMw@mail.gmail.com>

On Mon, 6 Mar 2023 at 12:16, Bevan Slattery <bevan at slattery.net.au> wrote:
>
> If you find yourself staggering the hallways at the Casino and looking for the SSID ?xrobau? then surely you?d be saying to yourself ?It must be 3am and I feel lonely.?

Or an invite to see how much quota Rob has on his mobile plan by
attaching to his wifi and downloading the whole Internet, or a game
update, whichever is larger.

> ________________________________
> From: AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Rob Thomas <xrobau at gmail.com>
> Sent: Sunday, March 5, 2023 5:55 am
> To: <ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
> Subject: [AusNOG] Everything Open Conf AusNog hallway track?
>
> I'm going to be in Melbourne for Everything Open next week, and if anyone else is planning to come please email me, and we'll do an unofficial ausnog hallway track!
>
> I'm also staying at the Casino next door, and you will know if you're near me by all the 'xrobau' SSIDs you'll see 8)
>
> --Rob
>
> (For those that missed it, EverythingOpen is replacing LCA for this year, and is in Melbourne - https://2023.everythingopen.au/ - but is basically the same group of people)
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog

From edepa at ieee.org  Wed Mar  8 19:46:47 2023
From: edepa at ieee.org (Etienne-Victor Depasquale)
Date: Wed, 8 Mar 2023 09:46:47 +0100
Subject: [AusNOG] Data on adoption of metro-area access technologies
Message-ID: <CAAcx0vBiy+r=oYNxPVud=aFX6ULo7wUCcGr=YrBNVkq198ZmUw@mail.gmail.com>

Good people,

I am seeking your evaluation of how well my data matches your perception.

Firstly, however, I beg pardon for cross-posting;
this thread appears in other operator mailing lists.
I am trying to reach as many operators as possible.

Please find here
<https://drive.google.com/open?id=1eZZeqbDaTj6nSJJDdK0iu5Xm9Z_691rR&authuser=etienne.depasquale%40um.edu.mt&usp=drive_fs>
a
snapshot of two datasets concerning access technologies in the metro area.
The bar chart on the left (not on the right, as inadvertently posted on
NANOG) summarizes data I collected last year from *NOGs;
the bar chart on the right summarizes data received through commissioned
market research, from Tier 1 and/or regional operators.

The y-axis shows cumulative responses for an option;
the x-axis shows monikers for the access technologies.

A second snapshot
<https://drive.google.com/open?id=1fMLiEGbvaD4pqCmQM75HWYoR37XaiHFQ&authuser=etienne.depasquale%40um.edu.mt&usp=drive_fs>
shows
the same data,
but this time with each response weighted by a conservative estimate of the
respondent's subscriber base across all geographical regions.
Here, the y-axis shows an estimate of the number of subscribers using the
access technology.


Thank you for taking the time to read this.

Sincerely,

-- 
Ing. Etienne-Victor Depasquale
Assistant Lecturer
Department of Communications & Computer Engineering
Faculty of Information & Communication Technology
University of Malta
Web. https://www.um.edu.mt/profile/etiennedepasquale
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230308/55b9313b/attachment.htm>

From spoofer-info at caida.org  Thu Mar  9 05:00:13 2023
From: spoofer-info at caida.org (CAIDA Spoofer Project)
Date: Wed, 8 Mar 2023 10:00:13 -0800
Subject: [AusNOG] Spoofer Report for AusNOG for Feb 2023
Message-ID: <1678298413.339525.21319.nullmailer@caida.org>

In response to feedback from operational security communities,
CAIDA's source address validation measurement project
(https://spoofer.caida.org) is automatically generating monthly
reports of ASes originating prefixes in BGP for systems from which
we received packets with a spoofed source address.
We are publishing these reports to network and security operations
lists in order to ensure this information reaches operational
contacts in these ASes.

This report summarises tests conducted within aus.

Inferred improvements during Feb 2023:
 none inferred

Source Address Validation issues inferred during Feb 2023:
ASN    Name                           First-Spoofed Last-Spoofed
24555  APRICOT-APNIC-ASN                 2017-02-26   2023-02-27
38195  SUPERLOOP                         2018-02-02   2023-02-22
4826   VOCUS-BACKBONE                    2018-03-07   2023-02-27
132458 PENTANET                          2022-12-08   2023-02-27

Further information for these tests where we received spoofed
packets is available at:
https://spoofer.caida.org/recent_tests.php?country_include=aus&no_block=1

Please send any feedback or suggestions to spoofer-info at caida.org

From troy at troykelly.com  Mon Mar 13 10:07:00 2023
From: troy at troykelly.com (Troy)
Date: Sun, 12 Mar 2023 23:07:00 +0000
Subject: [AusNOG] Aussie Broadband Outage
Message-ID: <A8510_r-U8zC150QdI98_t_99awf5jOvU3V_ePFzX1BbrF1KqGCMlu7byPLH9ZjiqOz0vHFDDc7Iw7JqxzUBjL_MoKqzyp5mRkf-NsjgO48=@troykelly.com>

Is anybody from Aussie able to share what's going on?

Looks like there's a major outage with services (no Carbon, phones, etc)

There was a tweet for about 40 seconds saying that there was an issue with a data centre, but it was deleted.

Regards, Troy
Brevity is the elixir of life.
Father Hector McGrath, Pixie 2020
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230312/34f0113c/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: publickey - troy at troykelly.com - 0x3DB9B3FB.asc
Type: application/pgp-keys
Size: 3109 bytes
Desc: not available
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230312/34f0113c/attachment.asc>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 855 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230312/34f0113c/attachment.sig>

From luke.t at tncrew.com.au  Mon Mar 13 10:10:41 2023
From: luke.t at tncrew.com.au (Luke Thompson)
Date: Mon, 13 Mar 2023 10:10:41 +1100
Subject: [AusNOG] Aussie Broadband Outage
In-Reply-To: <A8510_r-U8zC150QdI98_t_99awf5jOvU3V_ePFzX1BbrF1KqGCMlu7byPLH9ZjiqOz0vHFDDc7Iw7JqxzUBjL_MoKqzyp5mRkf-NsjgO48=@troykelly.com>
References: <A8510_r-U8zC150QdI98_t_99awf5jOvU3V_ePFzX1BbrF1KqGCMlu7byPLH9ZjiqOz0vHFDDc7Iw7JqxzUBjL_MoKqzyp5mRkf-NsjgO48=@troykelly.com>
Message-ID: <186d8172568.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>

James on Whirlpool has just shared what you likely saw tweeted before:

"We are currently experiencing issues with one of our data centres, which 
is impacting some of our services including MyAussie, VoIP, and our 
Customer Service phone lines / Live Chat"

Sounds like there's no redundancy for Carbon, My Aussie, VoIP, Chat, etc. Wild.

Luke Thompson
Chief Technical Officer

The Network Crew Pty Ltd
https://thenetworkcrew.com.au
On 13 March 2023 10:07:57 am Troy <troy at troykelly.com> wrote:
> Is anybody from Aussie able to share what's going on?
>
> Looks like there's a major outage with services (no Carbon, phones, etc)
>
> There was? a tweet for about 40 seconds saying that there was an issue with 
> a data centre, but it was deleted.
>
> Regards, Troy
> Brevity is the elixir of life.
> Father Hector McGrath, Pixie 2020

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230313/da0bcffa/attachment.htm>

From dazzagibbs at gmail.com  Mon Mar 13 10:21:13 2023
From: dazzagibbs at gmail.com (DaZZa)
Date: Mon, 13 Mar 2023 10:21:13 +1100
Subject: [AusNOG] Aussie Broadband Outage
In-Reply-To: <186d8172568.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
References: <A8510_r-U8zC150QdI98_t_99awf5jOvU3V_ePFzX1BbrF1KqGCMlu7byPLH9ZjiqOz0vHFDDc7Iw7JqxzUBjL_MoKqzyp5mRkf-NsjgO48=@troykelly.com>
 <186d8172568.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
Message-ID: <CAMpV8MqYi65O_MFdp+94NPR4uoUNVA5_0HYktBZ-DOwr+_47kw@mail.gmail.com>

Interesting.

I believe Aussie is based in Melbourne - I had a major outage on
several Vocus services in Melbourne between about 09:10 and 09:40 this
morning.

Wonder if they're in a shared DC and took a hit at the same time as
Aussie did...

D

On Mon, 13 Mar 2023 at 10:10, Luke Thompson <luke.t at tncrew.com.au> wrote:
>
> James on Whirlpool has just shared what you likely saw tweeted before:
>
> "We are currently experiencing issues with one of our data centres, which is impacting some of our services including MyAussie, VoIP, and our Customer Service phone lines / Live Chat"
>
> Sounds like there's no redundancy for Carbon, My Aussie, VoIP, Chat, etc. Wild.
>
> Luke Thompson
> Chief Technical Officer
>
> The Network Crew Pty Ltd
> https://thenetworkcrew.com.au
>
> On 13 March 2023 10:07:57 am Troy <troy at troykelly.com> wrote:
>>
>> Is anybody from Aussie able to share what's going on?
>>
>> Looks like there's a major outage with services (no Carbon, phones, etc)
>>
>> There was? a tweet for about 40 seconds saying that there was an issue with a data centre, but it was deleted.
>>
>> Regards, Troy
>> Brevity is the elixir of life.
>> Father Hector McGrath, Pixie 2020
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog



-- 
veg?e?tar?i?an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride

From chris at securityshift.com  Mon Mar 13 10:28:21 2023
From: chris at securityshift.com (Chris Wright)
Date: Sun, 12 Mar 2023 23:28:21 +0000
Subject: [AusNOG] Aussie Broadband Outage
In-Reply-To: <CAMpV8MqYi65O_MFdp+94NPR4uoUNVA5_0HYktBZ-DOwr+_47kw@mail.gmail.com>
References: <A8510_r-U8zC150QdI98_t_99awf5jOvU3V_ePFzX1BbrF1KqGCMlu7byPLH9ZjiqOz0vHFDDc7Iw7JqxzUBjL_MoKqzyp5mRkf-NsjgO48=@troykelly.com>
 <186d8172568.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
 <CAMpV8MqYi65O_MFdp+94NPR4uoUNVA5_0HYktBZ-DOwr+_47kw@mail.gmail.com>
Message-ID: <MEYPR01MB626395A30283BDAA1CCF0BC0A0B89@MEYPR01MB6263.ausprd01.prod.outlook.com>

I also had outages in Melbourne, (Equinix ME2) around the same time (~9:00 - > ~9:45am).

Still got a link down with Vocus and still having links go up and down with Swoop.

Something more significant is happening somewhere - would be great if anyone can shed some light on what's happening so we all don't keep chasing our tails.

Thanks

c.

-----Original Message-----
From: AusNOG <ausnog-bounces at lists.ausnog.net> On Behalf Of DaZZa
Sent: Monday, 13 March 2023 10:21
To: Luke Thompson <luke.t at tncrew.com.au>
Cc: ausnog at ausnog.net <ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] Aussie Broadband Outage

[You don't often get email from dazzagibbs at gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

This email came from an external source. Please be cautious with links and content.

Interesting.

I believe Aussie is based in Melbourne - I had a major outage on several Vocus services in Melbourne between about 09:10 and 09:40 this morning.

Wonder if they're in a shared DC and took a hit at the same time as Aussie did...

D

On Mon, 13 Mar 2023 at 10:10, Luke Thompson <luke.t at tncrew.com.au> wrote:
>
> James on Whirlpool has just shared what you likely saw tweeted before:
>
> "We are currently experiencing issues with one of our data centres, which is impacting some of our services including MyAussie, VoIP, and our Customer Service phone lines / Live Chat"
>
> Sounds like there's no redundancy for Carbon, My Aussie, VoIP, Chat, etc. Wild.
>
> Luke Thompson
> Chief Technical Officer
>
> The Network Crew Pty Ltd
> https://thenetworkcrew.com.au
>
> On 13 March 2023 10:07:57 am Troy <troy at troykelly.com> wrote:
>>
>> Is anybody from Aussie able to share what's going on?
>>
>> Looks like there's a major outage with services (no Carbon, phones, 
>> etc)
>>
>> There was? a tweet for about 40 seconds saying that there was an issue with a data centre, but it was deleted.
>>
>> Regards, Troy
>> Brevity is the elixir of life.
>> Father Hector McGrath, Pixie 2020
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog



--
veg*e*tar*i*an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride _______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

From chris at securityshift.com  Mon Mar 13 10:30:42 2023
From: chris at securityshift.com (Chris Wright)
Date: Sun, 12 Mar 2023 23:30:42 +0000
Subject: [AusNOG] Aussie Broadband Outage
In-Reply-To: <CAMpV8MqYi65O_MFdp+94NPR4uoUNVA5_0HYktBZ-DOwr+_47kw@mail.gmail.com>
References: <A8510_r-U8zC150QdI98_t_99awf5jOvU3V_ePFzX1BbrF1KqGCMlu7byPLH9ZjiqOz0vHFDDc7Iw7JqxzUBjL_MoKqzyp5mRkf-NsjgO48=@troykelly.com>
 <186d8172568.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
 <CAMpV8MqYi65O_MFdp+94NPR4uoUNVA5_0HYktBZ-DOwr+_47kw@mail.gmail.com>
Message-ID: <MEYPR01MB6263836694E6455D265168F2A0B89@MEYPR01MB6263.ausprd01.prod.outlook.com>

IAIX seem to be in the know, they have sent out two notifications now. First saying:


[OUTAGE] VIC-IX | VDC-MEL03/ MDC | Rack Power Loss

Incident status: Monitoring 

Power was partially restored ~9:24am (AEDT) which has brought member services back up. Pending full power re-establishment.

And one just now:	

[OUTAGE] VIC-IX | VDC-MEL03/ MDC | Rack Power Loss

Incident status: Identified 

Power dropped again to the site and is currently back up on partial power. Power drop may occur again. DC operator electricians working onsite to resolve.

So, looks like power issues at Vocus.

Thanks

c.


-----Original Message-----
From: AusNOG <ausnog-bounces at lists.ausnog.net> On Behalf Of DaZZa
Sent: Monday, 13 March 2023 10:21
To: Luke Thompson <luke.t at tncrew.com.au>
Cc: ausnog at ausnog.net <ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] Aussie Broadband Outage

[You don't often get email from dazzagibbs at gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

This email came from an external source. Please be cautious with links and content.

Interesting.

I believe Aussie is based in Melbourne - I had a major outage on several Vocus services in Melbourne between about 09:10 and 09:40 this morning.

Wonder if they're in a shared DC and took a hit at the same time as Aussie did...

D

On Mon, 13 Mar 2023 at 10:10, Luke Thompson <luke.t at tncrew.com.au> wrote:
>
> James on Whirlpool has just shared what you likely saw tweeted before:
>
> "We are currently experiencing issues with one of our data centres, which is impacting some of our services including MyAussie, VoIP, and our Customer Service phone lines / Live Chat"
>
> Sounds like there's no redundancy for Carbon, My Aussie, VoIP, Chat, etc. Wild.
>
> Luke Thompson
> Chief Technical Officer
>
> The Network Crew Pty Ltd
> https://thenetworkcrew.com.au
>
> On 13 March 2023 10:07:57 am Troy <troy at troykelly.com> wrote:
>>
>> Is anybody from Aussie able to share what's going on?
>>
>> Looks like there's a major outage with services (no Carbon, phones, 
>> etc)
>>
>> There was? a tweet for about 40 seconds saying that there was an issue with a data centre, but it was deleted.
>>
>> Regards, Troy
>> Brevity is the elixir of life.
>> Father Hector McGrath, Pixie 2020
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog



--
veg?e?tar?i?an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride _______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

From rendrag at rendrag.net  Mon Mar 13 10:31:16 2023
From: rendrag at rendrag.net (Damien Gardner Jnr)
Date: Mon, 13 Mar 2023 10:31:16 +1100
Subject: [AusNOG] Aussie Broadband Outage
In-Reply-To: <MEYPR01MB626395A30283BDAA1CCF0BC0A0B89@MEYPR01MB6263.ausprd01.prod.outlook.com>
References: <A8510_r-U8zC150QdI98_t_99awf5jOvU3V_ePFzX1BbrF1KqGCMlu7byPLH9ZjiqOz0vHFDDc7Iw7JqxzUBjL_MoKqzyp5mRkf-NsjgO48=@troykelly.com>
 <186d8172568.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
 <CAMpV8MqYi65O_MFdp+94NPR4uoUNVA5_0HYktBZ-DOwr+_47kw@mail.gmail.com>
 <MEYPR01MB626395A30283BDAA1CCF0BC0A0B89@MEYPR01MB6263.ausprd01.prod.outlook.com>
Message-ID: <CAPrfDLW0eVQ4FO=8a9p5Qczg_Cc4G6n5bhkp383Xs7kSeHZ8xg@mail.gmail.com>

I would assume it?s to do with VDC-MEL03 having lost power around 9:05am.
A few folks reported getting one rail back temporarily, and then lost power
again.  We?ve not seen power restore at all so far.

?DG

On Mon, 13 Mar 2023 at 10:28 am, Chris Wright <chris at securityshift.com>
wrote:

> I also had outages in Melbourne, (Equinix ME2) around the same time (~9:00
> - > ~9:45am).
>
> Still got a link down with Vocus and still having links go up and down
> with Swoop.
>
> Something more significant is happening somewhere - would be great if
> anyone can shed some light on what's happening so we all don't keep chasing
> our tails.
>
> Thanks
>
> c.
>
> -----Original Message-----
> From: AusNOG <ausnog-bounces at lists.ausnog.net> On Behalf Of DaZZa
> Sent: Monday, 13 March 2023 10:21
> To: Luke Thompson <luke.t at tncrew.com.au>
> Cc: ausnog at ausnog.net <ausnog at lists.ausnog.net>
> Subject: Re: [AusNOG] Aussie Broadband Outage
>
> [You don't often get email from dazzagibbs at gmail.com. Learn why this is
> important at https://aka.ms/LearnAboutSenderIdentification ]
>
> This email came from an external source. Please be cautious with links and
> content.
>
> Interesting.
>
> I believe Aussie is based in Melbourne - I had a major outage on several
> Vocus services in Melbourne between about 09:10 and 09:40 this morning.
>
> Wonder if they're in a shared DC and took a hit at the same time as Aussie
> did...
>
> D
>
> On Mon, 13 Mar 2023 at 10:10, Luke Thompson <luke.t at tncrew.com.au> wrote:
> >
> > James on Whirlpool has just shared what you likely saw tweeted before:
> >
> > "We are currently experiencing issues with one of our data centres,
> which is impacting some of our services including MyAussie, VoIP, and our
> Customer Service phone lines / Live Chat"
> >
> > Sounds like there's no redundancy for Carbon, My Aussie, VoIP, Chat,
> etc. Wild.
> >
> > Luke Thompson
> > Chief Technical Officer
> >
> > The Network Crew Pty Ltd
> > https://thenetworkcrew.com.au
> >
> > On 13 March 2023 10:07:57 am Troy <troy at troykelly.com> wrote:
> >>
> >> Is anybody from Aussie able to share what's going on?
> >>
> >> Looks like there's a major outage with services (no Carbon, phones,
> >> etc)
> >>
> >> There was? a tweet for about 40 seconds saying that there was an issue
> with a data centre, but it was deleted.
> >>
> >> Regards, Troy
> >> Brevity is the elixir of life.
> >> Father Hector McGrath, Pixie 2020
> >
> >
> > _______________________________________________
> > AusNOG mailing list
> > AusNOG at lists.ausnog.net
> > https://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
> --
> veg*e*tar*i*an:
> Ancient tribal slang for the village idiot who can't hunt, fish or ride
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
-- 

Damien Gardner Jnr
VK2TDG. Dip EE. GradIEAust
rendrag at rendrag.net -  http://www.rendrag.net/
--
We rode on the winds of the rising storm,
 We ran to the sounds of thunder.
We danced among the lightning bolts,
 and tore the world asunder
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230313/5e111171/attachment-0001.htm>

From luke.t at tncrew.com.au  Mon Mar 13 10:37:03 2023
From: luke.t at tncrew.com.au (Luke Thompson)
Date: Mon, 13 Mar 2023 10:37:03 +1100
Subject: [AusNOG] Aussie Broadband Outage
In-Reply-To: <CAPrfDLW0eVQ4FO=8a9p5Qczg_Cc4G6n5bhkp383Xs7kSeHZ8xg@mail.gmail.com>
References: <A8510_r-U8zC150QdI98_t_99awf5jOvU3V_ePFzX1BbrF1KqGCMlu7byPLH9ZjiqOz0vHFDDc7Iw7JqxzUBjL_MoKqzyp5mRkf-NsjgO48=@troykelly.com>
 <186d8172568.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
 <CAMpV8MqYi65O_MFdp+94NPR4uoUNVA5_0HYktBZ-DOwr+_47kw@mail.gmail.com>
 <MEYPR01MB626395A30283BDAA1CCF0BC0A0B89@MEYPR01MB6263.ausprd01.prod.outlook.com>
 <CAPrfDLW0eVQ4FO=8a9p5Qczg_Cc4G6n5bhkp383Xs7kSeHZ8xg@mail.gmail.com>
Message-ID: <186d82f4918.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>

Given the commonality of outage type, it's smelling like civil or grid 
problems isn't it?


Vocus and Equinix losing power over both rails, optical links dropping, etc 
sounds like a major problem in the earth or at a dist/sub station. Curious 
to learn the root cause.


A team somewhere is having a bad Monday morning, that's for sure. Good luck 
to all.

Luke Thompson
Chief Technical Officer

The Network Crew Pty Ltd
https://thenetworkcrew.com.au

On 13 March 2023 10:31:35 am Damien Gardner Jnr <rendrag at rendrag.net> wrote:
> I would assume it?s to do with VDC-MEL03 having lost power around 9:05am.  
> A few folks reported getting one rail back temporarily, and then lost power 
> again.  We?ve not seen power restore at all so far.
>
> ?DG
>
> On Mon, 13 Mar 2023 at 10:28 am, Chris Wright <chris at securityshift.com> wrote:
> I also had outages in Melbourne, (Equinix ME2) around the same time (~9:00 
> - > ~9:45am).
>
> Still got a link down with Vocus and still having links go up and down with 
> Swoop.
>
> Something more significant is happening somewhere - would be great if 
> anyone can shed some light on what's happening so we all don't keep chasing 
> our tails.
>
> Thanks
>
> c.
>
> -----Original Message-----
> From: AusNOG <ausnog-bounces at lists.ausnog.net> On Behalf Of DaZZa
> Sent: Monday, 13 March 2023 10:21
> To: Luke Thompson <luke.t at tncrew.com.au>
> Cc: ausnog at ausnog.net <ausnog at lists.ausnog.net>
> Subject: Re: [AusNOG] Aussie Broadband Outage
>
> [You don't often get email from dazzagibbs at gmail.com. Learn why this is 
> important at https://aka.ms/LearnAboutSenderIdentification ]
>
> This email came from an external source. Please be cautious with links and 
> content.
>
> Interesting.
>
> I believe Aussie is based in Melbourne - I had a major outage on several 
> Vocus services in Melbourne between about 09:10 and 09:40 this morning.
>
> Wonder if they're in a shared DC and took a hit at the same time as Aussie 
> did...
>
> D
>
> On Mon, 13 Mar 2023 at 10:10, Luke Thompson <luke.t at tncrew.com.au> wrote:
>>
>> James on Whirlpool has just shared what you likely saw tweeted before:
>>
>> "We are currently experiencing issues with one of our data centres, which 
>> is impacting some of our services including MyAussie, VoIP, and our 
>> Customer Service phone lines / Live Chat"
>>
>> Sounds like there's no redundancy for Carbon, My Aussie, VoIP, Chat, etc. Wild.
>>
>> Luke Thompson
>> Chief Technical Officer
>>
>> The Network Crew Pty Ltd
>> https://thenetworkcrew.com.au
>>
>> On 13 March 2023 10:07:57 am Troy <troy at troykelly.com> wrote:
>>>
>>> Is anybody from Aussie able to share what's going on?
>>>
>>> Looks like there's a major outage with services (no Carbon, phones,
>>> etc)
>>>
>>> There was? a tweet for about 40 seconds saying that there was an issue with 
>>> a data centre, but it was deleted.
>>>
>>> Regards, Troy
>>> Brevity is the elixir of life.
>>> Father Hector McGrath, Pixie 2020
>>
>>
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
> --
> veg*e*tar*i*an:
> Ancient tribal slang for the village idiot who can't hunt, fish or ride 
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
> --
>
> Damien Gardner Jnr
> VK2TDG. Dip EE. GradIEAust
> rendrag at rendrag.net -  http://www.rendrag.net/
> --
> We rode on the winds of the rising storm,
> We ran to the sounds of thunder.
> We danced among the lightning bolts,
> and tore the world asunder

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230313/030dec60/attachment.htm>

From james.braunegg at micron21.com  Mon Mar 13 10:38:53 2023
From: james.braunegg at micron21.com (James Braunegg)
Date: Sun, 12 Mar 2023 23:38:53 +0000
Subject: [AusNOG] Aussie Broadband Outage
In-Reply-To: <186d82f4918.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
References: <A8510_r-U8zC150QdI98_t_99awf5jOvU3V_ePFzX1BbrF1KqGCMlu7byPLH9ZjiqOz0vHFDDc7Iw7JqxzUBjL_MoKqzyp5mRkf-NsjgO48=@troykelly.com>
 <186d8172568.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
 <CAMpV8MqYi65O_MFdp+94NPR4uoUNVA5_0HYktBZ-DOwr+_47kw@mail.gmail.com>
 <MEYPR01MB626395A30283BDAA1CCF0BC0A0B89@MEYPR01MB6263.ausprd01.prod.outlook.com>
 <CAPrfDLW0eVQ4FO=8a9p5Qczg_Cc4G6n5bhkp383Xs7kSeHZ8xg@mail.gmail.com>,
 <186d82f4918.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
Message-ID: <edf576e8bb86484bbd3635254d0eb0eb@micron21.com>

Dear Luke

This is more than likely related to a power issue at 530 Collins Street we lost at 8:45am this morning supply A and then around 8:55am we also lost supply B for all services running at this location.

Power retuned around 9:45am for supply B however supply A was still dead? then at 10:20am both supply A and B both went down again?.

Currently it looks like supply B is back online as of 10:33am? but supply A is still offline so anyone using single power equipment on supply A only will be having issues !

I have raised a ticket with the vendor however no reply as of yet?. Will have to wait for the PIR to hear what happened.

 Kindest Regards


James Braunegg



[X]

1300 769 972<tel:1300%20769%20972> / 0488 997 207<tel:1300%20769%20972>

james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>

www.ddosprotection.com.au  <https://www.ddosprotection.com.au/>


<http://www.micron21.com/>


[X]<https://www.facebook.com/micron21/>

[X]<https://twitter.com/micron21>

Follow us on m21status.com<https://m21status.com> for important service and system updates.


This message is intended for the addressee named above. It may contain privileged or confidential information. If you are not the intended recipient of this message you must not use, copy, distribute or disclose it to anyone other than the addressee. If you have received this message in error please return the message to the sender by replying to it and then delete the message from your computer.




________________________________
From: AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Luke Thompson <luke.t at tncrew.com.au>
Sent: Monday, 13 March 2023 10:37:03 AM
To: Damien Gardner Jnr; Chris Wright
Cc: ausnog at ausnog.net
Subject: Re: [AusNOG] Aussie Broadband Outage

Given the commonality of outage type, it's smelling like civil or grid problems isn't it?

Vocus and Equinix losing power over both rails, optical links dropping, etc sounds like a major problem in the earth or at a dist/sub station. Curious to learn the root cause.

A team somewhere is having a bad Monday morning, that's for sure. Good luck to all.

Luke Thompson
Chief Technical Officer

The Network Crew Pty Ltd
https://thenetworkcrew.com.au



On 13 March 2023 10:31:35 am Damien Gardner Jnr <rendrag at rendrag.net> wrote:

I would assume it?s to do with VDC-MEL03 having lost power around 9:05am.  A few folks reported getting one rail back temporarily, and then lost power again.  We?ve not seen power restore at all so far.

?DG

On Mon, 13 Mar 2023 at 10:28 am, Chris Wright <chris at securityshift.com<mailto:chris at securityshift.com>> wrote:
I also had outages in Melbourne, (Equinix ME2) around the same time (~9:00 - > ~9:45am).

Still got a link down with Vocus and still having links go up and down with Swoop.

Something more significant is happening somewhere - would be great if anyone can shed some light on what's happening so we all don't keep chasing our tails.

Thanks

c.

-----Original Message-----
From: AusNOG <ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>> On Behalf Of DaZZa
Sent: Monday, 13 March 2023 10:21
To: Luke Thompson <luke.t at tncrew.com.au<mailto:luke.t at tncrew.com.au>>
Cc: ausnog at ausnog.net<mailto:ausnog at ausnog.net> <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: Re: [AusNOG] Aussie Broadband Outage

[You don't often get email from dazzagibbs at gmail.com<mailto:dazzagibbs at gmail.com>. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

This email came from an external source. Please be cautious with links and content.

Interesting.

I believe Aussie is based in Melbourne - I had a major outage on several Vocus services in Melbourne between about 09:10 and 09:40 this morning.

Wonder if they're in a shared DC and took a hit at the same time as Aussie did...

D

On Mon, 13 Mar 2023 at 10:10, Luke Thompson <luke.t at tncrew.com.au<mailto:luke.t at tncrew.com.au>> wrote:
>
> James on Whirlpool has just shared what you likely saw tweeted before:
>
> "We are currently experiencing issues with one of our data centres, which is impacting some of our services including MyAussie, VoIP, and our Customer Service phone lines / Live Chat"
>
> Sounds like there's no redundancy for Carbon, My Aussie, VoIP, Chat, etc. Wild.
>
> Luke Thompson
> Chief Technical Officer
>
> The Network Crew Pty Ltd
> https://thenetworkcrew.com.au
>
> On 13 March 2023 10:07:57 am Troy <troy at troykelly.com<mailto:troy at troykelly.com>> wrote:
>>
>> Is anybody from Aussie able to share what's going on?
>>
>> Looks like there's a major outage with services (no Carbon, phones,
>> etc)
>>
>> There was? a tweet for about 40 seconds saying that there was an issue with a data centre, but it was deleted.
>>
>> Regards, Troy
>> Brevity is the elixir of life.
>> Father Hector McGrath, Pixie 2020
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
> https://lists.ausnog.net/mailman/listinfo/ausnog



--
veg*e*tar*i*an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride _______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
https://lists.ausnog.net/mailman/listinfo/ausnog
_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
https://lists.ausnog.net/mailman/listinfo/ausnog
--

Damien Gardner Jnr
VK2TDG. Dip EE. GradIEAust
rendrag at rendrag.net<mailto:rendrag at rendrag.net> -  http://www.rendrag.net/
--
We rode on the winds of the rising storm,
 We ran to the sounds of thunder.
We danced among the lightning bolts,
 and tore the world asunder

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230312/3dd6c5af/attachment.htm>

From luke.t at tncrew.com.au  Mon Mar 13 10:42:49 2023
From: luke.t at tncrew.com.au (Luke Thompson)
Date: Mon, 13 Mar 2023 10:42:49 +1100
Subject: [AusNOG] Aussie Broadband Outage
In-Reply-To: <edf576e8bb86484bbd3635254d0eb0eb@micron21.com>
References: <A8510_r-U8zC150QdI98_t_99awf5jOvU3V_ePFzX1BbrF1KqGCMlu7byPLH9ZjiqOz0vHFDDc7Iw7JqxzUBjL_MoKqzyp5mRkf-NsjgO48=@troykelly.com>
 <186d8172568.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
 <CAMpV8MqYi65O_MFdp+94NPR4uoUNVA5_0HYktBZ-DOwr+_47kw@mail.gmail.com>
 <MEYPR01MB626395A30283BDAA1CCF0BC0A0B89@MEYPR01MB6263.ausprd01.prod.outlook.com>
 <CAPrfDLW0eVQ4FO=8a9p5Qczg_Cc4G6n5bhkp383Xs7kSeHZ8xg@mail.gmail.com>,
 <186d82f4918.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
 <edf576e8bb86484bbd3635254d0eb0eb@micron21.com>
Message-ID: <186d83490a8.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>

Reminds me of Vocus Sydney way back when. Doesn't sound brilliantly 
engineered - very curious for the explanation down the track to see if it 
could've been avoided.

Re: single feed, transfer switches have their place for single corded kit, 
have no doubt!

Cheers,

Luke Thompson
Chief Technical Officer

The Network Crew Pty Ltd
https://thenetworkcrew.com.au
On 13 March 2023 10:39:58 am James Braunegg <james.braunegg at micron21.com> 
wrote:
> Dear Luke
>
>
>
> This is more than likely related to a power issue at 530 Collins Street we 
> lost at 8:45am this morning supply A and then around 8:55am we also lost 
> supply B for all services running at this location.
>
>
>
> Power retuned around 9:45am for supply B however supply A was still dead? 
> then at 10:20am both supply A and B both went down again?.
>
>
>
> Currently it looks like supply B is back online as of 10:33am? but supply A 
> is still offline so anyone using single power equipment on supply A only 
> will be having issues !
>
>
>
> I have raised a ticket with the vendor however no reply as of yet?. Will 
> have to wait for the PIR to hear what happened.
>
>
>
>
> Kindest Regards
>
>
>
>
>
>
>
> James Braunegg
>
>
>
> 	
>
> 	
>
> 1300 769 972 / 0488 997 207
>
>
>
>
> james.braunegg at micron21.com
>
>
>
>
> www.ddosprotection.com.au
>
> 	
>
>
>
>
> 	
>
> 	
>
> Follow us on m21status.com for important service and system updates.
>
>
>
>
> This message is intended for the addressee named above. It may contain 
> privileged or confidential information. If you are not the intended 
> recipient of this message you must not use, copy, distribute or disclose it 
> to anyone other than the addressee. If you have received this message in 
> error please return the message to the sender by replying to it and then 
> delete the message from your computer.
>
>
> From: AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Luke Thompson 
> <luke.t at tncrew.com.au>
> Sent: Monday, 13 March 2023 10:37:03 AM
> To: Damien Gardner Jnr; Chris Wright
> Cc: ausnog at ausnog.net
> Subject: Re: [AusNOG] Aussie Broadband Outage
> Given the commonality of outage type, it's smelling like civil or grid 
> problems isn't it?
>
>
> Vocus and Equinix losing power over both rails, optical links dropping, etc 
> sounds like a major problem in the earth or at a dist/sub station. Curious 
> to learn the root cause.
>
>
> A team somewhere is having a bad Monday morning, that's for sure. Good luck 
> to all.
>
> Luke Thompson
> Chief Technical Officer
>
>
> The Network Crew Pty Ltd
> https://thenetworkcrew.com.au
>
> On 13 March 2023 10:31:35 am Damien Gardner Jnr <rendrag at rendrag.net> wrote:
>> I would assume it?s to do with VDC-MEL03 having lost power around 9:05am.  
>> A few folks reported getting one rail back temporarily, and then lost power 
>> again.  We?ve not seen power restore at all so far.
>>
>>
>> ?DG
>>
>>
>> On Mon, 13 Mar 2023 at 10:28 am, Chris Wright <chris at securityshift.com> wrote:
>>
>> I also had outages in Melbourne, (Equinix ME2) around the same time (~9:00 
>> - > ~9:45am).
>>
>> Still got a link down with Vocus and still having links go up and down with 
>> Swoop.
>>
>> Something more significant is happening somewhere - would be great if 
>> anyone can shed some light on what's happening so we all don't keep chasing 
>> our tails.
>>
>> Thanks
>>
>> c.
>>
>> -----Original Message-----
>> From: AusNOG <ausnog-bounces at lists.ausnog.net> On Behalf Of DaZZa
>> Sent: Monday, 13 March 2023 10:21
>> To: Luke Thompson <luke.t at tncrew.com.au>
>> Cc: ausnog at ausnog.net <ausnog at lists.ausnog.net>
>> Subject: Re: [AusNOG] Aussie Broadband Outage
>>
>> [You don't often get email from dazzagibbs at gmail.com. Learn why this is 
>> important at https://aka.ms/LearnAboutSenderIdentification ]
>>
>> This email came from an external source. Please be cautious with links and 
>> content.
>>
>> Interesting.
>>
>> I believe Aussie is based in Melbourne - I had a major outage on several 
>> Vocus services in Melbourne between about 09:10 and 09:40 this morning.
>>
>> Wonder if they're in a shared DC and took a hit at the same time as Aussie 
>> did...
>>
>> D
>>
>> On Mon, 13 Mar 2023 at 10:10, Luke Thompson <luke.t at tncrew.com.au> wrote:
>>>
>>> James on Whirlpool has just shared what you likely saw tweeted before:
>>>
>>> "We are currently experiencing issues with one of our data centres, which 
>>> is impacting some of our services including MyAussie, VoIP, and our 
>>> Customer Service phone lines / Live Chat"
>>>
>>> Sounds like there's no redundancy for Carbon, My Aussie, VoIP, Chat, etc. Wild.
>>>
>>> Luke Thompson
>>> Chief Technical Officer
>>>
>>> The Network Crew Pty Ltd
>>> https://thenetworkcrew.com.au
>>>
>>> On 13 March 2023 10:07:57 am Troy <troy at troykelly.com> wrote:
>>>>
>>>> Is anybody from Aussie able to share what's going on?
>>>>
>>>> Looks like there's a major outage with services (no Carbon, phones,
>>>> etc)
>>>>
>>>> There was? a tweet for about 40 seconds saying that there was an issue with 
>>>> a data centre, but it was deleted.
>>>>
>>>> Regards, Troy
>>>> Brevity is the elixir of life.
>>>> Father Hector McGrath, Pixie 2020
>>>
>>>
>>> _______________________________________________
>>> AusNOG mailing list
>>> AusNOG at lists.ausnog.net
>>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
>>
>>
>> --
>> veg*e*tar*i*an:
>> Ancient tribal slang for the village idiot who can't hunt, fish or ride 
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
>> --
>>
>>
>> Damien Gardner Jnr
>> VK2TDG. Dip EE. GradIEAust
>> rendrag at rendrag.net -  http://www.rendrag.net/
>> --
>> We rode on the winds of the rising storm,
>> We ran to the sounds of thunder.
>> We danced among the lightning bolts,
>> and tore the world asunder
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230313/29180f59/attachment.htm>

From chris at securityshift.com  Mon Mar 13 10:47:07 2023
From: chris at securityshift.com (Chris Wright)
Date: Sun, 12 Mar 2023 23:47:07 +0000
Subject: [AusNOG] Aussie Broadband Outage
In-Reply-To: <186d83490a8.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
References: <A8510_r-U8zC150QdI98_t_99awf5jOvU3V_ePFzX1BbrF1KqGCMlu7byPLH9ZjiqOz0vHFDDc7Iw7JqxzUBjL_MoKqzyp5mRkf-NsjgO48=@troykelly.com>
 <186d8172568.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
 <CAMpV8MqYi65O_MFdp+94NPR4uoUNVA5_0HYktBZ-DOwr+_47kw@mail.gmail.com>
 <MEYPR01MB626395A30283BDAA1CCF0BC0A0B89@MEYPR01MB6263.ausprd01.prod.outlook.com>
 <CAPrfDLW0eVQ4FO=8a9p5Qczg_Cc4G6n5bhkp383Xs7kSeHZ8xg@mail.gmail.com>,
 <186d82f4918.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
 <edf576e8bb86484bbd3635254d0eb0eb@micron21.com>
 <186d83490a8.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
Message-ID: <MEYPR01MB62635C7F399BB70E58415BCEA0B89@MEYPR01MB6263.ausprd01.prod.outlook.com>

Looks like somethings in vocus own network might be "singe power equipment" :(

traceroute to 91.204.80.53 (91.204.80.53), 30 hops max, 60 byte packets
1  144.48.37.193 (144.48.37.193)  0.654 ms  0.727 ms  0.781 ms
2  144.48.37.45 (144.48.37.45)  0.797 ms  0.976 ms  1.089 ms
3  core-a.me1.vic.hostuniversal.com.au (103.216.222.1)  0.158 ms  0.183 ms  0.216 ms
4  4826.mel.equinix.com (183.177.61.1)  0.785 ms  0.955 ms  1.193 ms
5  be150.cor02.mel11.vic.vocus.network (114.31.196.56)  0.632 ms  0.901 ms  0.753 ms
6  be151.bdr02.mel11.vic.vocus.network (114.31.196.53)  1.367 ms be150.bdr02.mel11.vic.vocus.network (114.31.196.55)  1.281 ms be151.bdr02.mel11.vic.vocus.network (114.31.196.53)  1.501 ms
7  be150.cor02.mel07.vic.vocus.network (114.31.196.54)  0.612 ms be151.cor02.mel11.vic.vocus.network (114.31.196.52)  0.952 ms be150.cor02.mel07.vic.vocus.network (114.31.196.54)  0.657 ms
8  be150.bdr03.mel07.vic.vocus.network (114.31.196.57)  1.196 ms  1.407 ms  1.579 ms
9  be150.cor02.mel11.vic.vocus.network (114.31.196.56)  1.036 ms  1.064 ms be151.cor02.mel07.vic.vocus.network (114.31.196.50)  1.120 ms
10  be150.bdr02.mel11.vic.vocus.network (114.31.196.55)  1.095 ms  1.419 ms be151.bdr02.mel11.vic.vocus.network (114.31.196.53)  1.505 ms
11  be151.cor02.mel11.vic.vocus.network (114.31.196.52)  0.883 ms be150.cor02.mel07.vic.vocus.network (114.31.196.54)  0.847 ms be151.cor02.mel11.vic.vocus.network (114.31.196.52)  1.040 ms
12  be151.bdr03.mel07.vic.vocus.network (114.31.196.51)  0.829 ms be150.bdr03.mel07.vic.vocus.network (114.31.196.57)  1.769 ms  1.602 ms
13  be150.cor02.mel11.vic.vocus.network (114.31.196.56)  1.225 ms be151.cor02.mel07.vic.vocus.network (114.31.196.50)  0.976 ms  1.007 ms
14  be150.bdr02.mel11.vic.vocus.network (114.31.196.55)  0.971 ms be151.bdr02.mel11.vic.vocus.network (114.31.196.53)  1.396 ms  1.460 ms
15  be151.cor02.mel11.vic.vocus.network (114.31.196.52)  1.208 ms  1.245 ms be150.cor02.mel07.vic.vocus.network (114.31.196.54)  1.222 ms
16  be150.bdr03.mel07.vic.vocus.network (114.31.196.57)  1.808 ms be151.bdr03.mel07.vic.vocus.network (114.31.196.51)  4.102 ms be150.bdr03.mel07.vic.vocus.network (114.31.196.57)  1.969 ms
17  be151.cor02.mel07.vic.vocus.network (114.31.196.50)  1.272 ms be150.cor02.mel11.vic.vocus.network (114.31.196.56)  1.318 ms  1.298 ms
18  be150.bdr02.mel11.vic.vocus.network (114.31.196.55)  1.223 ms be151.bdr02.mel11.vic.vocus.network (114.31.196.53)  1.299 ms  1.288 ms
19  be151.cor02.mel11.vic.vocus.network (114.31.196.52)  1.133 ms  1.144 ms  1.288 ms
20  be150.bdr03.mel07.vic.vocus.network (114.31.196.57)  2.011 ms  5.261 ms  5.254 ms
21  be150.cor02.mel11.vic.vocus.network (114.31.196.56)  1.326 ms be151.cor02.mel07.vic.vocus.network (114.31.196.50)  1.503 ms  1.529 ms
22  be150.bdr02.mel11.vic.vocus.network (114.31.196.55)  1.400 ms  1.296 ms be151.bdr02.mel11.vic.vocus.network (114.31.196.53)  1.490 ms
23  be150.cor02.mel07.vic.vocus.network (114.31.196.54)  1.615 ms  1.454 ms  1.629 ms
24  be150.bdr03.mel07.vic.vocus.network (114.31.196.57)  1.885 ms be151.bdr03.mel07.vic.vocus.network (114.31.196.51)  1.293 ms  1.377 ms
25  be151.cor02.mel07.vic.vocus.network (114.31.196.50)  1.364 ms be150.cor02.mel11.vic.vocus.network (114.31.196.56)  1.625 ms be151.cor02.mel07.vic.vocus.network (114.31.196.50)  1.792 ms
26  be151.bdr02.mel11.vic.vocus.network (114.31.196.53)  1.685 ms be150.bdr02.mel11.vic.vocus.network (114.31.196.55)  1.644 ms be151.bdr02.mel11.vic.vocus.network (114.31.196.53)  1.437 ms
27  be150.cor02.mel07.vic.vocus.network (114.31.196.54)  1.654 ms * *
28  be151.bdr03.mel07.vic.vocus.network (114.31.196.51)  1.342 ms  4.852 ms be150.bdr03.mel07.vic.vocus.network (114.31.196.57)  1.659 ms
29  be151.cor02.mel07.vic.vocus.network (114.31.196.50)  1.629 ms  1.639 ms be150.cor02.mel11.vic.vocus.network (114.31.196.56)  1.459 ms
30  be151.bdr02.mel11.vic.vocus.network (114.31.196.53)  1.780 ms  1.867 ms  1.553 ms


Thanks

c.

From: Luke Thompson <luke.t at tncrew.com.au>
Sent: Monday, 13 March 2023 10:43
To: James Braunegg <james.braunegg at micron21.com>; Damien Gardner Jnr <rendrag at rendrag.net>; Chris Wright <chris at securityshift.com>
Cc: ausnog at ausnog.net <ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] Aussie Broadband Outage

This email came from an external source. Please be cautious with links and content.
Reminds me of Vocus Sydney way back when. Doesn't sound brilliantly engineered - very curious for the explanation down the track to see if it could've been avoided.

Re: single feed, transfer switches have their place for single corded kit, have no doubt!

Cheers,

Luke Thompson
Chief Technical Officer

The Network Crew Pty Ltd
https://thenetworkcrew.com.au

On 13 March 2023 10:39:58 am James Braunegg <james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>> wrote:
Dear Luke

This is more than likely related to a power issue at 530 Collins Street we lost at 8:45am this morning supply A and then around 8:55am we also lost supply B for all services running at this location.

Power retuned around 9:45am for supply B however supply A was still dead... then at 10:20am both supply A and B both went down again....

Currently it looks like supply B is back online as of 10:33am... but supply A is still offline so anyone using single power equipment on supply A only will be having issues !

I have raised a ticket with the vendor however no reply as of yet.... Will have to wait for the PIR to hear what happened.

 Kindest Regards



James Braunegg
1300 769 972<tel:1300%20769%20972> / 0488 997 207<tel:1300%20769%20972>
james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>
www.ddosprotection.com.au  <https://www.ddosprotection.com.au/>
<https://www.facebook.com/micron21/>
<https://www.facebook.com/micron21/>
Follow us on m21status.com for important service and system updates.<https://www.facebook.com/micron21/>
This message is intended for the addressee named above. It may contain privileged or confidential information. If you are not the intended recipient of this message you must not use, copy, distribute or disclose it to anyone other than the addressee. If you have received this message in error please return the message to the sender by replying to it and then delete the message from your computer.<https://www.facebook.com/micron21/>
 <https://www.facebook.com/micron21/>
<https://www.facebook.com/micron21/>
________________________________
From: AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Luke Thompson <luke.t at tncrew.com.au>
Sent: Monday, 13 March 2023 10:37:03 AM
To: Damien Gardner Jnr; Chris Wright
Cc: ausnog at ausnog.net
Subject: Re: [AusNOG] Aussie Broadband Outage <https://www.facebook.com/micron21/>
 <https://www.facebook.com/micron21/>
Given the commonality of outage type, it's smelling like civil or grid problems isn't it?<https://www.facebook.com/micron21/>
 <https://www.facebook.com/micron21/>
Vocus and Equinix losing power over both rails, optical links dropping, etc sounds like a major problem in the earth or at a dist/sub station. Curious to learn the root cause. <https://www.facebook.com/micron21/>
 <https://www.facebook.com/micron21/>
A team somewhere is having a bad Monday morning, that's for sure. Good luck to all. <https://www.facebook.com/micron21/>
 <https://www.facebook.com/micron21/>
Luke Thompson<https://www.facebook.com/micron21/>
Chief Technical Officer<https://www.facebook.com/micron21/>
 <https://www.facebook.com/micron21/>
The Network Crew Pty Ltd<https://www.facebook.com/micron21/>
https://thenetworkcrew.com.au<https://www.facebook.com/micron21/>
 <https://www.facebook.com/micron21/>
 <https://www.facebook.com/micron21/>
On 13 March 2023 10:31:35 am Damien Gardner Jnr <rendrag at rendrag.net> wrote:<https://www.facebook.com/micron21/>
I would assume it's to do with VDC-MEL03 having lost power around 9:05am.  A few folks reported getting one rail back temporarily, and then lost power again.  We've not seen power restore at all so far.<https://www.facebook.com/micron21/>
 <https://www.facebook.com/micron21/>
-DG<https://www.facebook.com/micron21/>
 <https://www.facebook.com/micron21/>
On Mon, 13 Mar 2023 at 10:28 am, Chris Wright <chris at securityshift.com> wrote:<https://www.facebook.com/micron21/>
I also had outages in Melbourne, (Equinix ME2) around the same time (~9:00 - > ~9:45am).

Still got a link down with Vocus and still having links go up and down with Swoop.

Something more significant is happening somewhere - would be great if anyone can shed some light on what's happening so we all don't keep chasing our tails.

Thanks

c.

-----Original Message-----
From: AusNOG <ausnog-bounces at lists.ausnog.net> On Behalf Of DaZZa
Sent: Monday, 13 March 2023 10:21
To: Luke Thompson <luke.t at tncrew.com.au>
Cc: ausnog at ausnog.net <ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] Aussie Broadband Outage

[You don't often get email from dazzagibbs at gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

This email came from an external source. Please be cautious with links and content.

Interesting.

I believe Aussie is based in Melbourne - I had a major outage on several Vocus services in Melbourne between about 09:10 and 09:40 this morning.

Wonder if they're in a shared DC and took a hit at the same time as Aussie did...

D

On Mon, 13 Mar 2023 at 10:10, Luke Thompson <luke.t at tncrew.com.au> wrote:
>
> James on Whirlpool has just shared what you likely saw tweeted before:
>
> "We are currently experiencing issues with one of our data centres, which is impacting some of our services including MyAussie, VoIP, and our Customer Service phone lines / Live Chat"
>
> Sounds like there's no redundancy for Carbon, My Aussie, VoIP, Chat, etc. Wild.
>
> Luke Thompson
> Chief Technical Officer
>
> The Network Crew Pty Ltd
> https://thenetworkcrew.com.au
>
> On 13 March 2023 10:07:57 am Troy <troy at troykelly.com> wrote:
>>
>> Is anybody from Aussie able to share what's going on?
>>
>> Looks like there's a major outage with services (no Carbon, phones,
>> etc)
>>
>> There was? a tweet for about 40 seconds saying that there was an issue with a data centre, but it was deleted.
>>
>> Regards, Troy
>> Brevity is the elixir of life.
>> Father Hector McGrath, Pixie 2020
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog



--
veg*e*tar*i*an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride _______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog
_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog<https://www.facebook.com/micron21/>
-- <https://www.facebook.com/micron21/>
Damien Gardner Jnr
VK2TDG. Dip EE. GradIEAust
rendrag at rendrag.net -  http://www.rendrag.net/
--
We rode on the winds of the rising storm,
 We ran to the sounds of thunder.
We danced among the lightning bolts,
 and tore the world asunder<https://www.facebook.com/micron21/>
 <https://www.facebook.com/micron21/>
_______________________________________________<https://www.facebook.com/micron21/>
AusNOG mailing list<https://www.facebook.com/micron21/>
AusNOG at lists.ausnog.net<https://www.facebook.com/micron21/>
https://lists.ausnog.net/mailman/listinfo/ausnog<https://www.facebook.com/micron21/>
 <https://www.facebook.com/micron21/>
 <https://www.facebook.com/micron21/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230312/60b6d130/attachment.htm>

From james.braunegg at micron21.com  Mon Mar 13 10:52:15 2023
From: james.braunegg at micron21.com (James Braunegg)
Date: Sun, 12 Mar 2023 23:52:15 +0000
Subject: [AusNOG] Aussie Broadband Outage
In-Reply-To: <MEYPR01MB62635C7F399BB70E58415BCEA0B89@MEYPR01MB6263.ausprd01.prod.outlook.com>
References: <A8510_r-U8zC150QdI98_t_99awf5jOvU3V_ePFzX1BbrF1KqGCMlu7byPLH9ZjiqOz0vHFDDc7Iw7JqxzUBjL_MoKqzyp5mRkf-NsjgO48=@troykelly.com>
 <186d8172568.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
 <CAMpV8MqYi65O_MFdp+94NPR4uoUNVA5_0HYktBZ-DOwr+_47kw@mail.gmail.com>
 <MEYPR01MB626395A30283BDAA1CCF0BC0A0B89@MEYPR01MB6263.ausprd01.prod.outlook.com>
 <CAPrfDLW0eVQ4FO=8a9p5Qczg_Cc4G6n5bhkp383Xs7kSeHZ8xg@mail.gmail.com>,
 <186d82f4918.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
 <edf576e8bb86484bbd3635254d0eb0eb@micron21.com>
 <186d83490a8.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
 <MEYPR01MB62635C7F399BB70E58415BCEA0B89@MEYPR01MB6263.ausprd01.prod.outlook.com>
Message-ID: <626afe0934ae40c3ad6010b7a56917d1@micron21.com>

Dear All

Anyone who wants the official Vocus incident number it is INC30876585.

Kindest Regards

James Braunegg



[cid:image001.png at 01D95599.DE4AEDB0]

1300 769 972<tel:1300%20769%20972> / 0488 997 207<tel:1300%20769%20972>

james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>

www.ddosprotection.com.au  <https://www.ddosprotection.com.au/>


[cid:image002.png at 01D95599.DE4AEDB0]<http://www.micron21.com/>


[cid:image003.png at 01D95599.DE4AEDB0]<https://www.facebook.com/micron21/>

[cid:image004.png at 01D95599.DE4AEDB0]<https://twitter.com/micron21>

Follow us on m21status.com<https://m21status.com> for important service and system updates.


This message is intended for the addressee named above. It may contain privileged or confidential information. If you are not the intended recipient of this message you must not use, copy, distribute or disclose it to anyone other than the addressee. If you have received this message in error please return the message to the sender by replying to it and then delete the message from your computer.




From: Chris Wright <chris at securityshift.com>
Sent: Monday, 13 March 2023 10:47 AM
To: Luke Thompson <luke.t at tncrew.com.au>; James Braunegg <james.braunegg at micron21.com>; Damien Gardner Jnr <rendrag at rendrag.net>
Cc: ausnog at ausnog.net <ausnog at lists.ausnog.net>
Subject: RE: [AusNOG] Aussie Broadband Outage

Looks like somethings in vocus own network might be "singe power equipment" :(

traceroute to 91.204.80.53 (91.204.80.53), 30 hops max, 60 byte packets
1  144.48.37.193 (144.48.37.193)  0.654 ms  0.727 ms  0.781 ms
2  144.48.37.45 (144.48.37.45)  0.797 ms  0.976 ms  1.089 ms
3  core-a.me1.vic.hostuniversal.com.au (103.216.222.1)  0.158 ms  0.183 ms  0.216 ms
4  4826.mel.equinix.com (183.177.61.1)  0.785 ms  0.955 ms  1.193 ms
5  be150.cor02.mel11.vic.vocus.network (114.31.196.56)  0.632 ms  0.901 ms  0.753 ms
6  be151.bdr02.mel11.vic.vocus.network (114.31.196.53)  1.367 ms be150.bdr02.mel11.vic.vocus.network (114.31.196.55)  1.281 ms be151.bdr02.mel11.vic.vocus.network (114.31.196.53)  1.501 ms
7  be150.cor02.mel07.vic.vocus.network (114.31.196.54)  0.612 ms be151.cor02.mel11.vic.vocus.network (114.31.196.52)  0.952 ms be150.cor02.mel07.vic.vocus.network (114.31.196.54)  0.657 ms
8  be150.bdr03.mel07.vic.vocus.network (114.31.196.57)  1.196 ms  1.407 ms  1.579 ms
9  be150.cor02.mel11.vic.vocus.network (114.31.196.56)  1.036 ms  1.064 ms be151.cor02.mel07.vic.vocus.network (114.31.196.50)  1.120 ms
10  be150.bdr02.mel11.vic.vocus.network (114.31.196.55)  1.095 ms  1.419 ms be151.bdr02.mel11.vic.vocus.network (114.31.196.53)  1.505 ms
11  be151.cor02.mel11.vic.vocus.network (114.31.196.52)  0.883 ms be150.cor02.mel07.vic.vocus.network (114.31.196.54)  0.847 ms be151.cor02.mel11.vic.vocus.network (114.31.196.52)  1.040 ms
12  be151.bdr03.mel07.vic.vocus.network (114.31.196.51)  0.829 ms be150.bdr03.mel07.vic.vocus.network (114.31.196.57)  1.769 ms  1.602 ms
13  be150.cor02.mel11.vic.vocus.network (114.31.196.56)  1.225 ms be151.cor02.mel07.vic.vocus.network (114.31.196.50)  0.976 ms  1.007 ms
14  be150.bdr02.mel11.vic.vocus.network (114.31.196.55)  0.971 ms be151.bdr02.mel11.vic.vocus.network (114.31.196.53)  1.396 ms  1.460 ms
15  be151.cor02.mel11.vic.vocus.network (114.31.196.52)  1.208 ms  1.245 ms be150.cor02.mel07.vic.vocus.network (114.31.196.54)  1.222 ms
16  be150.bdr03.mel07.vic.vocus.network (114.31.196.57)  1.808 ms be151.bdr03.mel07.vic.vocus.network (114.31.196.51)  4.102 ms be150.bdr03.mel07.vic.vocus.network (114.31.196.57)  1.969 ms
17  be151.cor02.mel07.vic.vocus.network (114.31.196.50)  1.272 ms be150.cor02.mel11.vic.vocus.network (114.31.196.56)  1.318 ms  1.298 ms
18  be150.bdr02.mel11.vic.vocus.network (114.31.196.55)  1.223 ms be151.bdr02.mel11.vic.vocus.network (114.31.196.53)  1.299 ms  1.288 ms
19  be151.cor02.mel11.vic.vocus.network (114.31.196.52)  1.133 ms  1.144 ms  1.288 ms
20  be150.bdr03.mel07.vic.vocus.network (114.31.196.57)  2.011 ms  5.261 ms  5.254 ms
21  be150.cor02.mel11.vic.vocus.network (114.31.196.56)  1.326 ms be151.cor02.mel07.vic.vocus.network (114.31.196.50)  1.503 ms  1.529 ms
22  be150.bdr02.mel11.vic.vocus.network (114.31.196.55)  1.400 ms  1.296 ms be151.bdr02.mel11.vic.vocus.network (114.31.196.53)  1.490 ms
23  be150.cor02.mel07.vic.vocus.network (114.31.196.54)  1.615 ms  1.454 ms  1.629 ms
24  be150.bdr03.mel07.vic.vocus.network (114.31.196.57)  1.885 ms be151.bdr03.mel07.vic.vocus.network (114.31.196.51)  1.293 ms  1.377 ms
25  be151.cor02.mel07.vic.vocus.network (114.31.196.50)  1.364 ms be150.cor02.mel11.vic.vocus.network (114.31.196.56)  1.625 ms be151.cor02.mel07.vic.vocus.network (114.31.196.50)  1.792 ms
26  be151.bdr02.mel11.vic.vocus.network (114.31.196.53)  1.685 ms be150.bdr02.mel11.vic.vocus.network (114.31.196.55)  1.644 ms be151.bdr02.mel11.vic.vocus.network (114.31.196.53)  1.437 ms
27  be150.cor02.mel07.vic.vocus.network (114.31.196.54)  1.654 ms * *
28  be151.bdr03.mel07.vic.vocus.network (114.31.196.51)  1.342 ms  4.852 ms be150.bdr03.mel07.vic.vocus.network (114.31.196.57)  1.659 ms
29  be151.cor02.mel07.vic.vocus.network (114.31.196.50)  1.629 ms  1.639 ms be150.cor02.mel11.vic.vocus.network (114.31.196.56)  1.459 ms
30  be151.bdr02.mel11.vic.vocus.network (114.31.196.53)  1.780 ms  1.867 ms  1.553 ms


Thanks

c.

From: Luke Thompson <luke.t at tncrew.com.au<mailto:luke.t at tncrew.com.au>>
Sent: Monday, 13 March 2023 10:43
To: James Braunegg <james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>>; Damien Gardner Jnr <rendrag at rendrag.net<mailto:rendrag at rendrag.net>>; Chris Wright <chris at securityshift.com<mailto:chris at securityshift.com>>
Cc: ausnog at ausnog.net<mailto:ausnog at ausnog.net> <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: Re: [AusNOG] Aussie Broadband Outage

This email came from an external source. Please be cautious with links and content.
Reminds me of Vocus Sydney way back when. Doesn't sound brilliantly engineered - very curious for the explanation down the track to see if it could've been avoided.

Re: single feed, transfer switches have their place for single corded kit, have no doubt!

Cheers,

Luke Thompson
Chief Technical Officer

The Network Crew Pty Ltd
https://thenetworkcrew.com.au

On 13 March 2023 10:39:58 am James Braunegg <james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>> wrote:
Dear Luke

This is more than likely related to a power issue at 530 Collins Street we lost at 8:45am this morning supply A and then around 8:55am we also lost supply B for all services running at this location.

Power retuned around 9:45am for supply B however supply A was still dead... then at 10:20am both supply A and B both went down again....

Currently it looks like supply B is back online as of 10:33am... but supply A is still offline so anyone using single power equipment on supply A only will be having issues !

I have raised a ticket with the vendor however no reply as of yet.... Will have to wait for the PIR to hear what happened.

 Kindest Regards


James Braunegg



1300 769 972<tel:1300%20769%20972> / 0488 997 207<tel:1300%20769%20972>

james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>

www.ddosprotection.com.au  <https://www.ddosprotection.com.au/>




Follow us on m21status.com for important service and system updates.<https://www.facebook.com/micron21/>


This message is intended for the addressee named above. It may contain privileged or confidential information. If you are not the intended recipient of this message you must not use, copy, distribute or disclose it to anyone other than the addressee. If you have received this message in error please return the message to the sender by replying to it and then delete the message from your computer.<https://www.facebook.com/micron21/>


<https://www.facebook.com/micron21/>
<https://www.facebook.com/micron21/>
________________________________

From: AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Luke Thompson <luke.t at tncrew.com.au>
Sent: Monday, 13 March 2023 10:37:03 AM
To: Damien Gardner Jnr; Chris Wright
Cc: ausnog at ausnog.net
Subject: Re: [AusNOG] Aussie Broadband Outage <https://www.facebook.com/micron21/>
 <https://www.facebook.com/micron21/>
Given the commonality of outage type, it's smelling like civil or grid problems isn't it?<https://www.facebook.com/micron21/>
Vocus and Equinix losing power over both rails, optical links dropping, etc sounds like a major problem in the earth or at a dist/sub station. Curious to learn the root cause. <https://www.facebook.com/micron21/>
A team somewhere is having a bad Monday morning, that's for sure. Good luck to all. <https://www.facebook.com/micron21/>
Luke Thompson<https://www.facebook.com/micron21/>
Chief Technical Officer<https://www.facebook.com/micron21/>
The Network Crew Pty Ltd<https://www.facebook.com/micron21/>
https://thenetworkcrew.com.au<https://www.facebook.com/micron21/>
On 13 March 2023 10:31:35 am Damien Gardner Jnr <rendrag at rendrag.net> wrote:<https://www.facebook.com/micron21/>
I would assume it's to do with VDC-MEL03 having lost power around 9:05am.  A few folks reported getting one rail back temporarily, and then lost power again.  We've not seen power restore at all so far.<https://www.facebook.com/micron21/>
-DG<https://www.facebook.com/micron21/>
On Mon, 13 Mar 2023 at 10:28 am, Chris Wright <chris at securityshift.com> wrote:<https://www.facebook.com/micron21/>
I also had outages in Melbourne, (Equinix ME2) around the same time (~9:00 - > ~9:45am).

Still got a link down with Vocus and still having links go up and down with Swoop.

Something more significant is happening somewhere - would be great if anyone can shed some light on what's happening so we all don't keep chasing our tails.

Thanks

c.

-----Original Message-----
From: AusNOG <ausnog-bounces at lists.ausnog.net> On Behalf Of DaZZa
Sent: Monday, 13 March 2023 10:21
To: Luke Thompson <luke.t at tncrew.com.au>
Cc: ausnog at ausnog.net <ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] Aussie Broadband Outage

[You don't often get email from dazzagibbs at gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

This email came from an external source. Please be cautious with links and content.

Interesting.

I believe Aussie is based in Melbourne - I had a major outage on several Vocus services in Melbourne between about 09:10 and 09:40 this morning.

Wonder if they're in a shared DC and took a hit at the same time as Aussie did...

D

On Mon, 13 Mar 2023 at 10:10, Luke Thompson <luke.t at tncrew.com.au> wrote:
>
> James on Whirlpool has just shared what you likely saw tweeted before:
>
> "We are currently experiencing issues with one of our data centres, which is impacting some of our services including MyAussie, VoIP, and our Customer Service phone lines / Live Chat"
>
> Sounds like there's no redundancy for Carbon, My Aussie, VoIP, Chat, etc. Wild.
>
> Luke Thompson
> Chief Technical Officer
>
> The Network Crew Pty Ltd
> https://thenetworkcrew.com.au
>
> On 13 March 2023 10:07:57 am Troy <troy at troykelly.com> wrote:
>>
>> Is anybody from Aussie able to share what's going on?
>>
>> Looks like there's a major outage with services (no Carbon, phones,
>> etc)
>>
>> There was? a tweet for about 40 seconds saying that there was an issue with a data centre, but it was deleted.
>>
>> Regards, Troy
>> Brevity is the elixir of life.
>> Father Hector McGrath, Pixie 2020
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog



--
veg*e*tar*i*an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride _______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog
_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog<https://www.facebook.com/micron21/>
-- <https://www.facebook.com/micron21/>
Damien Gardner Jnr
VK2TDG. Dip EE. GradIEAust
rendrag at rendrag.net -  http://www.rendrag.net/
--
We rode on the winds of the rising storm,
 We ran to the sounds of thunder.
We danced among the lightning bolts,
 and tore the world asunder<https://www.facebook.com/micron21/>
_______________________________________________<https://www.facebook.com/micron21/>
AusNOG mailing list<https://www.facebook.com/micron21/>
AusNOG at lists.ausnog.net<https://www.facebook.com/micron21/>
https://lists.ausnog.net/mailman/listinfo/ausnog<https://www.facebook.com/micron21/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230312/65c3bf58/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 1047 bytes
Desc: image001.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230312/65c3bf58/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 4219 bytes
Desc: image002.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230312/65c3bf58/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 1137 bytes
Desc: image003.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230312/65c3bf58/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 1246 bytes
Desc: image004.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230312/65c3bf58/attachment-0007.png>

From matt at spectrum.com.au  Mon Mar 13 10:54:14 2023
From: matt at spectrum.com.au (Matt Perkins)
Date: Mon, 13 Mar 2023 10:54:14 +1100
Subject: [AusNOG] Aussie Broadband Outage
In-Reply-To: <CAMpV8MqYi65O_MFdp+94NPR4uoUNVA5_0HYktBZ-DOwr+_47kw@mail.gmail.com>
References: =?utf-8?q?=3CA8510=5Fr-U8zC150QdI98=5Ft=5F99awf5jOvU3V=5FePFzX1BbrF1KqGCMlu7byPLH9ZjiqOz0vHFDDc7Iw7JqxzUBj?=
 =?utf-8?q?L=5FMoKqzyp5mRkf-NsjgO48=3D=40troykelly=2Ecom=3E?=
 <186d8172568.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
 <CAMpV8MqYi65O_MFdp+94NPR4uoUNVA5_0HYktBZ-DOwr+_47kw@mail.gmail.com>
Message-ID: <a6f29dc8-e0c6-91a1-0d42-20051f59f62b@spectrum.com.au>

101 Collins lost power.

Matt


On 13/3/2023 10:21 am, DaZZa wrote:
> Interesting.
>
> I believe Aussie is based in Melbourne - I had a major outage on
> several Vocus services in Melbourne between about 09:10 and 09:40 this
> morning.
>
> Wonder if they're in a shared DC and took a hit at the same time as
> Aussie did...
>
> D
>
> On Mon, 13 Mar 2023 at 10:10, Luke Thompson <luke.t at tncrew.com.au> wrote:
>> James on Whirlpool has just shared what you likely saw tweeted before:
>>
>> "We are currently experiencing issues with one of our data centres, which is impacting some of our services including MyAussie, VoIP, and our Customer Service phone lines / Live Chat"
>>
>> Sounds like there's no redundancy for Carbon, My Aussie, VoIP, Chat, etc. Wild.
>>
>> Luke Thompson
>> Chief Technical Officer
>>
>> The Network Crew Pty Ltd
>> https://thenetworkcrew.com.au
>>
>> On 13 March 2023 10:07:57 am Troy <troy at troykelly.com> wrote:
>>> Is anybody from Aussie able to share what's going on?
>>>
>>> Looks like there's a major outage with services (no Carbon, phones, etc)
>>>
>>> There was? a tweet for about 40 seconds saying that there was an issue with a data centre, but it was deleted.
>>>
>>> Regards, Troy
>>> Brevity is the elixir of life.
>>> Father Hector McGrath, Pixie 2020
>>
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
>
>
-- 
** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL

/* Matt Perkins
         Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.
         Office 1300 133 299     matt at spectrum.com.au
         ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000

*/


From ckawchuk at gmail.com  Mon Mar 13 11:03:23 2023
From: ckawchuk at gmail.com (Chris Kawchuk)
Date: Mon, 13 Mar 2023 11:03:23 +1100
Subject: [AusNOG] Aussie Broadband Outage
In-Reply-To: <a6f29dc8-e0c6-91a1-0d42-20051f59f62b@spectrum.com.au>
References: <186d8172568.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
 <CAMpV8MqYi65O_MFdp+94NPR4uoUNVA5_0HYktBZ-DOwr+_47kw@mail.gmail.com>
 <a6f29dc8-e0c6-91a1-0d42-20051f59f62b@spectrum.com.au>
Message-ID: <AF46C893-B4B4-4B40-A295-30283EB7AD6C@gmail.com>

The Cisco offices??

- CK.


> On Mar 13, 2023, at 10:54, Matt Perkins <matt at spectrum.com.au> wrote:
> 
> 101 Collins lost power.
> 
> Matt
> 
> 
> On 13/3/2023 10:21 am, DaZZa wrote:
>> Interesting.
>> 
>> I believe Aussie is based in Melbourne - I had a major outage on
>> several Vocus services in Melbourne between about 09:10 and 09:40 this
>> morning.
>> 
>> Wonder if they're in a shared DC and took a hit at the same time as
>> Aussie did...
>> 
>> D
>> 
>> On Mon, 13 Mar 2023 at 10:10, Luke Thompson <luke.t at tncrew.com.au> wrote:
>>> James on Whirlpool has just shared what you likely saw tweeted before:
>>> 
>>> "We are currently experiencing issues with one of our data centres, which is impacting some of our services including MyAussie, VoIP, and our Customer Service phone lines / Live Chat"
>>> 
>>> Sounds like there's no redundancy for Carbon, My Aussie, VoIP, Chat, etc. Wild.
>>> 
>>> Luke Thompson
>>> Chief Technical Officer
>>> 
>>> The Network Crew Pty Ltd
>>> https://thenetworkcrew.com.au
>>> 
>>> On 13 March 2023 10:07:57 am Troy <troy at troykelly.com> wrote:
>>>> Is anybody from Aussie able to share what's going on?
>>>> 
>>>> Looks like there's a major outage with services (no Carbon, phones, etc)
>>>> 
>>>> There was? a tweet for about 40 seconds saying that there was an issue with a data centre, but it was deleted.
>>>> 
>>>> Regards, Troy
>>>> Brevity is the elixir of life.
>>>> Father Hector McGrath, Pixie 2020
>>> 
>>> _______________________________________________
>>> AusNOG mailing list
>>> AusNOG at lists.ausnog.net
>>> https://lists.ausnog.net/mailman/listinfo/ausnog
>> 
>> 
> -- 
> ** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL
> 
> /* Matt Perkins
>        Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.
>        Office 1300 133 299     matt at spectrum.com.au
>        ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000
> 
> */
> 
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog


From chad at cpkws.com.au  Mon Mar 13 16:00:33 2023
From: chad at cpkws.com.au (Chad Kelly)
Date: Mon, 13 Mar 2023 05:00:33 +0000
Subject: [AusNOG] Aussie Broadband Outage
Message-ID: <ME3PR01MB68782D49D2379FEBF5F1795BFDB99@ME3PR01MB6878.ausprd01.prod.outlook.com>

This isn't the first time Vocus have lost power both A and B feeds to a DC.

Happened in NSW I believe a year or two back.

The fact they lost both A and B feeds is just frankly bad design and should never happen.

That said this is why anything critical shouldn't be housed in a single Datacentre and should be load balanced between a few or at the very least any decent business grade provider should be able to provide you with backup connectivity in the case of internet. Even 4G or 5G backup connectivity is better than nothing.

This isn't hard to organize for most business grade providers and is the difference between a proper business grade provider and one selling business services on a residential grade network.

Any decent provider should be able to give you multiple Fibre links if required as well.

For redundancy I'd go NextDC M2 as that is on a totally separate power grid or even Micron21 for primary both these facilities are tier 4 so you can't go wrong.

I certainly know Micron21 have backup facilities that work at any rate, I actually thought Vocus had sold and was getting out of the Datacentre space altogether.

The Equinix facility in port Melbourne is decent as well.

I'd go Equinix or NextDc or both before Vocus for reliability. The Vocus Fibre network is still ok as you get a few months free if you contract for long enough and it does the job.

Regards Chad.

From: Luke Thompson <luke.t at tncrew.com.au<mailto:luke.t at tncrew.com.au>>

To: Damien Gardner Jnr <rendrag at rendrag.net<mailto:rendrag at rendrag.net>>, Chris Wright

                <chris at securityshift.com<mailto:chris at securityshift.com>>

Cc: "ausnog at ausnog.net<mailto:ausnog at ausnog.net>" <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>

Subject: Re: [AusNOG] Aussie Broadband Outage

Message-ID:

                <186d82f4918.283d.934dbec5914d2454830359510b9fa213 at tncrew.com.au<mailto:186d82f4918.283d.934dbec5914d2454830359510b9fa213 at tncrew.com.au>>

Content-Type: text/plain; charset="utf-8"; Format="flowed"



Given the commonality of outage type, it's smelling like civil or grid problems isn't it?





Vocus and Equinix losing power over both rails, optical links dropping, etc

sounds like a major problem in the earth or at a dist/sub station. Curious

to learn the root cause.






Chad Kelly
Manager
CPK Web Services
Phone 03 52730246
Web https://www.cpkws.com.au
[cid:image001.jpg at 01D955C0.0AC33F70]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230313/c7aca34f/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 97720 bytes
Desc: image001.jpg
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230313/c7aca34f/attachment-0001.jpg>

From chad at cpkws.com.au  Mon Mar 13 18:30:34 2023
From: chad at cpkws.com.au (Chad Kelly)
Date: Mon, 13 Mar 2023 07:30:34 +0000
Subject: [AusNOG] Vocus Datacentre outage.
Message-ID: <ME3PR01MB6878081AEFEDA25E520E1A3BFDB99@ME3PR01MB6878.ausprd01.prod.outlook.com>

Thought I would create a new thread on this, as the issue wasn't related to AussieBB.
It is looking like this Melbourne outage is similar to the Sydney outage back in 2018, in that it appears to have been yet again caused by an issue with their UPS systems yet again.
I'll leave this hear https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/ and this article on today's outage as well.
https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987

Regards Chad.



Chad Kelly
Manager
CPK Web Services
Phone 03 52730246
Web https://www.cpkws.com.au
[cid:image001.jpg at 01D955D7.3EA59970]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230313/e7a493d0/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 97720 bytes
Desc: image001.jpg
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230313/e7a493d0/attachment-0001.jpg>

From dazzagibbs at gmail.com  Tue Mar 14 10:08:03 2023
From: dazzagibbs at gmail.com (DaZZa)
Date: Tue, 14 Mar 2023 10:08:03 +1100
Subject: [AusNOG] Vocus Datacentre outage.
In-Reply-To: <ME3PR01MB6878081AEFEDA25E520E1A3BFDB99@ME3PR01MB6878.ausprd01.prod.outlook.com>
References: <ME3PR01MB6878081AEFEDA25E520E1A3BFDB99@ME3PR01MB6878.ausprd01.prod.outlook.com>
Message-ID: <CAMpV8Mpv98O-cqpGz+5SVRBsobsZzWAT0Rhy8UdiSQ6fUEiWjw@mail.gmail.com>

Doesn't say a whole lot about Vocus' data center design methodology, nor
their periodic contingency testing.

They're apparently replacing the entire UPS on Friday - guess we can all
expect more outages then when they stuff that up.

D

On Mon, 13 Mar 2023 at 18:30, Chad Kelly <chad at cpkws.com.au> wrote:

> Thought I would create a new thread on this, as the issue wasn?t related
> to AussieBB.
>
> It is looking like this Melbourne outage is similar to the Sydney outage
> back in 2018, in that it appears to have been yet again caused by an issue
> with their UPS systems yet again.
>
> I?ll leave this hear
> https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/
> and this article on today?s outage as well.
>
>
> https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987
>
>
>
> Regards Chad.
>
>
>
>
>
>
>
> Chad Kelly
>
> Manager
>
> CPK Web Services
>
> Phone 03 52730246
>
> Web https://www.cpkws.com.au
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>


-- 
veg?e?tar?i?an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230314/9ada06b3/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 97720 bytes
Desc: not available
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230314/9ada06b3/attachment-0001.jpg>

From brad at bradleyamm.com  Tue Mar 14 11:06:53 2023
From: brad at bradleyamm.com (Bradley Amm)
Date: Tue, 14 Mar 2023 08:06:53 +0800
Subject: [AusNOG] Vocus Datacentre outage.
In-Reply-To: <CAMpV8Mpv98O-cqpGz+5SVRBsobsZzWAT0Rhy8UdiSQ6fUEiWjw@mail.gmail.com>
References: <CAMpV8Mpv98O-cqpGz+5SVRBsobsZzWAT0Rhy8UdiSQ6fUEiWjw@mail.gmail.com>
Message-ID: <A7475E5D-495B-4B4B-91C1-169140FEACE1@bradleyamm.com>

An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230314/d8e19c18/attachment.htm>

From matt at spectrum.com.au  Tue Mar 14 11:18:27 2023
From: matt at spectrum.com.au (Matt Perkins)
Date: Tue, 14 Mar 2023 11:18:27 +1100
Subject: [AusNOG] Vocus Datacentre outage.
In-Reply-To: <A7475E5D-495B-4B4B-91C1-169140FEACE1@bradleyamm.com>
References: <CAMpV8Mpv98O-cqpGz+5SVRBsobsZzWAT0Rhy8UdiSQ6fUEiWjw@mail.gmail.com>
 <A7475E5D-495B-4B4B-91C1-169140FEACE1@bradleyamm.com>
Message-ID: <18719e27-053c-2ec2-f48e-2b69d5114716@spectrum.com.au>

Yea because Eqinix has never had a power outage. :)

I think it's a little premature to judge at the moment. Doesn't sound 
good based on the current info but we have all had bad days im sure.

Matt


On 14/3/2023 11:06 am, Bradley Amm wrote:
> Friends like let friends put gear in a Vocus dtacenter. Only NextDC or 
> Equinix :)
>
> Sent from my iPhone
>
>> On 14 Mar 2023, at 7:08 am, DaZZa <dazzagibbs at gmail.com> wrote:
>>
>> ?
>> Doesn't say a whole lot about Vocus' data center design methodology, 
>> nor their periodic contingency testing.
>>
>> They're apparently replacing the entire UPS on Friday - guess we can 
>> all expect more outages then when they stuff that up.
>>
>> D
>>
>> On Mon, 13 Mar 2023 at 18:30, Chad Kelly <chad at cpkws.com.au> wrote:
>>
>>     Thought I would create a new thread on this, as the issue wasn?t
>>     related to AussieBB.
>>
>>     It is looking like this Melbourne outage is similar to the Sydney
>>     outage back in 2018, in that it appears to have been yet again
>>     caused by an issue with their UPS systems yet again.
>>
>>     I?ll leave this hear
>>     https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/
>>     and this article on today?s outage as well.
>>
>>     https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987
>>
>>     Regards Chad.
>>
>>     Chad Kelly
>>
>>     Manager
>>
>>     CPK Web Services
>>
>>     Phone 03 52730246
>>
>>     Web https://www.cpkws.com.au
>>
>>     <image001.jpg>
>>
>>     _______________________________________________
>>     AusNOG mailing list
>>     AusNOG at lists.ausnog.net
>>     https://lists.ausnog.net/mailman/listinfo/ausnog
>>
>>
>>
>> -- 
>> veg?e?tar?i?an:
>> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog

-- 
** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL

/* Matt Perkins
         Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.
         Office 1300 133 299matt at spectrum.com.au
         ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000

*/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230314/742e0ceb/attachment.htm>

From Nathan.Brookfield at iperium.com.au  Tue Mar 14 11:26:48 2023
From: Nathan.Brookfield at iperium.com.au (Nathan Brookfield)
Date: Tue, 14 Mar 2023 00:26:48 +0000
Subject: [AusNOG] Vocus Datacentre outage.
In-Reply-To: <18719e27-053c-2ec2-f48e-2b69d5114716@spectrum.com.au>
References: <CAMpV8Mpv98O-cqpGz+5SVRBsobsZzWAT0Rhy8UdiSQ6fUEiWjw@mail.gmail.com>
 <A7475E5D-495B-4B4B-91C1-169140FEACE1@bradleyamm.com>
 <18719e27-053c-2ec2-f48e-2b69d5114716@spectrum.com.au>
Message-ID: <MEYP282MB2520B620B1438273E3A1009CDBBE9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>

I?ve never had an A+B power rail failure at Equinix in 18 years I?ve been in there facilities, TOUCH WOOD!!!

Nathan Brookfield | VK2NAB
General Manager

p: 1300 592 330 | m: 0412 266 008
e: Nathan.brookfield at iperium.com.au<mailto:Nathan.brookfield at iperium.com.au> | w: iperium.com.au

Suite 4.02, 189 Kent Street Sydney NSW 2000
[cid:image001.png at 01D95667.DCB722B0]
Your Telco Team
DISCLAIMER: This document is intended solely for the named addressee. This electronic communication, which includes any files or attachments thereto, contains proprietary or confidential information and may be privileged and otherwise protected under copyright or other applicable intellectual property laws. The use, disclosure, copying or distribution of any of the information contained in this document, by any person other than the addressee, is strictly prohibited. If you received this document in error, please contact the sender immediately and delete all the material from any computer. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you. Any views or opinions presented are solely those of the author and do not necessarily represent those of Iperium.

WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. Iperium accepts no liability for any damage caused by any virus transmitted by this email.
From: AusNOG <ausnog-bounces at lists.ausnog.net> On Behalf Of Matt Perkins
Sent: Tuesday, 14 March 2023 11:18 AM
To: ausnog at lists.ausnog.net
Subject: Re: [AusNOG] Vocus Datacentre outage.


Yea because Eqinix has never had a power outage. :)

I think it's a little premature to judge at the moment. Doesn't sound good based on the current info but we have all had bad days im sure.

Matt


On 14/3/2023 11:06 am, Bradley Amm wrote:
Friends like let friends put gear in a Vocus dtacenter. Only NextDC or Equinix :)
Sent from my iPhone


On 14 Mar 2023, at 7:08 am, DaZZa <dazzagibbs at gmail.com><mailto:dazzagibbs at gmail.com> wrote:
?
Doesn't say a whole lot about Vocus' data center design methodology, nor their periodic contingency testing.

They're apparently replacing the entire UPS on Friday - guess we can all expect more outages then when they stuff that up.

D

On Mon, 13 Mar 2023 at 18:30, Chad Kelly <chad at cpkws.com.au<mailto:chad at cpkws.com.au>> wrote:
Thought I would create a new thread on this, as the issue wasn?t related to AussieBB.
It is looking like this Melbourne outage is similar to the Sydney outage back in 2018, in that it appears to have been yet again caused by an issue with their UPS systems yet again.
I?ll leave this hear https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/ and this article on today?s outage as well.
https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987

Regards Chad.



Chad Kelly
Manager
CPK Web Services
Phone 03 52730246
Web https://www.cpkws.com.au
<image001.jpg>

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
https://lists.ausnog.net/mailman/listinfo/ausnog


--
veg?e?tar?i?an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
https://lists.ausnog.net/mailman/listinfo/ausnog



_______________________________________________

AusNOG mailing list

AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>

https://lists.ausnog.net/mailman/listinfo/ausnog

--

** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL



/* Matt Perkins

        Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.

        Office 1300 133 299     matt at spectrum.com.au<mailto:matt at spectrum.com.au>

        ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000



*/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230314/908bbfb5/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 4595 bytes
Desc: image001.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230314/908bbfb5/attachment.png>

From matt at spectrum.com.au  Tue Mar 14 11:30:28 2023
From: matt at spectrum.com.au (Matt Perkins)
Date: Tue, 14 Mar 2023 11:30:28 +1100
Subject: [AusNOG] Vocus Datacentre outage.
In-Reply-To: <MEYP282MB2520B620B1438273E3A1009CDBBE9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
References: <CAMpV8Mpv98O-cqpGz+5SVRBsobsZzWAT0Rhy8UdiSQ6fUEiWjw@mail.gmail.com>
 <A7475E5D-495B-4B4B-91C1-169140FEACE1@bradleyamm.com>
 <18719e27-053c-2ec2-f48e-2b69d5114716@spectrum.com.au>
 <MEYP282MB2520B620B1438273E3A1009CDBBE9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
Message-ID: <f35b78ea-6e27-1eb1-5314-20fde831056a@spectrum.com.au>

SY1 had A or B fail a few years back if my memory serves me correctly.? 
But I was referring? London and Manchester which both had total outages 
in recent years.

Matt.


On 14/3/2023 11:26 am, Nathan Brookfield wrote:
>
> I?ve never had an A+B power rail failure at Equinix in 18 years I?ve 
> been in there facilities, TOUCH WOOD!!!
>
> *Nathan Brookfield | VK2NAB
> *General Manager
>
> *p*: 1300 592 330 | *m*: 0412 266 008
> *e*: Nathan.brookfield at iperium.com.au | *w*:?iperium.com.au
>
> Suite 4.02, 189 Kent Street Sydney NSW 2000**
>
>
> *Your Telco Team*
>
> DISCLAIMER: This document is intended solely for the named addressee. 
> This electronic communication, which includes any files or attachments 
> thereto, contains proprietary or confidential information and may be 
> privileged and otherwise protected under copyright or other applicable 
> intellectual property laws. The use, disclosure, copying or 
> distribution of any of the information contained in this document, by 
> any person other than the addressee, is strictly prohibited. If you 
> received this document in error, please contact the sender immediately 
> and delete all the material from any computer. Confidentiality and 
> legal privilege are not waived or lost by reason of mistaken delivery 
> to you. Any views or opinions presented are solely those of the author 
> and do not necessarily represent those of Iperium.
>
> WARNING: Computer viruses can be transmitted via email. The recipient 
> should check this email and any attachments for the presence of 
> viruses. Iperium accepts no liability for any damage caused by any 
> virus transmitted by this email.
>
> *From:*AusNOG <ausnog-bounces at lists.ausnog.net> *On Behalf Of *Matt 
> Perkins
> *Sent:* Tuesday, 14 March 2023 11:18 AM
> *To:* ausnog at lists.ausnog.net
> *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>
> Yea because Eqinix has never had a power outage. :)
>
> I think it's a little premature to judge at the moment. Doesn't sound 
> good based on the current info but we have all had bad days im sure.
>
> Matt
>
> On 14/3/2023 11:06 am, Bradley Amm wrote:
>
>     Friends like let friends put gear in a Vocus dtacenter. Only
>     NextDC or Equinix :)
>
>     Sent from my iPhone
>
>
>
>         On 14 Mar 2023, at 7:08 am, DaZZa <dazzagibbs at gmail.com>
>         <mailto:dazzagibbs at gmail.com> wrote:
>
>         ?
>
>         Doesn't say a whole lot about Vocus' data center design
>         methodology, nor their periodic contingency testing.
>
>         They're apparently replacing the entire UPS on Friday - guess
>         we can all expect more outages then when they stuff that up.
>
>         D
>
>         On Mon, 13 Mar 2023 at 18:30, Chad Kelly <chad at cpkws.com.au>
>         wrote:
>
>             Thought I would create a new thread on this, as the issue
>             wasn?t related to AussieBB.
>
>             It is looking like this Melbourne outage is similar to the
>             Sydney outage back in 2018, in that it appears to have
>             been yet again caused by an issue with their UPS systems
>             yet again.
>
>             I?ll leave this hear
>             https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/
>             and this article on today?s outage as well.
>
>             https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987
>
>             Regards Chad.
>
>             Chad Kelly
>
>             Manager
>
>             CPK Web Services
>
>             Phone 03 52730246
>
>             Web https://www.cpkws.com.au
>
>             <image001.jpg>
>
>             _______________________________________________
>             AusNOG mailing list
>             AusNOG at lists.ausnog.net
>             https://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
>         -- 
>
>         veg?e?tar?i?an:
>         Ancient tribal slang for the village idiot who can't hunt,
>         fish or ride
>
>         _______________________________________________
>         AusNOG mailing list
>         AusNOG at lists.ausnog.net
>         https://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
>     _______________________________________________
>
>     AusNOG mailing list
>
>     AusNOG at lists.ausnog.net
>
>     https://lists.ausnog.net/mailman/listinfo/ausnog
>
> -- 
> ** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL
> /* Matt Perkins
>  ??????? Direct 02 8916 8101???? Spectrum Networks Ptd. Ltd.
>  ??????? Office 1300 133 299matt at spectrum.com.au
>  ??????? ABN 66 090 112 913????? Level 6, 350 George Street Sydney 2000
> */

-- 
** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL

/* Matt Perkins
         Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.
         Office 1300 133 299matt at spectrum.com.au
         ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000

*/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230314/958cefeb/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 4595 bytes
Desc: not available
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230314/958cefeb/attachment.png>

From Nathan.Brookfield at iperium.com.au  Tue Mar 14 11:40:01 2023
From: Nathan.Brookfield at iperium.com.au (Nathan Brookfield)
Date: Tue, 14 Mar 2023 00:40:01 +0000
Subject: [AusNOG] Vocus Datacentre outage.
In-Reply-To: <f35b78ea-6e27-1eb1-5314-20fde831056a@spectrum.com.au>
References: <CAMpV8Mpv98O-cqpGz+5SVRBsobsZzWAT0Rhy8UdiSQ6fUEiWjw@mail.gmail.com>
 <A7475E5D-495B-4B4B-91C1-169140FEACE1@bradleyamm.com>
 <18719e27-053c-2ec2-f48e-2b69d5114716@spectrum.com.au>
 <MEYP282MB2520B620B1438273E3A1009CDBBE9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <f35b78ea-6e27-1eb1-5314-20fde831056a@spectrum.com.au>
Message-ID: <MEYP282MB2520A3C7BCA133967365ED77DBBE9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>

Have 100% had A and B fail separately at SY1 but never both, thankfully!!!

Could have been part of the facility though so don?t quote me.

UK on the other hand, fail!

From: Matt Perkins <matt at spectrum.com.au>
Sent: Tuesday, 14 March 2023 11:30 AM
To: Nathan Brookfield <Nathan.Brookfield at iperium.com.au>; ausnog at lists.ausnog.net
Subject: Re: [AusNOG] Vocus Datacentre outage.


SY1 had A or B fail a few years back if my memory serves me correctly.  But I was referring  London and Manchester which both had total outages in recent years.

Matt.


On 14/3/2023 11:26 am, Nathan Brookfield wrote:
I?ve never had an A+B power rail failure at Equinix in 18 years I?ve been in there facilities, TOUCH WOOD!!!

Nathan Brookfield | VK2NAB
General Manager

p: 1300 592 330 | m: 0412 266 008
e: Nathan.brookfield at iperium.com.au<mailto:Nathan.brookfield at iperium.com.au> | w: iperium.com.au

Suite 4.02, 189 Kent Street Sydney NSW 2000
[cid:image001.png at 01D95669.B5A95AB0]
Your Telco Team
DISCLAIMER: This document is intended solely for the named addressee. This electronic communication, which includes any files or attachments thereto, contains proprietary or confidential information and may be privileged and otherwise protected under copyright or other applicable intellectual property laws. The use, disclosure, copying or distribution of any of the information contained in this document, by any person other than the addressee, is strictly prohibited. If you received this document in error, please contact the sender immediately and delete all the material from any computer. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you. Any views or opinions presented are solely those of the author and do not necessarily represent those of Iperium.

WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. Iperium accepts no liability for any damage caused by any virus transmitted by this email.
From: AusNOG <ausnog-bounces at lists.ausnog.net><mailto:ausnog-bounces at lists.ausnog.net> On Behalf Of Matt Perkins
Sent: Tuesday, 14 March 2023 11:18 AM
To: ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] Vocus Datacentre outage.


Yea because Eqinix has never had a power outage. :)

I think it's a little premature to judge at the moment. Doesn't sound good based on the current info but we have all had bad days im sure.

Matt


On 14/3/2023 11:06 am, Bradley Amm wrote:
Friends like let friends put gear in a Vocus dtacenter. Only NextDC or Equinix :)
Sent from my iPhone



On 14 Mar 2023, at 7:08 am, DaZZa <dazzagibbs at gmail.com><mailto:dazzagibbs at gmail.com> wrote:
?
Doesn't say a whole lot about Vocus' data center design methodology, nor their periodic contingency testing.

They're apparently replacing the entire UPS on Friday - guess we can all expect more outages then when they stuff that up.

D

On Mon, 13 Mar 2023 at 18:30, Chad Kelly <chad at cpkws.com.au<mailto:chad at cpkws.com.au>> wrote:
Thought I would create a new thread on this, as the issue wasn?t related to AussieBB.
It is looking like this Melbourne outage is similar to the Sydney outage back in 2018, in that it appears to have been yet again caused by an issue with their UPS systems yet again.
I?ll leave this hear https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/ and this article on today?s outage as well.
https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987

Regards Chad.



Chad Kelly
Manager
CPK Web Services
Phone 03 52730246
Web https://www.cpkws.com.au
<image001.jpg>

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
https://lists.ausnog.net/mailman/listinfo/ausnog


--
veg?e?tar?i?an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
https://lists.ausnog.net/mailman/listinfo/ausnog




_______________________________________________

AusNOG mailing list

AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>

https://lists.ausnog.net/mailman/listinfo/ausnog

--

** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL



/* Matt Perkins

        Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.

        Office 1300 133 299     matt at spectrum.com.au<mailto:matt at spectrum.com.au>

        ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000



*/

--

** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL



/* Matt Perkins

        Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.

        Office 1300 133 299     matt at spectrum.com.au<mailto:matt at spectrum.com.au>

        ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000



*/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230314/7cfe398d/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 4595 bytes
Desc: image001.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230314/7cfe398d/attachment-0001.png>

From david.dawson at apnic.net  Tue Mar 14 11:43:47 2023
From: david.dawson at apnic.net (Dave Dawson)
Date: Tue, 14 Mar 2023 00:43:47 +0000
Subject: [AusNOG] ISIF Asia 2023 grant applications now open
Message-ID: <SY4P282MB37326A4B06F856F441A4D7888EBE9@SY4P282MB3732.AUSP282.PROD.OUTLOOK.COM>

Hi all -


Just letting you know that the ISIF Asia grants are open. We'd love to see applications from AusNOG members. Applications close April 30. Info below.

I am happy to field queries, but also check the links below and our FAQs as they might have covered it: https://isif.asia/faq/

Thanks,
David Dawson
(APNIC Foundation comms)
--

ISIF Asia is opening its 2023 grant cycle for applications.

ISIF Asia Grants are focused on network operations and Internet development in the Asia Pacific. Almost any kind of organization can apply. Please share the news with your networks, we have a variety of different grants on offer.

The application link: https://isif.asia/apply/

Grants are across three broad areas:

Inclusion: Initiatives that help ensure everyone has meaningful access to the Internet, online applications and services.

Infrastructure: Initiatives to increase Internet speed, reduce maintenance and operational costs, and improve reliability and/or security.

Knowledge: Initiatives that develop technical capacity and/or research around Internet network operations for technical and non-technical audiences.

Within these three program areas, there are additional types of grant:

Ian Peter Grants for Internet and the Environment

The Ian Peter Grants for Internet and the Environment can be across any of the three program areas and are to support innovative solutions to the environmental impact of the Internet. They commemorate the life and work of Ian Peter, Internet pioneer and environmental activist.

IPv6 Deployment Grants

ISIF Asia's IPv6 Deployment Grants are also open for organizations planning to deploy IPv6 in the Asia Pacific. IPv6 Deployment Grants range between USD 30,000 and USD 250,000. These are open all year round and are part of the Infrastructure program.

Get in your applications now! Applications (excluding the IPv6 Deployment grants) close 30 April 2023, 11:59 AM (UTC+10).

More information is available on the ISIF Asia blog.

https://isif.asia/2023-isif-asia-grants-now-open-for-applications/


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230314/d6805950/attachment.htm>

From chris at thesysadmin.dev  Tue Mar 14 11:50:27 2023
From: chris at thesysadmin.dev (Christopher Hawker)
Date: Tue, 14 Mar 2023 00:50:27 +0000
Subject: [AusNOG] Vocus Datacentre outage.
In-Reply-To: <MEYP282MB2520A3C7BCA133967365ED77DBBE9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
References: <CAMpV8Mpv98O-cqpGz+5SVRBsobsZzWAT0Rhy8UdiSQ6fUEiWjw@mail.gmail.com>
 <A7475E5D-495B-4B4B-91C1-169140FEACE1@bradleyamm.com>
 <18719e27-053c-2ec2-f48e-2b69d5114716@spectrum.com.au>
 <MEYP282MB2520B620B1438273E3A1009CDBBE9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <f35b78ea-6e27-1eb1-5314-20fde831056a@spectrum.com.au>
 <MEYP282MB2520A3C7BCA133967365ED77DBBE9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
Message-ID: <SYBP282MB0538A5F3C118B6D6A4ED85F5AFBE9@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM>

SY1 apparently experienced a full power outage back in July 2009, according to AusNOG Archives. Some reported full outages, some partial. Was not a DC fault though, looks like a much larger grid issue where a fault at the Bayswater Power Station in the Hunter Valley (which affected generators in QLD and VIC) caused about 1000MW to be dropped from the grid, resulting in rolling blackouts across QLD, NSW, TAS, SA and VIC.

https://lists.ausnog.net/pipermail/ausnog/2009-July/003222.html

CH.
________________________________
From: AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Nathan Brookfield <Nathan.Brookfield at iperium.com.au>
Sent: Tuesday, March 14, 2023 11:40 AM
To: Matt Perkins <matt at spectrum.com.au>; ausnog at lists.ausnog.net <ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] Vocus Datacentre outage.


Have 100% had A and B fail separately at SY1 but never both, thankfully!!!



Could have been part of the facility though so don?t quote me.



UK on the other hand, fail!



From: Matt Perkins <matt at spectrum.com.au>
Sent: Tuesday, 14 March 2023 11:30 AM
To: Nathan Brookfield <Nathan.Brookfield at iperium.com.au>; ausnog at lists.ausnog.net
Subject: Re: [AusNOG] Vocus Datacentre outage.



SY1 had A or B fail a few years back if my memory serves me correctly.  But I was referring  London and Manchester which both had total outages in recent years.

Matt.



On 14/3/2023 11:26 am, Nathan Brookfield wrote:

I?ve never had an A+B power rail failure at Equinix in 18 years I?ve been in there facilities, TOUCH WOOD!!!



Nathan Brookfield | VK2NAB
General Manager

p: 1300 592 330 | m: 0412 266 008
e: Nathan.brookfield at iperium.com.au<mailto:Nathan.brookfield at iperium.com.au> | w: iperium.com.au

Suite 4.02, 189 Kent Street Sydney NSW 2000

[cid:image001.png at 01D95669.B5A95AB0]
Your Telco Team

DISCLAIMER: This document is intended solely for the named addressee. This electronic communication, which includes any files or attachments thereto, contains proprietary or confidential information and may be privileged and otherwise protected under copyright or other applicable intellectual property laws. The use, disclosure, copying or distribution of any of the information contained in this document, by any person other than the addressee, is strictly prohibited. If you received this document in error, please contact the sender immediately and delete all the material from any computer. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you. Any views or opinions presented are solely those of the author and do not necessarily represent those of Iperium.

WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. Iperium accepts no liability for any damage caused by any virus transmitted by this email.

From: AusNOG <ausnog-bounces at lists.ausnog.net><mailto:ausnog-bounces at lists.ausnog.net> On Behalf Of Matt Perkins
Sent: Tuesday, 14 March 2023 11:18 AM
To: ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] Vocus Datacentre outage.



Yea because Eqinix has never had a power outage. :)

I think it's a little premature to judge at the moment. Doesn't sound good based on the current info but we have all had bad days im sure.

Matt



On 14/3/2023 11:06 am, Bradley Amm wrote:

Friends like let friends put gear in a Vocus dtacenter. Only NextDC or Equinix :)

Sent from my iPhone




On 14 Mar 2023, at 7:08 am, DaZZa <dazzagibbs at gmail.com><mailto:dazzagibbs at gmail.com> wrote:

?

Doesn't say a whole lot about Vocus' data center design methodology, nor their periodic contingency testing.



They're apparently replacing the entire UPS on Friday - guess we can all expect more outages then when they stuff that up.



D



On Mon, 13 Mar 2023 at 18:30, Chad Kelly <chad at cpkws.com.au<mailto:chad at cpkws.com.au>> wrote:

Thought I would create a new thread on this, as the issue wasn?t related to AussieBB.

It is looking like this Melbourne outage is similar to the Sydney outage back in 2018, in that it appears to have been yet again caused by an issue with their UPS systems yet again.

I?ll leave this hear https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/ and this article on today?s outage as well.

https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987



Regards Chad.







Chad Kelly

Manager

CPK Web Services

Phone 03 52730246

Web https://www.cpkws.com.au

<image001.jpg>



_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
https://lists.ausnog.net/mailman/listinfo/ausnog


--

veg?e?tar?i?an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
https://lists.ausnog.net/mailman/listinfo/ausnog




_______________________________________________

AusNOG mailing list

AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>

https://lists.ausnog.net/mailman/listinfo/ausnog

--

** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL



/* Matt Perkins

        Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.

        Office 1300 133 299     matt at spectrum.com.au<mailto:matt at spectrum.com.au>

        ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000



*/

--

** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL



/* Matt Perkins

        Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.

        Office 1300 133 299     matt at spectrum.com.au<mailto:matt at spectrum.com.au>

        ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000



*/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230314/4d540eb5/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 4595 bytes
Desc: image001.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230314/4d540eb5/attachment.png>

From Nathan.Brookfield at iperium.com.au  Tue Mar 14 11:53:36 2023
From: Nathan.Brookfield at iperium.com.au (Nathan Brookfield)
Date: Tue, 14 Mar 2023 00:53:36 +0000
Subject: [AusNOG] Vocus Datacentre outage.
In-Reply-To: <SYBP282MB0538A5F3C118B6D6A4ED85F5AFBE9@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM>
References: <CAMpV8Mpv98O-cqpGz+5SVRBsobsZzWAT0Rhy8UdiSQ6fUEiWjw@mail.gmail.com>
 <A7475E5D-495B-4B4B-91C1-169140FEACE1@bradleyamm.com>
 <18719e27-053c-2ec2-f48e-2b69d5114716@spectrum.com.au>
 <MEYP282MB2520B620B1438273E3A1009CDBBE9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <f35b78ea-6e27-1eb1-5314-20fde831056a@spectrum.com.au>
 <MEYP282MB2520A3C7BCA133967365ED77DBBE9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <SYBP282MB0538A5F3C118B6D6A4ED85F5AFBE9@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM>
Message-ID: <8D195CA1-4E67-46F4-BF39-C9AED0616977@iperium.com.au>

I knew I didn?t remember it for a reason ?

<https://lists.ausnog.net/pipermail/ausnog/2009-July/003239.html>
[AusNOG] Equinix Outage<https://lists.ausnog.net/pipermail/ausnog/2009-July/003239.html>
lists.ausnog.net<https://lists.ausnog.net/pipermail/ausnog/2009-July/003239.html>
[X]<https://lists.ausnog.net/pipermail/ausnog/2009-July/003239.html>

On 14 Mar 2023, at 10:50, Christopher Hawker <chris at thesysadmin.dev> wrote:

?
SY1 apparently experienced a full power outage back in July 2009, according to AusNOG Archives. Some reported full outages, some partial. Was not a DC fault though, looks like a much larger grid issue where a fault at the Bayswater Power Station in the Hunter Valley (which affected generators in QLD and VIC) caused about 1000MW to be dropped from the grid, resulting in rolling blackouts across QLD, NSW, TAS, SA and VIC.

https://lists.ausnog.net/pipermail/ausnog/2009-July/003222.html

CH.
________________________________
From: AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Nathan Brookfield <Nathan.Brookfield at iperium.com.au>
Sent: Tuesday, March 14, 2023 11:40 AM
To: Matt Perkins <matt at spectrum.com.au>; ausnog at lists.ausnog.net <ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] Vocus Datacentre outage.


Have 100% had A and B fail separately at SY1 but never both, thankfully!!!



Could have been part of the facility though so don?t quote me.



UK on the other hand, fail!



From: Matt Perkins <matt at spectrum.com.au>
Sent: Tuesday, 14 March 2023 11:30 AM
To: Nathan Brookfield <Nathan.Brookfield at iperium.com.au>; ausnog at lists.ausnog.net
Subject: Re: [AusNOG] Vocus Datacentre outage.



SY1 had A or B fail a few years back if my memory serves me correctly.  But I was referring  London and Manchester which both had total outages in recent years.

Matt.



On 14/3/2023 11:26 am, Nathan Brookfield wrote:

I?ve never had an A+B power rail failure at Equinix in 18 years I?ve been in there facilities, TOUCH WOOD!!!



Nathan Brookfield | VK2NAB
General Manager

p: 1300 592 330 | m: 0412 266 008
e: Nathan.brookfield at iperium.com.au<mailto:Nathan.brookfield at iperium.com.au> | w: iperium.com.au

Suite 4.02, 189 Kent Street Sydney NSW 2000

<image001.png>

Your Telco Team

DISCLAIMER: This document is intended solely for the named addressee. This electronic communication, which includes any files or attachments thereto, contains proprietary or confidential information and may be privileged and otherwise protected under copyright or other applicable intellectual property laws. The use, disclosure, copying or distribution of any of the information contained in this document, by any person other than the addressee, is strictly prohibited. If you received this document in error, please contact the sender immediately and delete all the material from any computer. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you. Any views or opinions presented are solely those of the author and do not necessarily represent those of Iperium.

WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. Iperium accepts no liability for any damage caused by any virus transmitted by this email.

From: AusNOG <ausnog-bounces at lists.ausnog.net><mailto:ausnog-bounces at lists.ausnog.net> On Behalf Of Matt Perkins
Sent: Tuesday, 14 March 2023 11:18 AM
To: ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] Vocus Datacentre outage.



Yea because Eqinix has never had a power outage. :)

I think it's a little premature to judge at the moment. Doesn't sound good based on the current info but we have all had bad days im sure.

Matt



On 14/3/2023 11:06 am, Bradley Amm wrote:

Friends like let friends put gear in a Vocus dtacenter. Only NextDC or Equinix :)

Sent from my iPhone




On 14 Mar 2023, at 7:08 am, DaZZa <dazzagibbs at gmail.com><mailto:dazzagibbs at gmail.com> wrote:

?

Doesn't say a whole lot about Vocus' data center design methodology, nor their periodic contingency testing.



They're apparently replacing the entire UPS on Friday - guess we can all expect more outages then when they stuff that up.



D



On Mon, 13 Mar 2023 at 18:30, Chad Kelly <chad at cpkws.com.au<mailto:chad at cpkws.com.au>> wrote:

Thought I would create a new thread on this, as the issue wasn?t related to AussieBB.

It is looking like this Melbourne outage is similar to the Sydney outage back in 2018, in that it appears to have been yet again caused by an issue with their UPS systems yet again.

I?ll leave this hear https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/ and this article on today?s outage as well.

https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987



Regards Chad.







Chad Kelly

Manager

CPK Web Services

Phone 03 52730246

Web https://www.cpkws.com.au

<image001.jpg>



_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
https://lists.ausnog.net/mailman/listinfo/ausnog


--

veg?e?tar?i?an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
https://lists.ausnog.net/mailman/listinfo/ausnog




_______________________________________________

AusNOG mailing list

AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>

https://lists.ausnog.net/mailman/listinfo/ausnog

--

** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL



/* Matt Perkins

        Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.

        Office 1300 133 299     matt at spectrum.com.au<mailto:matt at spectrum.com.au>

        ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000



*/

--

** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL



/* Matt Perkins

        Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.

        Office 1300 133 299     matt at spectrum.com.au<mailto:matt at spectrum.com.au>

        ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000



*/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230314/7599e9df/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 4595 bytes
Desc: image001.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230314/7599e9df/attachment.png>

From matt at spectrum.com.au  Tue Mar 14 12:17:46 2023
From: matt at spectrum.com.au (Matt Perkins)
Date: Tue, 14 Mar 2023 12:17:46 +1100
Subject: [AusNOG] Vocus Datacentre outage.
In-Reply-To: <SYBP282MB0538A5F3C118B6D6A4ED85F5AFBE9@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM>
References: <CAMpV8Mpv98O-cqpGz+5SVRBsobsZzWAT0Rhy8UdiSQ6fUEiWjw@mail.gmail.com>
 <A7475E5D-495B-4B4B-91C1-169140FEACE1@bradleyamm.com>
 <18719e27-053c-2ec2-f48e-2b69d5114716@spectrum.com.au>
 <MEYP282MB2520B620B1438273E3A1009CDBBE9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <f35b78ea-6e27-1eb1-5314-20fde831056a@spectrum.com.au>
 <MEYP282MB2520A3C7BCA133967365ED77DBBE9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <SYBP282MB0538A5F3C118B6D6A4ED85F5AFBE9@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM>
Message-ID: <f59b62f8-4ab8-19cc-9890-0f1e45a13eb7@spectrum.com.au>

The questions left to be answered for Vocus is why did it take 12 hours 
to restore one rail. Where was the static bypass and failing that where 
was the maintenance bypass.? Im guessing that's what it's on maintenance 
bypass now and the slow action may have something to do with the public 
holiday in Vic..


I shouldn't speculate. But it is fun.


 ?Something seriously let go in that UPS. If i had to put my electrical 
engineers hat on I would say their load capacitor maintenance was not 
done and let go. Or someone was working upstream and broke the 
natural/earth bond. But there's a lot of parts in a system like that , 
most modern DC UPS setups are fairly robust.? I once had 3 strings of 
batteries catch fire and melt down and the UPS didn't miss a beat other 
then spewing a lot of alarms. We moved strings around on the fly and 
nobody even noticed.


The other things that seems odd is why did an A failure trip B. Sounds 
like a single supply is at surge/break capacity. Also a worry.


But in the end we all have redundancy around 530 Collins right ? Right ?


Matt



On 14/3/2023 11:50 am, Christopher Hawker wrote:
> SY1 apparently experienced a full power outage back in July 2009, 
> according to AusNOG Archives. Some reported full outages, some 
> partial. Was not a DC fault though, looks like a much larger grid 
> issue where a fault at the Bayswater Power Station in the Hunter 
> Valley (which affected generators in QLD and VIC) caused about 1000MW 
> to be dropped from the grid, resulting in rolling blackouts across 
> QLD, NSW, TAS, SA and VIC.
>
> https://lists.ausnog.net/pipermail/ausnog/2009-July/003222.html
>
> CH.
> ------------------------------------------------------------------------
> *From:* AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Nathan 
> Brookfield <Nathan.Brookfield at iperium.com.au>
> *Sent:* Tuesday, March 14, 2023 11:40 AM
> *To:* Matt Perkins <matt at spectrum.com.au>; ausnog at lists.ausnog.net 
> <ausnog at lists.ausnog.net>
> *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>
> Have 100% had A and B fail separately at SY1 but never both, thankfully!!!
>
> Could have been part of the facility though so don?t quote me.
>
> UK on the other hand, fail!
>
> *From:*Matt Perkins <matt at spectrum.com.au>
> *Sent:* Tuesday, 14 March 2023 11:30 AM
> *To:* Nathan Brookfield <Nathan.Brookfield at iperium.com.au>; 
> ausnog at lists.ausnog.net
> *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>
> SY1 had A or B fail a few years back if my memory serves me 
> correctly.? But I was referring? London and Manchester which both had 
> total outages in recent years.
>
> Matt.
>
> On 14/3/2023 11:26 am, Nathan Brookfield wrote:
>
>     I?ve never had an A+B power rail failure at Equinix in 18 years
>     I?ve been in there facilities, TOUCH WOOD!!!
>
>     *Nathan Brookfield | VK2NAB
>     *General Manager
>
>     *p*: 1300 592 330 | *m*: 0412 266 008
>     *e*: Nathan.brookfield at iperium.com.au | *w*:?iperium.com.au
>
>     Suite 4.02, 189 Kent Street Sydney NSW 2000
>
>
>     *Your Telco Team*
>
>     DISCLAIMER: This document is intended solely for the named
>     addressee. This electronic communication, which includes any files
>     or attachments thereto, contains proprietary or confidential
>     information and may be privileged and otherwise protected under
>     copyright or other applicable intellectual property laws. The use,
>     disclosure, copying or distribution of any of the information
>     contained in this document, by any person other than the
>     addressee, is strictly prohibited. If you received this document
>     in error, please contact the sender immediately and delete all the
>     material from any computer. Confidentiality and legal privilege
>     are not waived or lost by reason of mistaken delivery to you. Any
>     views or opinions presented are solely those of the author and do
>     not necessarily represent those of Iperium.
>
>     WARNING: Computer viruses can be transmitted via email. The
>     recipient should check this email and any attachments for the
>     presence of viruses. Iperium accepts no liability for any damage
>     caused by any virus transmitted by this email.
>
>     *From:*AusNOG <ausnog-bounces at lists.ausnog.net>
>     <mailto:ausnog-bounces at lists.ausnog.net> *On Behalf Of *Matt Perkins
>     *Sent:* Tuesday, 14 March 2023 11:18 AM
>     *To:* ausnog at lists.ausnog.net
>     *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>
>     Yea because Eqinix has never had a power outage. :)
>
>     I think it's a little premature to judge at the moment. Doesn't
>     sound good based on the current info but we have all had bad days
>     im sure.
>
>     Matt
>
>     On 14/3/2023 11:06 am, Bradley Amm wrote:
>
>         Friends like let friends put gear in a Vocus dtacenter. Only
>         NextDC or Equinix :)
>
>         Sent from my iPhone
>
>
>
>
>             On 14 Mar 2023, at 7:08 am, DaZZa <dazzagibbs at gmail.com>
>             <mailto:dazzagibbs at gmail.com> wrote:
>
>             ?
>
>             Doesn't say a whole lot about Vocus' data center design
>             methodology, nor their periodic contingency testing.
>
>             They're apparently replacing the entire UPS on Friday -
>             guess we can all expect more outages then when they stuff
>             that up.
>
>             D
>
>             On Mon, 13 Mar 2023 at 18:30, Chad Kelly
>             <chad at cpkws.com.au> wrote:
>
>                 Thought I would create a new thread on this, as the
>                 issue wasn?t related to AussieBB.
>
>                 It is looking like this Melbourne outage is similar to
>                 the Sydney outage back in 2018, in that it appears to
>                 have been yet again caused by an issue with their UPS
>                 systems yet again.
>
>                 I?ll leave this hear
>                 https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/
>                 and this article on today?s outage as well.
>
>                 https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987
>
>                 Regards Chad.
>
>                 Chad Kelly
>
>                 Manager
>
>                 CPK Web Services
>
>                 Phone 03 52730246
>
>                 Web https://www.cpkws.com.au
>
>                 <image001.jpg>
>
>                 _______________________________________________
>                 AusNOG mailing list
>                 AusNOG at lists.ausnog.net
>                 https://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
>             -- 
>
>             veg?e?tar?i?an:
>             Ancient tribal slang for the village idiot who can't hunt,
>             fish or ride
>
>             _______________________________________________
>             AusNOG mailing list
>             AusNOG at lists.ausnog.net
>             https://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
>
>         _______________________________________________
>
>         AusNOG mailing list
>
>         AusNOG at lists.ausnog.net
>
>         https://lists.ausnog.net/mailman/listinfo/ausnog
>
>     -- 
>
>     ** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL
>
>       
>
>     /* Matt Perkins
>
>      ??????? Direct 02 8916 8101???? Spectrum Networks Ptd. Ltd.
>
>      ??????? Office 1300 133 299matt at spectrum.com.au
>
>      ??????? ABN 66 090 112 913????? Level 6, 350 George Street Sydney 2000
>
>       
>
>     */
>
> -- 
> ** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL
>   
> /* Matt Perkins
>  ??????? Direct 02 8916 8101???? Spectrum Networks Ptd. Ltd.
>  ??????? Office 1300 133 299matt at spectrum.com.au
>  ??????? ABN 66 090 112 913????? Level 6, 350 George Street Sydney 2000
>   
> */

-- 
** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL

/* Matt Perkins
         Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.
         Office 1300 133 299matt at spectrum.com.au
         ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000

*/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230314/688896d1/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 4595 bytes
Desc: not available
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230314/688896d1/attachment-0001.png>

From joel at zonenetworks.com.au  Tue Mar 14 12:26:11 2023
From: joel at zonenetworks.com.au (Joel Nath)
Date: Tue, 14 Mar 2023 01:26:11 +0000
Subject: [AusNOG] Vocus Datacentre outage.
In-Reply-To: <8D195CA1-4E67-46F4-BF39-C9AED0616977@iperium.com.au>
References: <CAMpV8Mpv98O-cqpGz+5SVRBsobsZzWAT0Rhy8UdiSQ6fUEiWjw@mail.gmail.com>
 <A7475E5D-495B-4B4B-91C1-169140FEACE1@bradleyamm.com>
 <18719e27-053c-2ec2-f48e-2b69d5114716@spectrum.com.au>
 <MEYP282MB2520B620B1438273E3A1009CDBBE9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <f35b78ea-6e27-1eb1-5314-20fde831056a@spectrum.com.au>
 <MEYP282MB2520A3C7BCA133967365ED77DBBE9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <SYBP282MB0538A5F3C118B6D6A4ED85F5AFBE9@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM>
 <8D195CA1-4E67-46F4-BF39-C9AED0616977@iperium.com.au>
Message-ID: <076eb1f9595849319ad53e27dab953ed@zonenetworks.com.au>

In EQX defence Sy1 didn?t go dark? it was multiple (3-5) rows of rack that lost power in 2009.

We were one of the customers heavily impacted in 2009 but understood the reason why it happened and EQX fixed it all up.

Vocus Doody st different story and now melb

We pulled out of Doody after the drama we faced there.

Joel.

From: AusNOG <ausnog-bounces at lists.ausnog.net> On Behalf Of Nathan Brookfield
Sent: Tuesday, 14 March 2023 11:54 AM
To: Christopher Hawker <chris at thesysadmin.dev>
Cc: ausnog at lists.ausnog.net
Subject: Re: [AusNOG] Vocus Datacentre outage.

I knew I didn?t remember it for a reason ?

[AusNOG] Equinix Outage<https://lists.ausnog.net/pipermail/ausnog/2009-July/003239.html>
lists.ausnog.net<https://lists.ausnog.net/pipermail/ausnog/2009-July/003239.html>

[X]<https://lists.ausnog.net/pipermail/ausnog/2009-July/003239.html>



On 14 Mar 2023, at 10:50, Christopher Hawker <chris at thesysadmin.dev<mailto:chris at thesysadmin.dev>> wrote:

?
SY1 apparently experienced a full power outage back in July 2009, according to AusNOG Archives. Some reported full outages, some partial. Was not a DC fault though, looks like a much larger grid issue where a fault at the Bayswater Power Station in the Hunter Valley (which affected generators in QLD and VIC) caused about 1000MW to be dropped from the grid, resulting in rolling blackouts across QLD, NSW, TAS, SA and VIC.

https://lists.ausnog.net/pipermail/ausnog/2009-July/003222.html

CH.
________________________________
From: AusNOG <ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>> on behalf of Nathan Brookfield <Nathan.Brookfield at iperium.com.au<mailto:Nathan.Brookfield at iperium.com.au>>
Sent: Tuesday, March 14, 2023 11:40 AM
To: Matt Perkins <matt at spectrum.com.au<mailto:matt at spectrum.com.au>>; ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: Re: [AusNOG] Vocus Datacentre outage.


Have 100% had A and B fail separately at SY1 but never both, thankfully!!!



Could have been part of the facility though so don?t quote me.



UK on the other hand, fail!



From: Matt Perkins <matt at spectrum.com.au<mailto:matt at spectrum.com.au>>
Sent: Tuesday, 14 March 2023 11:30 AM
To: Nathan Brookfield <Nathan.Brookfield at iperium.com.au<mailto:Nathan.Brookfield at iperium.com.au>>; ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] Vocus Datacentre outage.



SY1 had A or B fail a few years back if my memory serves me correctly.  But I was referring  London and Manchester which both had total outages in recent years.

Matt.



On 14/3/2023 11:26 am, Nathan Brookfield wrote:

I?ve never had an A+B power rail failure at Equinix in 18 years I?ve been in there facilities, TOUCH WOOD!!!



Nathan Brookfield | VK2NAB
General Manager

p: 1300 592 330 | m: 0412 266 008
e: Nathan.brookfield at iperium.com.au<mailto:Nathan.brookfield at iperium.com.au> | w: iperium.com.au

Suite 4.02, 189 Kent Street Sydney NSW 2000
<image001.png>

Your Telco Team

DISCLAIMER: This document is intended solely for the named addressee. This electronic communication, which includes any files or attachments thereto, contains proprietary or confidential information and may be privileged and otherwise protected under copyright or other applicable intellectual property laws. The use, disclosure, copying or distribution of any of the information contained in this document, by any person other than the addressee, is strictly prohibited. If you received this document in error, please contact the sender immediately and delete all the material from any computer. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you. Any views or opinions presented are solely those of the author and do not necessarily represent those of Iperium.

WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. Iperium accepts no liability for any damage caused by any virus transmitted by this email.

From: AusNOG <ausnog-bounces at lists.ausnog.net><mailto:ausnog-bounces at lists.ausnog.net> On Behalf Of Matt Perkins
Sent: Tuesday, 14 March 2023 11:18 AM
To: ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] Vocus Datacentre outage.



Yea because Eqinix has never had a power outage. :)

I think it's a little premature to judge at the moment. Doesn't sound good based on the current info but we have all had bad days im sure.

Matt



On 14/3/2023 11:06 am, Bradley Amm wrote:

Friends like let friends put gear in a Vocus dtacenter. Only NextDC or Equinix :)

Sent from my iPhone



On 14 Mar 2023, at 7:08 am, DaZZa <dazzagibbs at gmail.com><mailto:dazzagibbs at gmail.com> wrote:

?

Doesn't say a whole lot about Vocus' data center design methodology, nor their periodic contingency testing.



They're apparently replacing the entire UPS on Friday - guess we can all expect more outages then when they stuff that up.



D



On Mon, 13 Mar 2023 at 18:30, Chad Kelly <chad at cpkws.com.au<mailto:chad at cpkws.com.au>> wrote:

Thought I would create a new thread on this, as the issue wasn?t related to AussieBB.

It is looking like this Melbourne outage is similar to the Sydney outage back in 2018, in that it appears to have been yet again caused by an issue with their UPS systems yet again.

I?ll leave this hear https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/ and this article on today?s outage as well.

https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987



Regards Chad.







Chad Kelly

Manager

CPK Web Services

Phone 03 52730246

Web https://www.cpkws.com.au

<image001.jpg>



_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
https://lists.ausnog.net/mailman/listinfo/ausnog


--

veg?e?tar?i?an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
https://lists.ausnog.net/mailman/listinfo/ausnog



_______________________________________________

AusNOG mailing list

AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>

https://lists.ausnog.net/mailman/listinfo/ausnog

--

** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL



/* Matt Perkins

        Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.

        Office 1300 133 299     matt at spectrum.com.au<mailto:matt at spectrum.com.au>

        ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000



*/

--

** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL



/* Matt Perkins

        Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.

        Office 1300 133 299     matt at spectrum.com.au<mailto:matt at spectrum.com.au>

        ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000



*/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230314/3c017c01/attachment.htm>

From jaedwards at gmail.com  Tue Mar 14 12:49:33 2023
From: jaedwards at gmail.com (John Edwards)
Date: Tue, 14 Mar 2023 12:19:33 +1030
Subject: [AusNOG] Vocus Datacentre outage.
In-Reply-To: <f59b62f8-4ab8-19cc-9890-0f1e45a13eb7@spectrum.com.au>
References: <CAMpV8Mpv98O-cqpGz+5SVRBsobsZzWAT0Rhy8UdiSQ6fUEiWjw@mail.gmail.com>
 <A7475E5D-495B-4B4B-91C1-169140FEACE1@bradleyamm.com>
 <18719e27-053c-2ec2-f48e-2b69d5114716@spectrum.com.au>
 <MEYP282MB2520B620B1438273E3A1009CDBBE9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <f35b78ea-6e27-1eb1-5314-20fde831056a@spectrum.com.au>
 <MEYP282MB2520A3C7BCA133967365ED77DBBE9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <SYBP282MB0538A5F3C118B6D6A4ED85F5AFBE9@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM>
 <f59b62f8-4ab8-19cc-9890-0f1e45a13eb7@spectrum.com.au>
Message-ID: <CAOSsYkrKcwcfZZ5tDUKjcmr1guHSwQgvgSwNEio8HFD+PzdARg@mail.gmail.com>

Don't rule out a firmware setting that lay dormant or was introduced
through overzealous or simultaneous patching in a maintenance window.

This was how SA ended up with a total blackout a few years back - a default
setting in a particular model of wind turbine shut it down in response to a
previously unseen power anomaly - caused by a power outage elsewhere on the
grid.

John


On Tue, 14 Mar 2023 at 11:48, Matt Perkins <matt at spectrum.com.au> wrote:

> The questions left to be answered for Vocus is why did it take 12 hours to
> restore one rail. Where was the static bypass and failing that where was
> the maintenance bypass.  Im guessing that's what it's on maintenance bypass
> now and the slow action may have something to do with the public holiday in
> Vic..
>
>
> I shouldn't speculate. But it is fun.
>
>
>  Something seriously let go in that UPS. If i had to put my electrical
> engineers hat on I would say their load capacitor maintenance was not done
> and let go. Or someone was working upstream and broke the natural/earth
> bond. But there's a lot of parts in a system like that , most modern DC UPS
> setups are fairly robust.  I once had 3 strings of batteries catch fire and
> melt down and the UPS didn't miss a beat other then spewing a lot of
> alarms. We moved strings around on the fly and nobody even noticed.
>
>
> The other things that seems odd is why did an A failure trip B.  Sounds
> like a single supply is at surge/break capacity. Also a worry.
>
>
> But in the end we all have redundancy around 530 Collins right ? Right ?
>
>
> Matt
>
>
>
> On 14/3/2023 11:50 am, Christopher Hawker wrote:
>
> SY1 apparently experienced a full power outage back in July 2009,
> according to AusNOG Archives. Some reported full outages, some partial. Was
> not a DC fault though, looks like a much larger grid issue where a fault at
> the Bayswater Power Station in the Hunter Valley (which affected generators
> in QLD and VIC) caused about 1000MW to be dropped from the grid, resulting
> in rolling blackouts across QLD, NSW, TAS, SA and VIC.
>
> https://lists.ausnog.net/pipermail/ausnog/2009-July/003222.html
>
> CH.
> ------------------------------
> *From:* AusNOG <ausnog-bounces at lists.ausnog.net>
> <ausnog-bounces at lists.ausnog.net> on behalf of Nathan Brookfield
> <Nathan.Brookfield at iperium.com.au> <Nathan.Brookfield at iperium.com.au>
> *Sent:* Tuesday, March 14, 2023 11:40 AM
> *To:* Matt Perkins <matt at spectrum.com.au> <matt at spectrum.com.au>;
> ausnog at lists.ausnog.net <ausnog at lists.ausnog.net>
> <ausnog at lists.ausnog.net>
> *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>
>
> Have 100% had A and B fail separately at SY1 but never both, thankfully!!!
>
>
>
> Could have been part of the facility though so don?t quote me.
>
>
>
> UK on the other hand, fail!
>
>
>
> *From:* Matt Perkins <matt at spectrum.com.au> <matt at spectrum.com.au>
> *Sent:* Tuesday, 14 March 2023 11:30 AM
> *To:* Nathan Brookfield <Nathan.Brookfield at iperium.com.au>
> <Nathan.Brookfield at iperium.com.au>; ausnog at lists.ausnog.net
> *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>
>
>
> SY1 had A or B fail a few years back if my memory serves me correctly.
> But I was referring  London and Manchester which both had total outages in
> recent years.
>
> Matt.
>
>
>
> On 14/3/2023 11:26 am, Nathan Brookfield wrote:
>
> I?ve never had an A+B power rail failure at Equinix in 18 years I?ve been
> in there facilities, TOUCH WOOD!!!
>
>
>
>
> *Nathan Brookfield | VK2NAB *General Manager
>
> *p*: 1300 592 330 | *m*: 0412 266 008
> *e*: Nathan.brookfield at iperium.com.au | *w*: iperium.com.au
>
> Suite 4.02, 189 Kent Street Sydney NSW 2000
>
>
> *Your Telco Team*
>
> DISCLAIMER: This document is intended solely for the named addressee. This
> electronic communication, which includes any files or attachments thereto,
> contains proprietary or confidential information and may be privileged and
> otherwise protected under copyright or other applicable intellectual
> property laws. The use, disclosure, copying or distribution of any of the
> information contained in this document, by any person other than the
> addressee, is strictly prohibited. If you received this document in error,
> please contact the sender immediately and delete all the material from any
> computer. Confidentiality and legal privilege are not waived or lost by
> reason of mistaken delivery to you. Any views or opinions presented are
> solely those of the author and do not necessarily represent those of
> Iperium.
>
> WARNING: Computer viruses can be transmitted via email. The recipient
> should check this email and any attachments for the presence of viruses.
> Iperium accepts no liability for any damage caused by any virus transmitted
> by this email.
>
> *From:* AusNOG <ausnog-bounces at lists.ausnog.net>
> <ausnog-bounces at lists.ausnog.net> *On Behalf Of *Matt Perkins
> *Sent:* Tuesday, 14 March 2023 11:18 AM
> *To:* ausnog at lists.ausnog.net
> *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>
>
>
> Yea because Eqinix has never had a power outage. :)
>
> I think it's a little premature to judge at the moment. Doesn't sound good
> based on the current info but we have all had bad days im sure.
>
> Matt
>
>
>
> On 14/3/2023 11:06 am, Bradley Amm wrote:
>
> Friends like let friends put gear in a Vocus dtacenter. Only NextDC or
> Equinix :)
>
> Sent from my iPhone
>
>
>
>
> On 14 Mar 2023, at 7:08 am, DaZZa <dazzagibbs at gmail.com>
> <dazzagibbs at gmail.com> wrote:
>
> ?
>
> Doesn't say a whole lot about Vocus' data center design methodology, nor
> their periodic contingency testing.
>
>
>
> They're apparently replacing the entire UPS on Friday - guess we can all
> expect more outages then when they stuff that up.
>
>
>
> D
>
>
>
> On Mon, 13 Mar 2023 at 18:30, Chad Kelly <chad at cpkws.com.au> wrote:
>
> Thought I would create a new thread on this, as the issue wasn?t related
> to AussieBB.
>
> It is looking like this Melbourne outage is similar to the Sydney outage
> back in 2018, in that it appears to have been yet again caused by an issue
> with their UPS systems yet again.
>
> I?ll leave this hear
> https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/
> and this article on today?s outage as well.
>
>
> https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987
>
>
>
> Regards Chad.
>
>
>
>
>
>
>
> Chad Kelly
>
> Manager
>
> CPK Web Services
>
> Phone 03 52730246
>
> Web https://www.cpkws.com.au
>
> <image001.jpg>
>
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
> --
>
> veg?e?tar?i?an:
> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
>
> _______________________________________________
>
> AusNOG mailing list
>
> AusNOG at lists.ausnog.net
>
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
> --
>
> ** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL
>
>
>
> /* Matt Perkins
>
>         Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.
>
>         Office 1300 133 299     matt at spectrum.com.au
>
>         ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000
>
>
>
> */
>
> --
>
> ** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL
>
>
>
> /* Matt Perkins
>
>         Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.
>
>         Office 1300 133 299     matt at spectrum.com.au
>
>         ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000
>
>
>
> */
>
> --
> ** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL
>
> /* Matt Perkins
>         Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.
>         Office 1300 133 299     matt at spectrum.com.au
>         ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000
>
> */
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230314/ed80a3f2/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 4595 bytes
Desc: not available
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230314/ed80a3f2/attachment.png>

From mark at innaloo.net  Tue Mar 14 13:23:07 2023
From: mark at innaloo.net (Mark Dignam)
Date: Tue, 14 Mar 2023 10:23:07 +0800
Subject: [AusNOG] Vocus Datacentre outage.
In-Reply-To: <f59b62f8-4ab8-19cc-9890-0f1e45a13eb7@spectrum.com.au>
References: <CAMpV8Mpv98O-cqpGz+5SVRBsobsZzWAT0Rhy8UdiSQ6fUEiWjw@mail.gmail.com>
 <A7475E5D-495B-4B4B-91C1-169140FEACE1@bradleyamm.com>
 <18719e27-053c-2ec2-f48e-2b69d5114716@spectrum.com.au>
 <MEYP282MB2520B620B1438273E3A1009CDBBE9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <f35b78ea-6e27-1eb1-5314-20fde831056a@spectrum.com.au>
 <MEYP282MB2520A3C7BCA133967365ED77DBBE9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <SYBP282MB0538A5F3C118B6D6A4ED85F5AFBE9@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM>
 <f59b62f8-4ab8-19cc-9890-0f1e45a13eb7@spectrum.com.au>
Message-ID: <000901d9561b$ea361960$bea24c20$@innaloo.net>

Something that I have to keep reminding beancounters with A/B planning is don?t go above 40% load on either side.

 

Its very difficult to convince a CFO who?s entire life is spent with a spreadsheet open in front of him is that N+1 means typically Double, and that Kw != Kva.

 

 

From: AusNOG <ausnog-bounces at lists.ausnog.net> On Behalf Of Matt Perkins
Sent: Tuesday, March 14, 2023 9:18 AM
To: Christopher Hawker <chris at thesysadmin.dev>; Nathan Brookfield <Nathan.Brookfield at iperium.com.au>; ausnog at lists.ausnog.net
Subject: Re: [AusNOG] Vocus Datacentre outage.

 

The questions left to be answered for Vocus is why did it take 12 hours to restore one rail. Where was the static bypass and failing that where was the maintenance bypass.  Im guessing that's what it's on maintenance bypass now and the slow action may have something to do with the public holiday in Vic..  

 

I shouldn't speculate. But it is fun. 

 

 Something seriously let go in that UPS. If i had to put my electrical engineers hat on I would say their load capacitor maintenance was not done and let go. Or someone was working upstream and broke the natural/earth bond. But there's a lot of parts in a system like that , most modern DC UPS setups are fairly robust.  I once had 3 strings of batteries catch fire and melt down and the UPS didn't miss a beat other then spewing a lot of alarms. We moved strings around on the fly and nobody even noticed.   

 

The other things that seems odd is why did an A failure trip B.  Sounds like a single supply is at surge/break capacity. Also a worry.  

 

But in the end we all have redundancy around 530 Collins right ? Right ? 

 

Matt

 

 

On 14/3/2023 11:50 am, Christopher Hawker wrote:

SY1 apparently experienced a full power outage back in July 2009, according to AusNOG Archives. Some reported full outages, some partial. Was not a DC fault though, looks like a much larger grid issue where a fault at the Bayswater Power Station in the Hunter Valley (which affected generators in QLD and VIC) caused about 1000MW to be dropped from the grid, resulting in rolling blackouts across QLD, NSW, TAS, SA and VIC.

 

https://lists.ausnog.net/pipermail/ausnog/2009-July/003222.html

 

CH.

  _____  

From: AusNOG  <mailto:ausnog-bounces at lists.ausnog.net> <ausnog-bounces at lists.ausnog.net> on behalf of Nathan Brookfield  <mailto:Nathan.Brookfield at iperium.com.au> <Nathan.Brookfield at iperium.com.au>
Sent: Tuesday, March 14, 2023 11:40 AM
To: Matt Perkins  <mailto:matt at spectrum.com.au> <matt at spectrum.com.au>; ausnog at lists.ausnog.net <mailto:ausnog at lists.ausnog.net>   <mailto:ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] Vocus Datacentre outage. 

 

Have 100% had A and B fail separately at SY1 but never both, thankfully!!!

 

Could have been part of the facility though so don?t quote me.

 

UK on the other hand, fail!

 

From: Matt Perkins  <mailto:matt at spectrum.com.au> <matt at spectrum.com.au> 
Sent: Tuesday, 14 March 2023 11:30 AM
To: Nathan Brookfield  <mailto:Nathan.Brookfield at iperium.com.au> <Nathan.Brookfield at iperium.com.au>; ausnog at lists.ausnog.net <mailto:ausnog at lists.ausnog.net> 
Subject: Re: [AusNOG] Vocus Datacentre outage.

 

SY1 had A or B fail a few years back if my memory serves me correctly.  But I was referring  London and Manchester which both had total outages in recent years. 

Matt.

 

On 14/3/2023 11:26 am, Nathan Brookfield wrote:

I?ve never had an A+B power rail failure at Equinix in 18 years I?ve been in there facilities, TOUCH WOOD!!!

 

Nathan Brookfield | VK2NAB
General Manager

p: 1300 592 330 | m: 0412 266 008
e: Nathan.brookfield at iperium.com.au <mailto:Nathan.brookfield at iperium.com.au>  | w: iperium.com.au

Suite 4.02, 189 Kent Street Sydney NSW 2000


Your Telco Team

DISCLAIMER: This document is intended solely for the named addressee. This electronic communication, which includes any files or attachments thereto, contains proprietary or confidential information and may be privileged and otherwise protected under copyright or other applicable intellectual property laws. The use, disclosure, copying or distribution of any of the information contained in this document, by any person other than the addressee, is strictly prohibited. If you received this document in error, please contact the sender immediately and delete all the material from any computer. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you. Any views or opinions presented are solely those of the author and do not necessarily represent those of Iperium. 

WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. Iperium accepts no liability for any damage caused by any virus transmitted by this email.

From: AusNOG  <mailto:ausnog-bounces at lists.ausnog.net> <ausnog-bounces at lists.ausnog.net> On Behalf Of Matt Perkins
Sent: Tuesday, 14 March 2023 11:18 AM
To: ausnog at lists.ausnog.net <mailto:ausnog at lists.ausnog.net> 
Subject: Re: [AusNOG] Vocus Datacentre outage.

 

Yea because Eqinix has never had a power outage. :) 

I think it's a little premature to judge at the moment. Doesn't sound good based on the current info but we have all had bad days im sure.  

Matt

 

On 14/3/2023 11:06 am, Bradley Amm wrote:

Friends like let friends put gear in a Vocus dtacenter. Only NextDC or Equinix :) 

Sent from my iPhone





On 14 Mar 2023, at 7:08 am, DaZZa  <mailto:dazzagibbs at gmail.com> <dazzagibbs at gmail.com> wrote:

? 

Doesn't say a whole lot about Vocus' data center design methodology, nor their periodic contingency testing.

 

They're apparently replacing the entire UPS on Friday - guess we can all expect more outages then when they stuff that up.

 

D

 

On Mon, 13 Mar 2023 at 18:30, Chad Kelly <chad at cpkws.com.au <mailto:chad at cpkws.com.au> > wrote:

Thought I would create a new thread on this, as the issue wasn?t related to AussieBB. 

It is looking like this Melbourne outage is similar to the Sydney outage back in 2018, in that it appears to have been yet again caused by an issue with their UPS systems yet again.

I?ll leave this hear https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/ and this article on today?s outage as well. 

https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987

 

Regards Chad. 

 

 

 

Chad Kelly 

Manager 

CPK Web Services 

Phone 03 52730246

Web https://www.cpkws.com.au

<image001.jpg>

 

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net <mailto:AusNOG at lists.ausnog.net> 
https://lists.ausnog.net/mailman/listinfo/ausnog



-- 

veg?e?tar?i?an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net <mailto:AusNOG at lists.ausnog.net> 
https://lists.ausnog.net/mailman/listinfo/ausnog





_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net <mailto:AusNOG at lists.ausnog.net> 
https://lists.ausnog.net/mailman/listinfo/ausnog

-- 
** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL 
 
/* Matt Perkins
        Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.
        Office 1300 133 299     matt at spectrum.com.au <mailto:matt at spectrum.com.au> 
        ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000
 
*/

-- 
** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL 
 
/* Matt Perkins
        Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.
        Office 1300 133 299     matt at spectrum.com.au <mailto:matt at spectrum.com.au> 
        ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000
 
*/

-- 
** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL 
 
/* Matt Perkins
        Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.
        Office 1300 133 299     matt at spectrum.com.au <mailto:matt at spectrum.com.au> 
        ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000
 
*/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230314/a8cb5a33/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 4595 bytes
Desc: not available
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230314/a8cb5a33/attachment.png>

From jared.hirst at serversaustralia.com.au  Tue Mar 14 13:44:58 2023
From: jared.hirst at serversaustralia.com.au (Jared Hirst | Servers Australia)
Date: Tue, 14 Mar 2023 02:44:58 +0000
Subject: [AusNOG] Vocus Datacentre outage.
In-Reply-To: <076eb1f9595849319ad53e27dab953ed@zonenetworks.com.au>
References: <ME3PR01MB6878081AEFEDA25E520E1A3BFDB99@ME3PR01MB6878.ausprd01.prod.outlook.com>
 <CAMpV8Mpv98O-cqpGz+5SVRBsobsZzWAT0Rhy8UdiSQ6fUEiWjw@mail.gmail.com>
 <A7475E5D-495B-4B4B-91C1-169140FEACE1@bradleyamm.com>
 <18719e27-053c-2ec2-f48e-2b69d5114716@spectrum.com.au>
 <MEYP282MB2520B620B1438273E3A1009CDBBE9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <f35b78ea-6e27-1eb1-5314-20fde831056a@spectrum.com.au>
 <MEYP282MB2520A3C7BCA133967365ED77DBBE9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <SYBP282MB0538A5F3C118B6D6A4ED85F5AFBE9@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM>
 <8D195CA1-4E67-46F4-BF39-C9AED0616977@iperium.com.au>
 <f59b62f8-4ab8-19cc-9890-0f1e45a13eb7@spectrum.com.au>
 <076eb1f9595849319ad53e27dab953ed@zonenetworks.com.au>
 <CAOSsYkrKcwcfZZ5tDUKjcmr1guHSwQgvgSwNEio8HFD+PzdARg@mail.gmail.com>
 <000901d9561b$ea361960$bea24c20$@innaloo.net>
 <a412a0188100fc5979e7daeb81628876@frontapp.com>
Message-ID: <a412a0188100fc5979e7daeb81628876@frontapp.com>

Good times back then Joel! It was a UPS issue back there too if I remember, and the A and B feeds were all fed into the one UPS?.


A tonne of people shifted within that week, for me it was the way that it was palmed off, as you say, others might have issues but they own those issues and rectify them?



Jared Hirst
Chief Executive Officer
P: (02) 8115 8801 
4 Amy Close, Wyong, NSW 2259
Need assistance? We are here 24/7 +61 2 8115 8888 

--- original message ---
On March 14, 2023 at 12:26 PM GMT+11 joel at zonenetworks.com.au wrote:



In EQX defence Sy1 didn?t go dark? it was multiple (3-5) rows of rack that lost power in 2009.



We were one of the customers heavily impacted in 2009 but understood the reason why it happened and EQX fixed it all up.



Vocus Doody st different story and now melb



We pulled out of Doody after the drama we faced there.



Joel.



From: AusNOG <ausnog-bounces at lists.ausnog.net>
On Behalf Of Nathan Brookfield

Sent: Tuesday, 14 March 2023 11:54 AM

To: Christopher Hawker <chris at thesysadmin.dev>

Cc: ausnog at lists.ausnog.net

Subject: Re: [AusNOG] Vocus Datacentre outage.





I knew I didn?t remember it for a reason 
? 




[AusNOG] Equinix Outage

lists.ausnog.net











On 14 Mar 2023, at 10:50, Christopher Hawker <chris at thesysadmin.dev> wrote:






SY1 apparently experienced a full power outage back in July 2009, according to AusNOG Archives. Some reported full outages, some partial. Was not a DC fault
though, looks like a much larger grid issue where a fault at the Bayswater Power Station in the Hunter Valley (which affected generators in QLD and VIC) caused about 1000MW to be dropped from the grid, resulting in rolling blackouts across QLD, NSW, TAS, SA
and VIC.





https://lists.ausnog.net/pipermail/ausnog/2009-July/003222.html





CH.




From: AusNOG <ausnog-bounces at lists.ausnog.net>
on behalf of Nathan Brookfield <Nathan.Brookfield at iperium.com.au>

Sent: Tuesday, March 14, 2023 11:40 AM

To: Matt Perkins <matt at spectrum.com.au>;
ausnog at lists.ausnog.net <ausnog at lists.ausnog.net>

Subject: Re: [AusNOG] Vocus Datacentre outage. 





Have 100% had A and B fail separately at SY1 but never both, thankfully!!!



Could have been part of the facility though so don?t quote me.



UK on the other hand, fail!



From: Matt Perkins <matt at spectrum.com.au>

Sent: Tuesday, 14 March 2023 11:30 AM

To: Nathan Brookfield <Nathan.Brookfield at iperium.com.au>;
ausnog at lists.ausnog.net

Subject: Re: [AusNOG] Vocus Datacentre outage.





SY1 had A or B fail a few years back if my memory serves me correctly. But I was referring London and Manchester which both had total outages in recent years.

Matt.



On 14/3/2023 11:26 am, Nathan Brookfield wrote:


> I?ve never had an A+B power rail failure at Equinix in 18 years I?ve been in there facilities, TOUCH WOOD!!!

> 

> Nathan Brookfield | VK2NAB

> General Manager


> p: 1300 592 330 |>m: 0412 266 008

> e:>Nathan.brookfield at iperium.com.au | w: iperium.com.au


> Suite 4.02, 189 Kent Street Sydney NSW 2000

> <image001.png>



> Your Telco Team


> DISCLAIMER: This document is intended solely for the named addressee. This electronic communication, which includes any files or attachments thereto, contains proprietary> or confidential information and may be privileged and otherwise protected under copyright or other applicable intellectual property laws. The use, disclosure, copying or distribution of any of the information contained in this document, by any person other> than the addressee, is strictly prohibited. If you received this document in error, please contact the sender immediately and delete all the material from any computer. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery> to you. Any views or opinions presented are solely those of the author and do not necessarily represent those of Iperium. 


> WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. Iperium accepts no liability for any damage caused by any virus transmitted by this email.

> From: AusNOG><ausnog-bounces at lists.ausnog.net>>On Behalf Of Matt Perkins

> Sent: Tuesday, 14 March 2023 11:18 AM

> To: ausnog at lists.ausnog.net

> Subject: Re: [AusNOG] Vocus Datacentre outage.



> 

> Yea because Eqinix has never had a power outage. :) 

> I think it's a little premature to judge at the moment. Doesn't sound good based on the current info but we have all had bad days im sure. 

> Matt

> 

> On 14/3/2023 11:06 am, Bradley Amm wrote:


>> Friends like let friends put gear in a Vocus dtacenter. Only NextDC or Equinix :) 

>> Sent from my iPhone





>>> On 14 Mar 2023, at 7:08 am, DaZZa>>><dazzagibbs at gmail.com> wrote:




>>> Doesn't say a whole lot about Vocus' data center design methodology, nor their periodic contingency testing.


>>> 


>>> They're apparently replacing the entire UPS on Friday - guess we can all expect more outages then when they stuff that up.


>>> 


>>> D



>>> 

>>> On Mon, 13 Mar 2023 at 18:30, Chad Kelly <chad at cpkws.com.au> wrote:


>>>> Thought I would create a new thread on this, as the issue wasn?t related to AussieBB.

>>>> It is looking like this Melbourne outage is similar to the Sydney outage back in 2018, in that it appears to have been yet again caused by an issue with their UPS systems yet again.

>>>> I?ll leave this hear >>>>https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/ and this article on today?s outage as well.

>>>> https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987

>>>> 

>>>> Regards Chad. 

>>>> 

>>>> 

>>>> 

>>>> Chad Kelly 

>>>> Manager 

>>>> CPK Web Services 

>>>> Phone 03 52730246

>>>> Web https://www.cpkws.com.au

>>>> <image001.jpg>


>>>> 



>>>> _______________________________________________

>>>> AusNOG mailing list

>>>> AusNOG at lists.ausnog.net

>>>> https://lists.ausnog.net/mailman/listinfo/ausnog






>>> -- 

>>> veg?e?tar?i?an:

>>> Ancient tribal slang for the village idiot who can't hunt, fish or ride


>>> _______________________________________________

>>> AusNOG mailing list

>>> AusNOG at lists.ausnog.net

>>> https://lists.ausnog.net/mailman/listinfo/ausnog






>> _______________________________________________

>> AusNOG mailing list

>> AusNOG at lists.ausnog.net

>> https://lists.ausnog.net/mailman/listinfo/ausnog


> -- 

> ** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL 

> 

> /* Matt Perkins

> Direct 02 8916 8101 Spectrum Networks Ptd. Ltd.

> Office 1300 133 299 matt at spectrum.com.au

> ABN 66 090 112 913 Level 6, 350 George Street Sydney 2000

> 

> */


-- 

** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL 



/* Matt Perkins

Direct 02 8916 8101 Spectrum Networks Ptd. Ltd.

Office 1300 133 299 matt at spectrum.com.au

ABN 66 090 112 913 Level 6, 350 George Street Sydney 2000



*/
--- end of original message ---
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230314/8d2515dd/attachment-0001.htm>

From dave.taht at gmail.com  Tue Mar 14 21:42:14 2023
From: dave.taht at gmail.com (Dave Taht)
Date: Tue, 14 Mar 2023 03:42:14 -0700
Subject: [AusNOG] fq_codel/cake/libreqos beta testers wanted
Message-ID: <CAA93jw7Kd=7BCfCdk3R3uXmtRJxFfLSjBAZW3T-mGF=gEtu+_A@mail.gmail.com>

A) I have always been curious to what extent, if any, fq_codel, or
cake made it into any CPE and home routers or deployment, down under?
Or is it all policers?

I remember Mark Nottingham ran cake for a while, then ran a heat gun
on edgerouter X he was using... (it was a hilarious picture, I don?t
remember what forum it was on?)

B) For the last 9 months I have been helping out the libreqos.io
project leverage xdp, ebpf, rust, and cake (in diffserv4 mode) to
build a high performance transparent bridge to better shape traffic.

We have got it to where it can push 25Gbits at about 40% of cpu on 16
cores on a xeon gold for about 10k ISP subscribrs on various plans. We
are now in rc2 on the v1.4 release, which has a bunch more speedups
and analytical tools. Another cool feature in it is "on a stick" mode
where it can go inband on a single port and do the magic across vlans.

I (wearing my scientist hat) am very interested in evaluating network
behaviors and performance at a mixture of shorter and longer RTTs and
fiber vs wireless transports. Is there anyone down there willing to
put this inline on their network? It?s pretty stable code, now, I
think... I'm pretty sure... and takes about 20 minutes to get setup
(most of which is compiling the rust)

https://github.com/LibreQoE/LibreQoS/wiki/v1.4 has the install
instructions (for the truly daring there is the "heimdall" branch where
the post 1.4 work is going) and we did a podcast about it here:
https://packetpushers.net/podcast/heavy-networking-666-improving-quality-of-experience-with-libreqos/

thx for any help you can offer!

--
I keep wondering if sqrt(flows) bdp is correct:
https://blog.cerowrt.org/post/juniper/
Dave T?ht CEO, TekLibre, LLC

From davidtradconsulting at gmail.com  Tue Mar 14 21:59:55 2023
From: davidtradconsulting at gmail.com (David Trad)
Date: Tue, 14 Mar 2023 20:59:55 +1000
Subject: [AusNOG] fq_codel/cake/libreqos beta testers wanted
In-Reply-To: <CAA93jw7Kd=7BCfCdk3R3uXmtRJxFfLSjBAZW3T-mGF=gEtu+_A@mail.gmail.com>
References: <CAA93jw7Kd=7BCfCdk3R3uXmtRJxFfLSjBAZW3T-mGF=gEtu+_A@mail.gmail.com>
Message-ID: <CAHyDPG66WnYD41ULqjFuKBNB3sod618X7S9M1A-+gTMQ_L8Zqw@mail.gmail.com>

I'm curious and looking into this.

On Tue, 14 Mar 2023, 8:42 pm Dave Taht, <dave.taht at gmail.com> wrote:

> A) I have always been curious to what extent, if any, fq_codel, or
> cake made it into any CPE and home routers or deployment, down under?
> Or is it all policers?
>
> I remember Mark Nottingham ran cake for a while, then ran a heat gun
> on edgerouter X he was using... (it was a hilarious picture, I don?t
> remember what forum it was on?)
>
> B) For the last 9 months I have been helping out the libreqos.io
> project leverage xdp, ebpf, rust, and cake (in diffserv4 mode) to
> build a high performance transparent bridge to better shape traffic.
>
> We have got it to where it can push 25Gbits at about 40% of cpu on 16
> cores on a xeon gold for about 10k ISP subscribrs on various plans. We
> are now in rc2 on the v1.4 release, which has a bunch more speedups
> and analytical tools. Another cool feature in it is "on a stick" mode
> where it can go inband on a single port and do the magic across vlans.
>
> I (wearing my scientist hat) am very interested in evaluating network
> behaviors and performance at a mixture of shorter and longer RTTs and
> fiber vs wireless transports. Is there anyone down there willing to
> put this inline on their network? It?s pretty stable code, now, I
> think... I'm pretty sure... and takes about 20 minutes to get setup
> (most of which is compiling the rust)
>
> https://github.com/LibreQoE/LibreQoS/wiki/v1.4 has the install
> instructions (for the truly daring there is the "heimdall" branch where
> the post 1.4 work is going) and we did a podcast about it here:
>
> https://packetpushers.net/podcast/heavy-networking-666-improving-quality-of-experience-with-libreqos/
>
> thx for any help you can offer!
>
> --
> I keep wondering if sqrt(flows) bdp is correct:
> https://blog.cerowrt.org/post/juniper/
> Dave T?ht CEO, TekLibre, LLC
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230314/09760e69/attachment.htm>

From ausnog at jdmnet.com.au  Wed Mar 15 00:05:53 2023
From: ausnog at jdmnet.com.au (Jacob Bisby)
Date: Tue, 14 Mar 2023 21:05:53 +0800
Subject: [AusNOG] fq_codel/cake/libreqos beta testers wanted
In-Reply-To: <CAA93jw7Kd=7BCfCdk3R3uXmtRJxFfLSjBAZW3T-mGF=gEtu+_A@mail.gmail.com>
References: <CAA93jw7Kd=7BCfCdk3R3uXmtRJxFfLSjBAZW3T-mGF=gEtu+_A@mail.gmail.com>
Message-ID: <035f01d95675$b6060330$22120990$@jdmnet.com.au>

We've had a good chat or two on this subject already Dave :), chipping in my 2c / support here:

LibreQOS is already a part of a project that I am working on at the moment (pending some more deliveries of parts), so I'm hoping to get some testing done myself soon(tm) and be able to share some results with Dave after pushing it with the right hardware. It'd be great if others could also pickup LibreQoS and test it for their own requirements too if you can find the time - maybe we'll even end up with another interesting AusNOG talk about open-source network technology this year, who knows :)

For the unaware - LibreQOS is tied to bufferbloat.net, cerowrt, etc, all of which Dave is heavily involved with. So definitely someone to talk to and a project to checkout if you've been following / are interested in AQM QOS mechanisms like CoDeL, CAKE, etc and the general fun around managing buffer bloat.

- Jacob Bisby

-----Original Message-----
From: AusNOG <ausnog-bounces at lists.ausnog.net> On Behalf Of Dave Taht
Sent: Tuesday, March 14, 2023 6:42 PM
To: <ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>; Mark Nottingham <mnot at mnot.net>
Subject: [AusNOG] fq_codel/cake/libreqos beta testers wanted

A) I have always been curious to what extent, if any, fq_codel, or cake made it into any CPE and home routers or deployment, down under?
Or is it all policers?

I remember Mark Nottingham ran cake for a while, then ran a heat gun on edgerouter X he was using... (it was a hilarious picture, I don?t remember what forum it was on?)

B) For the last 9 months I have been helping out the libreqos.io project leverage xdp, ebpf, rust, and cake (in diffserv4 mode) to build a high performance transparent bridge to better shape traffic.

We have got it to where it can push 25Gbits at about 40% of cpu on 16 cores on a xeon gold for about 10k ISP subscribrs on various plans. We are now in rc2 on the v1.4 release, which has a bunch more speedups and analytical tools. Another cool feature in it is "on a stick" mode where it can go inband on a single port and do the magic across vlans.

I (wearing my scientist hat) am very interested in evaluating network behaviors and performance at a mixture of shorter and longer RTTs and fiber vs wireless transports. Is there anyone down there willing to put this inline on their network? It?s pretty stable code, now, I think... I'm pretty sure... and takes about 20 minutes to get setup (most of which is compiling the rust)

https://github.com/LibreQoE/LibreQoS/wiki/v1.4 has the install instructions (for the truly daring there is the "heimdall" branch where the post 1.4 work is going) and we did a podcast about it here:
https://packetpushers.net/podcast/heavy-networking-666-improving-quality-of-experience-with-libreqos/

thx for any help you can offer!

--
I keep wondering if sqrt(flows) bdp is correct:
https://blog.cerowrt.org/post/juniper/
Dave T?ht CEO, TekLibre, LLC
_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog


From edepa at ieee.org  Wed Mar 15 09:38:46 2023
From: edepa at ieee.org (Etienne-Victor Depasquale)
Date: Tue, 14 Mar 2023 23:38:46 +0100
Subject: [AusNOG] Request for additional data points
Message-ID: <CAAcx0vBK0v3zYqjbwbV=wXjTf0PhpRkiGb+PGi7pdqHP=iXyWw@mail.gmail.com>

Good day AUSNOGers,

Last year, I collected some data for my study from AUSNOG.
I published a summary of the data for a short time and
recently sent a snapshot regarding metro area access technologies.

I am currently writing my paper and would love to have some additional data
points.

If anyone who has ***not yet contributed***
would like to do so, I am still collecting data here:
https://forms.gle/ypcCZvrHbKXRbXqn9


Sincerely,

Etienne

-- 
Ing. Etienne-Victor Depasquale
Assistant Lecturer
Department of Communications & Computer Engineering
Faculty of Information & Communication Technology
University of Malta
Web. https://www.um.edu.mt/profile/etiennedepasquale
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230314/8457a790/attachment.htm>

From mail at andrewwelch.net  Thu Mar 16 12:51:12 2023
From: mail at andrewwelch.net (Andrew Welch)
Date: Thu, 16 Mar 2023 12:21:12 +1030
Subject: [AusNOG] Vocus Datacentre outage.
In-Reply-To: <a412a0188100fc5979e7daeb81628876@frontapp.com>
References: <ME3PR01MB6878081AEFEDA25E520E1A3BFDB99@ME3PR01MB6878.ausprd01.prod.outlook.com>
 <CAMpV8Mpv98O-cqpGz+5SVRBsobsZzWAT0Rhy8UdiSQ6fUEiWjw@mail.gmail.com>
 <A7475E5D-495B-4B4B-91C1-169140FEACE1@bradleyamm.com>
 <18719e27-053c-2ec2-f48e-2b69d5114716@spectrum.com.au>
 <MEYP282MB2520B620B1438273E3A1009CDBBE9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <f35b78ea-6e27-1eb1-5314-20fde831056a@spectrum.com.au>
 <MEYP282MB2520A3C7BCA133967365ED77DBBE9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <SYBP282MB0538A5F3C118B6D6A4ED85F5AFBE9@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM>
 <8D195CA1-4E67-46F4-BF39-C9AED0616977@iperium.com.au>
 <f59b62f8-4ab8-19cc-9890-0f1e45a13eb7@spectrum.com.au>
 <CAOSsYkrKcwcfZZ5tDUKjcmr1guHSwQgvgSwNEio8HFD+PzdARg@mail.gmail.com>
 <000901d9561b$ea361960$bea24c20$@innaloo.net>
 <a412a0188100fc5979e7daeb81628876@frontapp.com>
 <076eb1f9595849319ad53e27dab953ed@zonenetworks.com.au>
 <a412a0188100fc5979e7daeb81628876@frontapp.com>
Message-ID: <CAKCOLifGrbav7zUZjzpMvWBqO8-FkmpyRMPLQzbPPdJfo9hfbw@mail.gmail.com>

Has anyone heard any details about the planned UPS maintenance that's
scheduled for tomorrow at VDC-MEL03? Only info I've seen is what's in the
last outage update, which is just that UPS-A is being replaced, nothing on
whether this is service impacting or anything like that.

Thanks,
Andrew Welch
mail at andrewwelch.net


On Tue, 14 Mar 2023 at 13:15, Jared Hirst | Servers Australia <
jared.hirst at serversaustralia.com.au> wrote:

> Good times back then Joel! It was a UPS issue back there too if I
> remember, and the A and B feeds were all fed into the one UPS?.
>
> A tonne of people shifted within that week, for me it was the way that it
> was palmed off, as you say, others might have issues but they own those
> issues and rectify them?
>
>
>
>
>
> Jared Hirst
> Chief Executive Officer
> P: (02) 8115 8801
> 4 Amy Close, Wyong, NSW 2259
> Need assistance? We are here 24/7 +61 2 8115 8888
> [image: Sent from Front]
>
> On March 14, 2023 at 12:26 PM GMT+11 joel at zonenetworks.com.au wrote:
>
> In EQX defence Sy1 didn?t go dark? it was multiple (3-5) rows of rack that
> lost power in 2009.
>
>
>
> We were one of the customers heavily impacted in 2009 but understood the
> reason why it happened and EQX fixed it all up.
>
>
>
> Vocus Doody st different story and now melb
>
>
>
> We pulled out of Doody after the drama we faced there.
>
>
>
> Joel.
>
>
>
> *From:* AusNOG <ausnog-bounces at lists.ausnog.net> *On Behalf Of *Nathan
> Brookfield
> *Sent:* Tuesday, 14 March 2023 11:54 AM
> *To:* Christopher Hawker <chris at thesysadmin.dev>
> *Cc:* ausnog at lists.ausnog.net
> *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>
>
>
> I knew I didn?t remember it for a reason ?
>
>
>
> *[AusNOG] Equinix Outage
> <https://lists.ausnog.net/pipermail/ausnog/2009-July/003239.html>*
>
> *lists.ausnog.net
> <https://lists.ausnog.net/pipermail/ausnog/2009-July/003239.html>*
>
> <https://lists.ausnog.net/pipermail/ausnog/2009-July/003239.html>
>
>
>
> On 14 Mar 2023, at 10:50, Christopher Hawker <chris at thesysadmin.dev>
> wrote:
>
>
>
> ?
>
> SY1 apparently experienced a full power outage back in July 2009,
> according to AusNOG Archives. Some reported full outages, some partial. Was
> not a DC fault though, looks like a much larger grid issue where a fault at
> the Bayswater Power Station in the Hunter Valley (which affected generators
> in QLD and VIC) caused about 1000MW to be dropped from the grid, resulting
> in rolling blackouts across QLD, NSW, TAS, SA and VIC.
>
>
>
> https://lists.ausnog.net/pipermail/ausnog/2009-July/003222.html
>
>
>
> CH.
> ------------------------------
>
> *From:* AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Nathan
> Brookfield <Nathan.Brookfield at iperium.com.au>
> *Sent:* Tuesday, March 14, 2023 11:40 AM
> *To:* Matt Perkins <matt at spectrum.com.au>; ausnog at lists.ausnog.net <
> ausnog at lists.ausnog.net>
> *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>
>
>
> Have 100% had A and B fail separately at SY1 but never both, thankfully!!!
>
>
>
> Could have been part of the facility though so don?t quote me.
>
>
>
> UK on the other hand, fail!
>
>
>
> *From:* Matt Perkins <matt at spectrum.com.au>
> *Sent:* Tuesday, 14 March 2023 11:30 AM
> *To:* Nathan Brookfield <Nathan.Brookfield at iperium.com.au>;
> ausnog at lists.ausnog.net
> *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>
>
>
> SY1 had A or B fail a few years back if my memory serves me correctly.
> But I was referring  London and Manchester which both had total outages in
> recent years.
>
> Matt.
>
>
>
> On 14/3/2023 11:26 am, Nathan Brookfield wrote:
>
> I?ve never had an A+B power rail failure at Equinix in 18 years I?ve been
> in there facilities, TOUCH WOOD!!!
>
>
>
>
> *Nathan Brookfield | VK2NAB *General Manager
>
> *p*: 1300 592 330 | *m*: 0412 266 008
> *e*: Nathan.brookfield at iperium.com.au | *w*: iperium.com.au
>
> Suite 4.02, 189 Kent Street Sydney NSW 2000
>
> <image001.png>
>
>
> *Your Telco Team*
>
> DISCLAIMER: This document is intended solely for the named addressee. This
> electronic communication, which includes any files or attachments thereto,
> contains proprietary or confidential information and may be privileged and
> otherwise protected under copyright or other applicable intellectual
> property laws. The use, disclosure, copying or distribution of any of the
> information contained in this document, by any person other than the
> addressee, is strictly prohibited. If you received this document in error,
> please contact the sender immediately and delete all the material from any
> computer. Confidentiality and legal privilege are not waived or lost by
> reason of mistaken delivery to you. Any views or opinions presented are
> solely those of the author and do not necessarily represent those of
> Iperium.
>
> WARNING: Computer viruses can be transmitted via email. The recipient
> should check this email and any attachments for the presence of viruses.
> Iperium accepts no liability for any damage caused by any virus transmitted
> by this email.
>
> *From:* AusNOG <ausnog-bounces at lists.ausnog.net>
> <ausnog-bounces at lists.ausnog.net> *On Behalf Of *Matt Perkins
> *Sent:* Tuesday, 14 March 2023 11:18 AM
> *To:* ausnog at lists.ausnog.net
> *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>
>
>
> Yea because Eqinix has never had a power outage. :)
>
> I think it's a little premature to judge at the moment. Doesn't sound good
> based on the current info but we have all had bad days im sure.
>
> Matt
>
>
>
> On 14/3/2023 11:06 am, Bradley Amm wrote:
>
> Friends like let friends put gear in a Vocus dtacenter. Only NextDC or
> Equinix :)
>
> Sent from my iPhone
>
>
>
> On 14 Mar 2023, at 7:08 am, DaZZa <dazzagibbs at gmail.com>
> <dazzagibbs at gmail.com> wrote:
>
> ?
>
> Doesn't say a whole lot about Vocus' data center design methodology, nor
> their periodic contingency testing.
>
>
>
> They're apparently replacing the entire UPS on Friday - guess we can all
> expect more outages then when they stuff that up.
>
>
>
> D
>
>
>
> On Mon, 13 Mar 2023 at 18:30, Chad Kelly <chad at cpkws.com.au> wrote:
>
> Thought I would create a new thread on this, as the issue wasn?t related
> to AussieBB.
>
> It is looking like this Melbourne outage is similar to the Sydney outage
> back in 2018, in that it appears to have been yet again caused by an issue
> with their UPS systems yet again.
>
> I?ll leave this hear
> https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/
> and this article on today?s outage as well.
>
>
> https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987
>
>
>
> Regards Chad.
>
>
>
>
>
>
>
> Chad Kelly
>
> Manager
>
> CPK Web Services
>
> Phone 03 52730246
>
> Web https://www.cpkws.com.au
>
> <image001.jpg>
>
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
> --
>
> veg?e?tar?i?an:
> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
> _______________________________________________
>
> AusNOG mailing list
>
> AusNOG at lists.ausnog.net
>
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
> --
>
> ** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL
>
>
>
> /* Matt Perkins
>
>         Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.
>
>         Office 1300 133 299     matt at spectrum.com.au
>
>         ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000
>
>
>
> */
>
> --
>
> ** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL
>
>
>
> /* Matt Perkins
>
>         Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.
>
>         Office 1300 133 299     matt at spectrum.com.au
>
>         ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000
>
>
>
> */
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230316/0014972e/attachment.htm>

From dazzagibbs at gmail.com  Thu Mar 16 13:13:45 2023
From: dazzagibbs at gmail.com (DaZZa)
Date: Thu, 16 Mar 2023 13:13:45 +1100
Subject: [AusNOG] Vocus Datacentre outage.
In-Reply-To: <CAKCOLifGrbav7zUZjzpMvWBqO8-FkmpyRMPLQzbPPdJfo9hfbw@mail.gmail.com>
References: <ME3PR01MB6878081AEFEDA25E520E1A3BFDB99@ME3PR01MB6878.ausprd01.prod.outlook.com>
 <CAMpV8Mpv98O-cqpGz+5SVRBsobsZzWAT0Rhy8UdiSQ6fUEiWjw@mail.gmail.com>
 <A7475E5D-495B-4B4B-91C1-169140FEACE1@bradleyamm.com>
 <18719e27-053c-2ec2-f48e-2b69d5114716@spectrum.com.au>
 <MEYP282MB2520B620B1438273E3A1009CDBBE9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <f35b78ea-6e27-1eb1-5314-20fde831056a@spectrum.com.au>
 <MEYP282MB2520A3C7BCA133967365ED77DBBE9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <SYBP282MB0538A5F3C118B6D6A4ED85F5AFBE9@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM>
 <8D195CA1-4E67-46F4-BF39-C9AED0616977@iperium.com.au>
 <f59b62f8-4ab8-19cc-9890-0f1e45a13eb7@spectrum.com.au>
 <CAOSsYkrKcwcfZZ5tDUKjcmr1guHSwQgvgSwNEio8HFD+PzdARg@mail.gmail.com>
 <000901d9561b$ea361960$bea24c20$@innaloo.net>
 <076eb1f9595849319ad53e27dab953ed@zonenetworks.com.au>
 <a412a0188100fc5979e7daeb81628876@frontapp.com>
 <CAKCOLifGrbav7zUZjzpMvWBqO8-FkmpyRMPLQzbPPdJfo9hfbw@mail.gmail.com>
Message-ID: <CAMpV8MqTDA0KAdWAJitgbrhpBAFO-bzq61-0kym9Yv=bJXnYYw@mail.gmail.com>

Nope. I've asked for an RFO/PIR for the outage, and been put in the queue
for when they actually produce it.

D

On Thu, 16 Mar 2023 at 12:51, Andrew Welch <mail at andrewwelch.net> wrote:

> Has anyone heard any details about the planned UPS maintenance that's
> scheduled for tomorrow at VDC-MEL03? Only info I've seen is what's in the
> last outage update, which is just that UPS-A is being replaced, nothing on
> whether this is service impacting or anything like that.
>
> Thanks,
> Andrew Welch
> mail at andrewwelch.net
>
>
> On Tue, 14 Mar 2023 at 13:15, Jared Hirst | Servers Australia <
> jared.hirst at serversaustralia.com.au> wrote:
>
>> Good times back then Joel! It was a UPS issue back there too if I
>> remember, and the A and B feeds were all fed into the one UPS?.
>>
>> A tonne of people shifted within that week, for me it was the way that
>> it was palmed off, as you say, others might have issues but they own those
>> issues and rectify them?
>>
>>
>>
>>
>>
>> Jared Hirst
>> Chief Executive Officer
>> P: (02) 8115 8801
>> 4 Amy Close, Wyong, NSW 2259
>> Need assistance? We are here 24/7 +61 2 8115 8888
>> [image: Sent from Front]
>>
>> On March 14, 2023 at 12:26 PM GMT+11 joel at zonenetworks.com.au wrote:
>>
>> In EQX defence Sy1 didn?t go dark? it was multiple (3-5) rows of rack
>> that lost power in 2009.
>>
>>
>>
>> We were one of the customers heavily impacted in 2009 but understood the
>> reason why it happened and EQX fixed it all up.
>>
>>
>>
>> Vocus Doody st different story and now melb
>>
>>
>>
>> We pulled out of Doody after the drama we faced there.
>>
>>
>>
>> Joel.
>>
>>
>>
>> *From:* AusNOG <ausnog-bounces at lists.ausnog.net> *On Behalf Of *Nathan
>> Brookfield
>> *Sent:* Tuesday, 14 March 2023 11:54 AM
>> *To:* Christopher Hawker <chris at thesysadmin.dev>
>> *Cc:* ausnog at lists.ausnog.net
>> *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>>
>>
>>
>> I knew I didn?t remember it for a reason ?
>>
>>
>>
>> *[AusNOG] Equinix Outage
>> <https://lists.ausnog.net/pipermail/ausnog/2009-July/003239.html>*
>>
>> *lists.ausnog.net
>> <https://lists.ausnog.net/pipermail/ausnog/2009-July/003239.html>*
>>
>> <https://lists.ausnog.net/pipermail/ausnog/2009-July/003239.html>
>>
>>
>>
>> On 14 Mar 2023, at 10:50, Christopher Hawker <chris at thesysadmin.dev>
>> wrote:
>>
>>
>>
>> ?
>>
>> SY1 apparently experienced a full power outage back in July 2009,
>> according to AusNOG Archives. Some reported full outages, some partial. Was
>> not a DC fault though, looks like a much larger grid issue where a fault at
>> the Bayswater Power Station in the Hunter Valley (which affected generators
>> in QLD and VIC) caused about 1000MW to be dropped from the grid, resulting
>> in rolling blackouts across QLD, NSW, TAS, SA and VIC.
>>
>>
>>
>> https://lists.ausnog.net/pipermail/ausnog/2009-July/003222.html
>>
>>
>>
>> CH.
>> ------------------------------
>>
>> *From:* AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Nathan
>> Brookfield <Nathan.Brookfield at iperium.com.au>
>> *Sent:* Tuesday, March 14, 2023 11:40 AM
>> *To:* Matt Perkins <matt at spectrum.com.au>; ausnog at lists.ausnog.net <
>> ausnog at lists.ausnog.net>
>> *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>>
>>
>>
>> Have 100% had A and B fail separately at SY1 but never both, thankfully!!!
>>
>>
>>
>> Could have been part of the facility though so don?t quote me.
>>
>>
>>
>> UK on the other hand, fail!
>>
>>
>>
>> *From:* Matt Perkins <matt at spectrum.com.au>
>> *Sent:* Tuesday, 14 March 2023 11:30 AM
>> *To:* Nathan Brookfield <Nathan.Brookfield at iperium.com.au>;
>> ausnog at lists.ausnog.net
>> *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>>
>>
>>
>> SY1 had A or B fail a few years back if my memory serves me correctly.
>> But I was referring  London and Manchester which both had total outages in
>> recent years.
>>
>> Matt.
>>
>>
>>
>> On 14/3/2023 11:26 am, Nathan Brookfield wrote:
>>
>> I?ve never had an A+B power rail failure at Equinix in 18 years I?ve been
>> in there facilities, TOUCH WOOD!!!
>>
>>
>>
>>
>> *Nathan Brookfield | VK2NAB *General Manager
>>
>> *p*: 1300 592 330 | *m*: 0412 266 008
>> *e*: Nathan.brookfield at iperium.com.au | *w*: iperium.com.au
>>
>> Suite 4.02, 189 Kent Street Sydney NSW 2000
>>
>> <image001.png>
>>
>>
>> *Your Telco Team*
>>
>> DISCLAIMER: This document is intended solely for the named addressee.
>> This electronic communication, which includes any files or attachments
>> thereto, contains proprietary or confidential information and may be
>> privileged and otherwise protected under copyright or other applicable
>> intellectual property laws. The use, disclosure, copying or distribution of
>> any of the information contained in this document, by any person other than
>> the addressee, is strictly prohibited. If you received this document in
>> error, please contact the sender immediately and delete all the material
>> from any computer. Confidentiality and legal privilege are not waived or
>> lost by reason of mistaken delivery to you. Any views or opinions presented
>> are solely those of the author and do not necessarily represent those of
>> Iperium.
>>
>> WARNING: Computer viruses can be transmitted via email. The recipient
>> should check this email and any attachments for the presence of viruses.
>> Iperium accepts no liability for any damage caused by any virus transmitted
>> by this email.
>>
>> *From:* AusNOG <ausnog-bounces at lists.ausnog.net>
>> <ausnog-bounces at lists.ausnog.net> *On Behalf Of *Matt Perkins
>> *Sent:* Tuesday, 14 March 2023 11:18 AM
>> *To:* ausnog at lists.ausnog.net
>> *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>>
>>
>>
>> Yea because Eqinix has never had a power outage. :)
>>
>> I think it's a little premature to judge at the moment. Doesn't sound
>> good based on the current info but we have all had bad days im sure.
>>
>> Matt
>>
>>
>>
>> On 14/3/2023 11:06 am, Bradley Amm wrote:
>>
>> Friends like let friends put gear in a Vocus dtacenter. Only NextDC or
>> Equinix :)
>>
>> Sent from my iPhone
>>
>>
>>
>> On 14 Mar 2023, at 7:08 am, DaZZa <dazzagibbs at gmail.com>
>> <dazzagibbs at gmail.com> wrote:
>>
>> ?
>>
>> Doesn't say a whole lot about Vocus' data center design methodology, nor
>> their periodic contingency testing.
>>
>>
>>
>> They're apparently replacing the entire UPS on Friday - guess we can all
>> expect more outages then when they stuff that up.
>>
>>
>>
>> D
>>
>>
>>
>> On Mon, 13 Mar 2023 at 18:30, Chad Kelly <chad at cpkws.com.au> wrote:
>>
>> Thought I would create a new thread on this, as the issue wasn?t related
>> to AussieBB.
>>
>> It is looking like this Melbourne outage is similar to the Sydney outage
>> back in 2018, in that it appears to have been yet again caused by an issue
>> with their UPS systems yet again.
>>
>> I?ll leave this hear
>> https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/
>> and this article on today?s outage as well.
>>
>>
>> https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987
>>
>>
>>
>> Regards Chad.
>>
>>
>>
>>
>>
>>
>>
>> Chad Kelly
>>
>> Manager
>>
>> CPK Web Services
>>
>> Phone 03 52730246
>>
>> Web https://www.cpkws.com.au
>>
>> <image001.jpg>
>>
>>
>>
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
>>
>>
>> --
>>
>> veg?e?tar?i?an:
>> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>>
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
>>
>>
>> _______________________________________________
>>
>> AusNOG mailing list
>>
>> AusNOG at lists.ausnog.net
>>
>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
>> --
>>
>> ** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL
>>
>>
>>
>> /* Matt Perkins
>>
>>         Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.
>>
>>         Office 1300 133 299     matt at spectrum.com.au
>>
>>         ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000
>>
>>
>>
>> */
>>
>> --
>>
>> ** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL
>>
>>
>>
>> /* Matt Perkins
>>
>>         Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.
>>
>>         Office 1300 133 299     matt at spectrum.com.au
>>
>>         ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000
>>
>>
>>
>> */
>>
>>
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>


-- 
veg?e?tar?i?an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230316/ee5bcd10/attachment.htm>

From ben_colt at yahoo.com  Thu Mar 16 15:02:14 2023
From: ben_colt at yahoo.com (Ben Colt)
Date: Thu, 16 Mar 2023 04:02:14 +0000 (UTC)
Subject: [AusNOG] NBN to offshore NOC?
References: <662796588.508705.1678939334531.ref@mail.yahoo.com>
Message-ID: <662796588.508705.1678939334531@mail.yahoo.com>

Talk today about this happening. It would be disaster for jobs and technology support if it happened. Australia the dumb country?


Sent from Yahoo Mail for iPhone
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230316/a422e2c0/attachment.htm>

From luke.t at tncrew.com.au  Thu Mar 16 16:06:04 2023
From: luke.t at tncrew.com.au (Luke Thompson)
Date: Thu, 16 Mar 2023 16:06:04 +1100
Subject: [AusNOG] NBN to offshore NOC?
In-Reply-To: <662796588.508705.1678939334531@mail.yahoo.com>
References: <662796588.508705.1678939334531.ref@mail.yahoo.com>
 <662796588.508705.1678939334531@mail.yahoo.com>
Message-ID: <186e8cf9660.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>

I'm fairly sure field techs have to ring through to an offshore team for 
fibre splice checking / approvals already. So they have some crews overseas 
now for several years (unless this has since been onshored, though I'm 
doubtful).

Many techs have remarked about how long the queue can be to speak with that 
team, language barrier, quality of staff, etc.

NOC offshoring is nightmare territory - depending on how it's done. The 
national broadband network ought to be just that.

Cheers,

Luke Thompson
Chief Technical Officer

The Network Crew Pty Ltd
https://thenetworkcrew.com.au
On 16 March 2023 3:04:57 pm Ben Colt <ben_colt at yahoo.com> wrote:
> Talk today about this happening. It would be disaster for jobs and 
> technology support if it happened. Australia the dumb country?
>
>
> Sent from Yahoo Mail for iPhone
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230316/662c8df4/attachment.htm>

From ausnog at studio442.com.au  Thu Mar 16 19:50:23 2023
From: ausnog at studio442.com.au (Julien Goodwin)
Date: Thu, 16 Mar 2023 19:50:23 +1100
Subject: [AusNOG] NBN to offshore NOC?
In-Reply-To: <186e8cf9660.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
References: <662796588.508705.1678939334531.ref@mail.yahoo.com>
 <662796588.508705.1678939334531@mail.yahoo.com>
 <186e8cf9660.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
Message-ID: <10280944-00e0-3b82-8694-42d26920e52a@studio442.com.au>

On 16/3/23 4:06?pm, Luke Thompson wrote:
> NOC offshoring is nightmare territory - depending on how it's done. The 
> national broadband network ought to be just that.

I'll mostly agree with that.

Pure NOC offshoring isn't great. Splitting a NOC into two or three sites 
for follow-the-sun coverage can work really well if smartly managed, 
saves trying to find good staff for a night shift, although of course it 
does add plenty of issues on its own.

Even just NOC remote from engineering, having a NOC in Brisbane, but 
engineering all done out of Melbourne has plenty of issues if you don't 
work strongly to ensure the various people actually talk to each other.

Or, if you're large enough, first tier separate from second can be much 
the same.

I'm currently working with our corporate network group, and it's my 
constant lament that I can't just sit in a room where their NOC folk are 
working to learn the things that aren't getting escalated, processes 
that don't quite work properly, and other things I might not see from 
reading their tickets. (The reasons why I can't just do this are long 
and far off topic for this)

If NBN were spinning up a NOC in the UK, Ireland, or possibly the 
US/Canada east coast that could be a very good idea, but it wouldn't be 
the cheap one.

If NBN are shifting things done in Australia as an attempt at a cost 
saving, I doubt it really will by the time all relevant costs are 
considered.

From luke.t at tncrew.com.au  Thu Mar 16 22:40:26 2023
From: luke.t at tncrew.com.au (Luke Thompson)
Date: Thu, 16 Mar 2023 22:40:26 +1100
Subject: [AusNOG] NBN to offshore NOC?
In-Reply-To: <10280944-00e0-3b82-8694-42d26920e52a@studio442.com.au>
References: <662796588.508705.1678939334531.ref@mail.yahoo.com>
 <662796588.508705.1678939334531@mail.yahoo.com>
 <186e8cf9660.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
 <10280944-00e0-3b82-8694-42d26920e52a@studio442.com.au>
Message-ID: <186ea38a410.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>

I'm with you on follow the sun, much like iiNet & Co. did back in the day 
(very effectively, almost a work of art it was so good) with some functions.

In the context of a national network though, much as with Telstra the 
people are its greatest asset - and while it can be nightmarish landing the 
right folk especially over odd hours, building out a collaborative 
powerhouse of a team on-shore is the beauty spot if it fits. Especially 
with NOCs/SOCs.

Follow the sun isn't blatant off shoring as you say. Removing local 
functions entirely though will land you with internal chaos and severe 
problems down the track when trying to insource.

Having multiple sites has its benefits (and challenges), I just don't think 
binning local makes any sort of sense with the NBN NOC.

On 16 March 2023 7:51:12 pm Julien Goodwin <ausnog at studio442.com.au> wrote:

> On 16/3/23 4:06?pm, Luke Thompson wrote:
>> NOC offshoring is nightmare territory - depending on how it's done. The
>> national broadband network ought to be just that.
>
> I'll mostly agree with that.
>
> Pure NOC offshoring isn't great. Splitting a NOC into two or three sites
> for follow-the-sun coverage can work really well if smartly managed,
> saves trying to find good staff for a night shift, although of course it
> does add plenty of issues on its own.
>
> Even just NOC remote from engineering, having a NOC in Brisbane, but
> engineering all done out of Melbourne has plenty of issues if you don't
> work strongly to ensure the various people actually talk to each other.
>
> Or, if you're large enough, first tier separate from second can be much
> the same.
>
> I'm currently working with our corporate network group, and it's my
> constant lament that I can't just sit in a room where their NOC folk are
> working to learn the things that aren't getting escalated, processes
> that don't quite work properly, and other things I might not see from
> reading their tickets. (The reasons why I can't just do this are long
> and far off topic for this)
>
> If NBN were spinning up a NOC in the UK, Ireland, or possibly the
> US/Canada east coast that could be a very good idea, but it wouldn't be
> the cheap one.
>
> If NBN are shifting things done in Australia as an attempt at a cost
> saving, I doubt it really will by the time all relevant costs are
> considered.
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230316/95a0aad5/attachment.htm>

From mmc at mmc.com.au  Fri Mar 17 11:01:06 2023
From: mmc at mmc.com.au (Matthew Moyle-Croft)
Date: Fri, 17 Mar 2023 10:31:06 +1030
Subject: [AusNOG] NBN to offshore NOC?
In-Reply-To: <186ea38a410.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
References: <662796588.508705.1678939334531.ref@mail.yahoo.com>
 <662796588.508705.1678939334531@mail.yahoo.com>
 <186e8cf9660.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
 <10280944-00e0-3b82-8694-42d26920e52a@studio442.com.au>
 <186ea38a410.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
Message-ID: <CAANqdPdEzryrkoqU_dhZfVUGFeK2o8OGPm43qJdUxZDqbGTQQw@mail.gmail.com>

Saving money on one thing can cost a LOT in other ways. The large siloed
organisation run by accountants story.

On Thu, Mar 16, 2023 at 10:10?PM Luke Thompson <luke.t at tncrew.com.au> wrote:

> I'm with you on follow the sun, much like iiNet & Co. did back in the day
> (very effectively, almost a work of art it was so good) with some
> functions.
>
> In the context of a national network though, much as with Telstra the
> people are its greatest asset - and while it can be nightmarish landing the
> right folk especially over odd hours, building out a collaborative
> powerhouse of a team on-shore is the beauty spot if it fits. Especially
> with NOCs/SOCs.
>
> Follow the sun isn't blatant off shoring as you say. Removing local
> functions entirely though will land you with internal chaos and severe
> problems down the track when trying to insource.
>
> Having multiple sites has its benefits (and challenges), I just don't
> think binning local makes any sort of sense with the NBN NOC.
>
> On 16 March 2023 7:51:12 pm Julien Goodwin <ausnog at studio442.com.au>
> wrote:
>
> On 16/3/23 4:06?pm, Luke Thompson wrote:
>>
>>> NOC offshoring is nightmare territory - depending on how it's done. The
>>> national broadband network ought to be just that.
>>>
>>
>> I'll mostly agree with that.
>>
>> Pure NOC offshoring isn't great. Splitting a NOC into two or three sites
>> for follow-the-sun coverage can work really well if smartly managed,
>> saves trying to find good staff for a night shift, although of course it
>> does add plenty of issues on its own.
>>
>> Even just NOC remote from engineering, having a NOC in Brisbane, but
>> engineering all done out of Melbourne has plenty of issues if you don't
>> work strongly to ensure the various people actually talk to each other.
>>
>> Or, if you're large enough, first tier separate from second can be much
>> the same.
>>
>> I'm currently working with our corporate network group, and it's my
>> constant lament that I can't just sit in a room where their NOC folk are
>> working to learn the things that aren't getting escalated, processes
>> that don't quite work properly, and other things I might not see from
>> reading their tickets. (The reasons why I can't just do this are long
>> and far off topic for this)
>>
>> If NBN were spinning up a NOC in the UK, Ireland, or possibly the
>> US/Canada east coast that could be a very good idea, but it wouldn't be
>> the cheap one.
>>
>> If NBN are shifting things done in Australia as an attempt at a cost
>> saving, I doubt it really will by the time all relevant costs are
>> considered.
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230317/2e6d42da/attachment-0001.htm>

From jaedwards at gmail.com  Fri Mar 17 11:29:07 2023
From: jaedwards at gmail.com (John Edwards)
Date: Fri, 17 Mar 2023 10:59:07 +1030
Subject: [AusNOG] NBN to offshore NOC?
In-Reply-To: <CAANqdPdEzryrkoqU_dhZfVUGFeK2o8OGPm43qJdUxZDqbGTQQw@mail.gmail.com>
References: <662796588.508705.1678939334531.ref@mail.yahoo.com>
 <662796588.508705.1678939334531@mail.yahoo.com>
 <186e8cf9660.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
 <10280944-00e0-3b82-8694-42d26920e52a@studio442.com.au>
 <186ea38a410.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
 <CAANqdPdEzryrkoqU_dhZfVUGFeK2o8OGPm43qJdUxZDqbGTQQw@mail.gmail.com>
Message-ID: <CAOSsYkoR3zrtpFm3aU+0Pj5PqJ3YE4pc7075noFg-+dxGwVP6A@mail.gmail.com>

NBN's purpose is to lift the digital capability of Australia.

https://www.nbnco.com.au/corporate-information/about-nbn-co/our-purpose

The takeaway from offshoring activities is that the Australian government
does not consider Network Operations to be a digital capability.

John



On Fri, 17 Mar 2023 at 10:31, Matthew Moyle-Croft <mmc at mmc.com.au> wrote:

> Saving money on one thing can cost a LOT in other ways. The large siloed
> organisation run by accountants story.
>
> On Thu, Mar 16, 2023 at 10:10?PM Luke Thompson <luke.t at tncrew.com.au>
> wrote:
>
>> I'm with you on follow the sun, much like iiNet & Co. did back in the day
>> (very effectively, almost a work of art it was so good) with some
>> functions.
>>
>> In the context of a national network though, much as with Telstra the
>> people are its greatest asset - and while it can be nightmarish landing the
>> right folk especially over odd hours, building out a collaborative
>> powerhouse of a team on-shore is the beauty spot if it fits. Especially
>> with NOCs/SOCs.
>>
>> Follow the sun isn't blatant off shoring as you say. Removing local
>> functions entirely though will land you with internal chaos and severe
>> problems down the track when trying to insource.
>>
>> Having multiple sites has its benefits (and challenges), I just don't
>> think binning local makes any sort of sense with the NBN NOC.
>>
>> On 16 March 2023 7:51:12 pm Julien Goodwin <ausnog at studio442.com.au>
>> wrote:
>>
>> On 16/3/23 4:06?pm, Luke Thompson wrote:
>>>
>>>> NOC offshoring is nightmare territory - depending on how it's done. The
>>>> national broadband network ought to be just that.
>>>>
>>>
>>> I'll mostly agree with that.
>>>
>>> Pure NOC offshoring isn't great. Splitting a NOC into two or three sites
>>> for follow-the-sun coverage can work really well if smartly managed,
>>> saves trying to find good staff for a night shift, although of course it
>>> does add plenty of issues on its own.
>>>
>>> Even just NOC remote from engineering, having a NOC in Brisbane, but
>>> engineering all done out of Melbourne has plenty of issues if you don't
>>> work strongly to ensure the various people actually talk to each other.
>>>
>>> Or, if you're large enough, first tier separate from second can be much
>>> the same.
>>>
>>> I'm currently working with our corporate network group, and it's my
>>> constant lament that I can't just sit in a room where their NOC folk are
>>> working to learn the things that aren't getting escalated, processes
>>> that don't quite work properly, and other things I might not see from
>>> reading their tickets. (The reasons why I can't just do this are long
>>> and far off topic for this)
>>>
>>> If NBN were spinning up a NOC in the UK, Ireland, or possibly the
>>> US/Canada east coast that could be a very good idea, but it wouldn't be
>>> the cheap one.
>>>
>>> If NBN are shifting things done in Australia as an attempt at a cost
>>> saving, I doubt it really will by the time all relevant costs are
>>> considered.
>>> _______________________________________________
>>> AusNOG mailing list
>>> AusNOG at lists.ausnog.net
>>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>>
>>
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230317/219c1917/attachment.htm>

From mmc at mmc.com.au  Fri Mar 17 11:35:10 2023
From: mmc at mmc.com.au (Matthew Moyle-Croft)
Date: Fri, 17 Mar 2023 11:05:10 +1030
Subject: [AusNOG] NBN to offshore NOC?
In-Reply-To: <CAOSsYkoR3zrtpFm3aU+0Pj5PqJ3YE4pc7075noFg-+dxGwVP6A@mail.gmail.com>
References: <662796588.508705.1678939334531.ref@mail.yahoo.com>
 <662796588.508705.1678939334531@mail.yahoo.com>
 <186e8cf9660.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
 <10280944-00e0-3b82-8694-42d26920e52a@studio442.com.au>
 <186ea38a410.283d.934dbec5914d2454830359510b9fa213@tncrew.com.au>
 <CAANqdPdEzryrkoqU_dhZfVUGFeK2o8OGPm43qJdUxZDqbGTQQw@mail.gmail.com>
 <CAOSsYkoR3zrtpFm3aU+0Pj5PqJ3YE4pc7075noFg-+dxGwVP6A@mail.gmail.com>
Message-ID: <CAANqdPcV==2cd-HHKiekkw3NWFYtaBYtHw1ycxJj-LoQuA9cCg@mail.gmail.com>

Can someone print that and show NBN next time they claim their purpose
above all else is to "deliver an investment grade return" when they argue
about price/reliability/competition/etc?

On Fri, Mar 17, 2023 at 10:59?AM John Edwards <jaedwards at gmail.com> wrote:

> NBN's purpose is to lift the digital capability of Australia.
>
> https://www.nbnco.com.au/corporate-information/about-nbn-co/our-purpose
>
> The takeaway from offshoring activities is that the Australian government
> does not consider Network Operations to be a digital capability.
>
> John
>
>
>
> On Fri, 17 Mar 2023 at 10:31, Matthew Moyle-Croft <mmc at mmc.com.au> wrote:
>
>> Saving money on one thing can cost a LOT in other ways. The large siloed
>> organisation run by accountants story.
>>
>> On Thu, Mar 16, 2023 at 10:10?PM Luke Thompson <luke.t at tncrew.com.au>
>> wrote:
>>
>>> I'm with you on follow the sun, much like iiNet & Co. did back in the
>>> day (very effectively, almost a work of art it was so good) with some
>>> functions.
>>>
>>> In the context of a national network though, much as with Telstra the
>>> people are its greatest asset - and while it can be nightmarish landing the
>>> right folk especially over odd hours, building out a collaborative
>>> powerhouse of a team on-shore is the beauty spot if it fits. Especially
>>> with NOCs/SOCs.
>>>
>>> Follow the sun isn't blatant off shoring as you say. Removing local
>>> functions entirely though will land you with internal chaos and severe
>>> problems down the track when trying to insource.
>>>
>>> Having multiple sites has its benefits (and challenges), I just don't
>>> think binning local makes any sort of sense with the NBN NOC.
>>>
>>> On 16 March 2023 7:51:12 pm Julien Goodwin <ausnog at studio442.com.au>
>>> wrote:
>>>
>>> On 16/3/23 4:06?pm, Luke Thompson wrote:
>>>>
>>>>> NOC offshoring is nightmare territory - depending on how it's done.
>>>>> The
>>>>> national broadband network ought to be just that.
>>>>>
>>>>
>>>> I'll mostly agree with that.
>>>>
>>>> Pure NOC offshoring isn't great. Splitting a NOC into two or three
>>>> sites
>>>> for follow-the-sun coverage can work really well if smartly managed,
>>>> saves trying to find good staff for a night shift, although of course
>>>> it
>>>> does add plenty of issues on its own.
>>>>
>>>> Even just NOC remote from engineering, having a NOC in Brisbane, but
>>>> engineering all done out of Melbourne has plenty of issues if you don't
>>>> work strongly to ensure the various people actually talk to each other.
>>>>
>>>> Or, if you're large enough, first tier separate from second can be much
>>>> the same.
>>>>
>>>> I'm currently working with our corporate network group, and it's my
>>>> constant lament that I can't just sit in a room where their NOC folk
>>>> are
>>>> working to learn the things that aren't getting escalated, processes
>>>> that don't quite work properly, and other things I might not see from
>>>> reading their tickets. (The reasons why I can't just do this are long
>>>> and far off topic for this)
>>>>
>>>> If NBN were spinning up a NOC in the UK, Ireland, or possibly the
>>>> US/Canada east coast that could be a very good idea, but it wouldn't be
>>>> the cheap one.
>>>>
>>>> If NBN are shifting things done in Australia as an attempt at a cost
>>>> saving, I doubt it really will by the time all relevant costs are
>>>> considered.
>>>> _______________________________________________
>>>> AusNOG mailing list
>>>> AusNOG at lists.ausnog.net
>>>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>>>
>>>
>>> _______________________________________________
>>> AusNOG mailing list
>>> AusNOG at lists.ausnog.net
>>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>>
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230317/a7c455c0/attachment.htm>

From ausnog at perthwebhosting.net  Fri Mar 17 15:07:49 2023
From: ausnog at perthwebhosting.net (ausnog at perthwebhosting.net)
Date: Fri, 17 Mar 2023 12:07:49 +0800
Subject: [AusNOG] iiNet/Westnet Hosted domain passwords
Message-ID: <00a901d95886$0a36b640$1ea422c0$@perthwebhosting.net>

Hi All,

 

If anyone works for iiNet/Westnet that is on the list and works in the
hosting department, we have a few clients wishing to transfer their
international TLDs out to us. When we login to their iiNet/WN control panel
retrieve the EPP code we get an error message.

 

Domain Password is unknown for domain id=xxxxx
Error Code: 0xb58872c0.
Report ID: 2287534

 

I have at least 6 clients that have all rung in to the iiNet support desk
waited on the phone for over an hour only to be get told that iiNet/Westnet
will get back to them but they don't. Can someone get back to us off-list
please?

 

Thanks

Cameron 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230317/86eacb66/attachment.htm>

From ausnog at perthwebhosting.net  Fri Mar 17 15:40:08 2023
From: ausnog at perthwebhosting.net (ausnog at perthwebhosting.net)
Date: Fri, 17 Mar 2023 12:40:08 +0800
Subject: [AusNOG] iiNet/Westnet Hosted domain passwords
In-Reply-To: <e2142365-c82f-55e1-fa5d-c4d24ef3a5ac@hostaway.net.au>
References: <00a901d95886$0a36b640$1ea422c0$@perthwebhosting.net>
 <e2142365-c82f-55e1-fa5d-c4d24ef3a5ac@hostaway.net.au>
Message-ID: <00b501d9588a$8d875cd0$a8961670$@perthwebhosting.net>

Yep, Tried sync?ing the domain to the registrar but no go. 

 

Synchronization from registrar failed.

 

Ta

C.

From: Ender <ender at hostaway.net.au> 
Sent: Friday, 17 March 2023 12:13 PM
To: ausnog at perthwebhosting.net
Subject: Re: [AusNOG] iiNet/Westnet Hosted domain passwords

 

Hi Cameron,

Have you tried using the "Sync from Registry" button in ODIN?
We've done a few dozen transfers this week, and that (sometimes) helps sync the domain password in sometimes.

Responses from Westnet have taken about two days on these issues, they've given far too many customers the boot to cope with demand.

Cheers,
 Ender
 HostAway Support

On 17/3/23 12:07, ausnog at perthwebhosting.net <mailto:ausnog at perthwebhosting.net>  wrote:

Hi All,

 

If anyone works for iiNet/Westnet that is on the list and works in the hosting department, we have a few clients wishing to transfer their international TLDs out to us. When we login to their iiNet/WN control panel retrieve the EPP code we get an error message.

 

Domain Password is unknown for domain id=xxxxx
Error Code: 0xb58872c0.
Report ID: 2287534

 

I have at least 6 clients that have all rung in to the iiNet support desk waited on the phone for over an hour only to be get told that iiNet/Westnet will get back to them but they don?t. Can someone get back to us off-list please?

 

Thanks

Cameron 





_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net <mailto:AusNOG at lists.ausnog.net> 
https://lists.ausnog.net/mailman/listinfo/ausnog

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230317/303ca5ab/attachment.htm>

From ttdykes at gmail.com  Sat Mar 18 09:33:18 2023
From: ttdykes at gmail.com (Tim Dykes)
Date: Sat, 18 Mar 2023 09:33:18 +1100
Subject: [AusNOG] Vocus Datacentre outage.
In-Reply-To: <CAMpV8MqTDA0KAdWAJitgbrhpBAFO-bzq61-0kym9Yv=bJXnYYw@mail.gmail.com>
References: <ME3PR01MB6878081AEFEDA25E520E1A3BFDB99@ME3PR01MB6878.ausprd01.prod.outlook.com>
 <CAMpV8Mpv98O-cqpGz+5SVRBsobsZzWAT0Rhy8UdiSQ6fUEiWjw@mail.gmail.com>
 <A7475E5D-495B-4B4B-91C1-169140FEACE1@bradleyamm.com>
 <18719e27-053c-2ec2-f48e-2b69d5114716@spectrum.com.au>
 <MEYP282MB2520B620B1438273E3A1009CDBBE9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <f35b78ea-6e27-1eb1-5314-20fde831056a@spectrum.com.au>
 <MEYP282MB2520A3C7BCA133967365ED77DBBE9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <SYBP282MB0538A5F3C118B6D6A4ED85F5AFBE9@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM>
 <8D195CA1-4E67-46F4-BF39-C9AED0616977@iperium.com.au>
 <f59b62f8-4ab8-19cc-9890-0f1e45a13eb7@spectrum.com.au>
 <CAOSsYkrKcwcfZZ5tDUKjcmr1guHSwQgvgSwNEio8HFD+PzdARg@mail.gmail.com>
 <000901d9561b$ea361960$bea24c20$@innaloo.net>
 <076eb1f9595849319ad53e27dab953ed@zonenetworks.com.au>
 <a412a0188100fc5979e7daeb81628876@frontapp.com>
 <CAKCOLifGrbav7zUZjzpMvWBqO8-FkmpyRMPLQzbPPdJfo9hfbw@mail.gmail.com>
 <CAMpV8MqTDA0KAdWAJitgbrhpBAFO-bzq61-0kym9Yv=bJXnYYw@mail.gmail.com>
Message-ID: <CAJ=3pYERESZjGTpSruyT-HVwJh6TUt-8H-VmBtNasWmZcsLXog@mail.gmail.com>

Current outage is unrelated, suspected fibre cut in Melbourne... Incase
others are checking this thread for the same reason I just did.

On Thu, 16 Mar 2023, 1:14 pm DaZZa, <dazzagibbs at gmail.com> wrote:

> Nope. I've asked for an RFO/PIR for the outage, and been put in the queue
> for when they actually produce it.
>
> D
>
> On Thu, 16 Mar 2023 at 12:51, Andrew Welch <mail at andrewwelch.net> wrote:
>
>> Has anyone heard any details about the planned UPS maintenance that's
>> scheduled for tomorrow at VDC-MEL03? Only info I've seen is what's in the
>> last outage update, which is just that UPS-A is being replaced, nothing on
>> whether this is service impacting or anything like that.
>>
>> Thanks,
>> Andrew Welch
>> mail at andrewwelch.net
>>
>>
>> On Tue, 14 Mar 2023 at 13:15, Jared Hirst | Servers Australia <
>> jared.hirst at serversaustralia.com.au> wrote:
>>
>>> Good times back then Joel! It was a UPS issue back there too if I
>>> remember, and the A and B feeds were all fed into the one UPS?.
>>>
>>> A tonne of people shifted within that week, for me it was the way that
>>> it was palmed off, as you say, others might have issues but they own those
>>> issues and rectify them?
>>>
>>>
>>>
>>>
>>>
>>> Jared Hirst
>>> Chief Executive Officer
>>> P: (02) 8115 8801
>>> 4 Amy Close, Wyong, NSW 2259
>>> Need assistance? We are here 24/7 +61 2 8115 8888
>>> [image: Sent from Front]
>>>
>>> On March 14, 2023 at 12:26 PM GMT+11 joel at zonenetworks.com.au wrote:
>>>
>>> In EQX defence Sy1 didn?t go dark? it was multiple (3-5) rows of rack
>>> that lost power in 2009.
>>>
>>>
>>>
>>> We were one of the customers heavily impacted in 2009 but understood the
>>> reason why it happened and EQX fixed it all up.
>>>
>>>
>>>
>>> Vocus Doody st different story and now melb
>>>
>>>
>>>
>>> We pulled out of Doody after the drama we faced there.
>>>
>>>
>>>
>>> Joel.
>>>
>>>
>>>
>>> *From:* AusNOG <ausnog-bounces at lists.ausnog.net> *On Behalf Of *Nathan
>>> Brookfield
>>> *Sent:* Tuesday, 14 March 2023 11:54 AM
>>> *To:* Christopher Hawker <chris at thesysadmin.dev>
>>> *Cc:* ausnog at lists.ausnog.net
>>> *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>>>
>>>
>>>
>>> I knew I didn?t remember it for a reason ?
>>>
>>>
>>>
>>> *[AusNOG] Equinix Outage
>>> <https://lists.ausnog.net/pipermail/ausnog/2009-July/003239.html>*
>>>
>>> *lists.ausnog.net
>>> <https://lists.ausnog.net/pipermail/ausnog/2009-July/003239.html>*
>>>
>>> <https://lists.ausnog.net/pipermail/ausnog/2009-July/003239.html>
>>>
>>>
>>>
>>> On 14 Mar 2023, at 10:50, Christopher Hawker <chris at thesysadmin.dev>
>>> wrote:
>>>
>>>
>>>
>>> ?
>>>
>>> SY1 apparently experienced a full power outage back in July 2009,
>>> according to AusNOG Archives. Some reported full outages, some partial. Was
>>> not a DC fault though, looks like a much larger grid issue where a fault at
>>> the Bayswater Power Station in the Hunter Valley (which affected generators
>>> in QLD and VIC) caused about 1000MW to be dropped from the grid, resulting
>>> in rolling blackouts across QLD, NSW, TAS, SA and VIC.
>>>
>>>
>>>
>>> https://lists.ausnog.net/pipermail/ausnog/2009-July/003222.html
>>>
>>>
>>>
>>> CH.
>>> ------------------------------
>>>
>>> *From:* AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Nathan
>>> Brookfield <Nathan.Brookfield at iperium.com.au>
>>> *Sent:* Tuesday, March 14, 2023 11:40 AM
>>> *To:* Matt Perkins <matt at spectrum.com.au>; ausnog at lists.ausnog.net <
>>> ausnog at lists.ausnog.net>
>>> *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>>>
>>>
>>>
>>> Have 100% had A and B fail separately at SY1 but never both,
>>> thankfully!!!
>>>
>>>
>>>
>>> Could have been part of the facility though so don?t quote me.
>>>
>>>
>>>
>>> UK on the other hand, fail!
>>>
>>>
>>>
>>> *From:* Matt Perkins <matt at spectrum.com.au>
>>> *Sent:* Tuesday, 14 March 2023 11:30 AM
>>> *To:* Nathan Brookfield <Nathan.Brookfield at iperium.com.au>;
>>> ausnog at lists.ausnog.net
>>> *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>>>
>>>
>>>
>>> SY1 had A or B fail a few years back if my memory serves me correctly.
>>> But I was referring  London and Manchester which both had total outages in
>>> recent years.
>>>
>>> Matt.
>>>
>>>
>>>
>>> On 14/3/2023 11:26 am, Nathan Brookfield wrote:
>>>
>>> I?ve never had an A+B power rail failure at Equinix in 18 years I?ve
>>> been in there facilities, TOUCH WOOD!!!
>>>
>>>
>>>
>>>
>>> *Nathan Brookfield | VK2NAB *General Manager
>>>
>>> *p*: 1300 592 330 | *m*: 0412 266 008
>>> *e*: Nathan.brookfield at iperium.com.au | *w*: iperium.com.au
>>>
>>> Suite 4.02, 189 Kent Street Sydney NSW 2000
>>>
>>> <image001.png>
>>>
>>>
>>> *Your Telco Team*
>>>
>>> DISCLAIMER: This document is intended solely for the named addressee.
>>> This electronic communication, which includes any files or attachments
>>> thereto, contains proprietary or confidential information and may be
>>> privileged and otherwise protected under copyright or other applicable
>>> intellectual property laws. The use, disclosure, copying or distribution of
>>> any of the information contained in this document, by any person other than
>>> the addressee, is strictly prohibited. If you received this document in
>>> error, please contact the sender immediately and delete all the material
>>> from any computer. Confidentiality and legal privilege are not waived or
>>> lost by reason of mistaken delivery to you. Any views or opinions presented
>>> are solely those of the author and do not necessarily represent those of
>>> Iperium.
>>>
>>> WARNING: Computer viruses can be transmitted via email. The recipient
>>> should check this email and any attachments for the presence of viruses.
>>> Iperium accepts no liability for any damage caused by any virus transmitted
>>> by this email.
>>>
>>> *From:* AusNOG <ausnog-bounces at lists.ausnog.net>
>>> <ausnog-bounces at lists.ausnog.net> *On Behalf Of *Matt Perkins
>>> *Sent:* Tuesday, 14 March 2023 11:18 AM
>>> *To:* ausnog at lists.ausnog.net
>>> *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>>>
>>>
>>>
>>> Yea because Eqinix has never had a power outage. :)
>>>
>>> I think it's a little premature to judge at the moment. Doesn't sound
>>> good based on the current info but we have all had bad days im sure.
>>>
>>> Matt
>>>
>>>
>>>
>>> On 14/3/2023 11:06 am, Bradley Amm wrote:
>>>
>>> Friends like let friends put gear in a Vocus dtacenter. Only NextDC or
>>> Equinix :)
>>>
>>> Sent from my iPhone
>>>
>>>
>>>
>>> On 14 Mar 2023, at 7:08 am, DaZZa <dazzagibbs at gmail.com>
>>> <dazzagibbs at gmail.com> wrote:
>>>
>>> ?
>>>
>>> Doesn't say a whole lot about Vocus' data center design methodology, nor
>>> their periodic contingency testing.
>>>
>>>
>>>
>>> They're apparently replacing the entire UPS on Friday - guess we can all
>>> expect more outages then when they stuff that up.
>>>
>>>
>>>
>>> D
>>>
>>>
>>>
>>> On Mon, 13 Mar 2023 at 18:30, Chad Kelly <chad at cpkws.com.au> wrote:
>>>
>>> Thought I would create a new thread on this, as the issue wasn?t related
>>> to AussieBB.
>>>
>>> It is looking like this Melbourne outage is similar to the Sydney outage
>>> back in 2018, in that it appears to have been yet again caused by an issue
>>> with their UPS systems yet again.
>>>
>>> I?ll leave this hear
>>> https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/
>>> and this article on today?s outage as well.
>>>
>>>
>>> https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987
>>>
>>>
>>>
>>> Regards Chad.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Chad Kelly
>>>
>>> Manager
>>>
>>> CPK Web Services
>>>
>>> Phone 03 52730246
>>>
>>> Web https://www.cpkws.com.au
>>>
>>> <image001.jpg>
>>>
>>>
>>>
>>> _______________________________________________
>>> AusNOG mailing list
>>> AusNOG at lists.ausnog.net
>>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>>
>>>
>>>
>>> --
>>>
>>> veg?e?tar?i?an:
>>> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>>>
>>> _______________________________________________
>>> AusNOG mailing list
>>> AusNOG at lists.ausnog.net
>>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>>
>>>
>>>
>>> _______________________________________________
>>>
>>> AusNOG mailing list
>>>
>>> AusNOG at lists.ausnog.net
>>>
>>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>>
>>> --
>>>
>>> ** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL
>>>
>>>
>>>
>>> /* Matt Perkins
>>>
>>>         Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.
>>>
>>>         Office 1300 133 299     matt at spectrum.com.au
>>>
>>>         ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000
>>>
>>>
>>>
>>> */
>>>
>>> --
>>>
>>> ** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL
>>>
>>>
>>>
>>> /* Matt Perkins
>>>
>>>         Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.
>>>
>>>         Office 1300 133 299     matt at spectrum.com.au
>>>
>>>         ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000
>>>
>>>
>>>
>>> */
>>>
>>>
>>> _______________________________________________
>>> AusNOG mailing list
>>> AusNOG at lists.ausnog.net
>>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>>
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
>
>
> --
> veg?e?tar?i?an:
> Ancient tribal slang for the village idiot who can't hunt, fish or ride
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230318/d64a0021/attachment.htm>

From dave.taht at gmail.com  Sun Mar 19 02:15:25 2023
From: dave.taht at gmail.com (Dave Taht)
Date: Sat, 18 Mar 2023 08:15:25 -0700
Subject: [AusNOG] fq_codel/cake/libreqos beta testers wanted
In-Reply-To: <035f01d95675$b6060330$22120990$@jdmnet.com.au>
References: <CAA93jw7Kd=7BCfCdk3R3uXmtRJxFfLSjBAZW3T-mGF=gEtu+_A@mail.gmail.com>
 <035f01d95675$b6060330$22120990$@jdmnet.com.au>
Message-ID: <CAA93jw7ZFgX+eTzstq+xabccrdB+R8ipOugvVR+4ty6mPB=zbQ@mail.gmail.com>

It was very nice to see several of you join the #libreqos:matrix.org chatroom!

I still have big, researchy questions about what the Net looks like
there. I don't know what the fiber rollout's state is, how much
fq_codel deployed in radios, how much mikrotik/cambium/ubnt wireless
tech is there vs a vs juniper (does anyone actually configure RED on
their juniper/etc boxes? Can you share packet caps of a test like
this: https://blog.cerowrt.org/post/juniper/ )

I am very interested in optimizing transports and fq_codel/cake for
wildly varying RTTs and bandwidths. Right now I am looking at the
hardest problem there is, Africa, which is not running much faster
that 10/6 mbit even in the best cases, and 90% of traffic exits the
continent. How much Australian traffic exits the continent? (and if
there are good resources on this please let me know? A BGP AS number
to typical RTT tool?)

From the.damo at gmail.com  Mon Mar 20 15:33:26 2023
From: the.damo at gmail.com (Damian Guppy)
Date: Mon, 20 Mar 2023 12:33:26 +0800
Subject: [AusNOG] Vocus Datacentre outage.
In-Reply-To: <076eb1f9595849319ad53e27dab953ed@zonenetworks.com.au>
References: <CAMpV8Mpv98O-cqpGz+5SVRBsobsZzWAT0Rhy8UdiSQ6fUEiWjw@mail.gmail.com>
 <A7475E5D-495B-4B4B-91C1-169140FEACE1@bradleyamm.com>
 <18719e27-053c-2ec2-f48e-2b69d5114716@spectrum.com.au>
 <MEYP282MB2520B620B1438273E3A1009CDBBE9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <f35b78ea-6e27-1eb1-5314-20fde831056a@spectrum.com.au>
 <MEYP282MB2520A3C7BCA133967365ED77DBBE9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <SYBP282MB0538A5F3C118B6D6A4ED85F5AFBE9@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM>
 <8D195CA1-4E67-46F4-BF39-C9AED0616977@iperium.com.au>
 <076eb1f9595849319ad53e27dab953ed@zonenetworks.com.au>
Message-ID: <CAONJJ+SgrNPT9FEYG1orBLhhVAYDUADm+NDHJJc4m+bDuuTwUQ@mail.gmail.com>

I know I'm a bit late here, but surprised to see no one mention 1 William
in Perth. Seems to be a pattern with Vocus and A+B outages...

--Damian

On Tue, Mar 14, 2023 at 9:30?AM Joel Nath <joel at zonenetworks.com.au> wrote:

> In EQX defence Sy1 didn?t go dark? it was multiple (3-5) rows of rack that
> lost power in 2009.
>
>
>
> We were one of the customers heavily impacted in 2009 but understood the
> reason why it happened and EQX fixed it all up.
>
>
>
> Vocus Doody st different story and now melb
>
>
>
> We pulled out of Doody after the drama we faced there.
>
>
>
> Joel.
>
>
>
> *From:* AusNOG <ausnog-bounces at lists.ausnog.net> *On Behalf Of *Nathan
> Brookfield
> *Sent:* Tuesday, 14 March 2023 11:54 AM
> *To:* Christopher Hawker <chris at thesysadmin.dev>
> *Cc:* ausnog at lists.ausnog.net
> *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>
>
>
> I knew I didn?t remember it for a reason ?
>
>
>
> *[AusNOG] Equinix Outage
> <https://lists.ausnog.net/pipermail/ausnog/2009-July/003239.html>*
>
> *lists.ausnog.net
> <https://lists.ausnog.net/pipermail/ausnog/2009-July/003239.html>*
>
> <https://lists.ausnog.net/pipermail/ausnog/2009-July/003239.html>
>
>
>
> On 14 Mar 2023, at 10:50, Christopher Hawker <chris at thesysadmin.dev>
> wrote:
>
>
>
> ?
>
> SY1 apparently experienced a full power outage back in July 2009,
> according to AusNOG Archives. Some reported full outages, some partial. Was
> not a DC fault though, looks like a much larger grid issue where a fault at
> the Bayswater Power Station in the Hunter Valley (which affected generators
> in QLD and VIC) caused about 1000MW to be dropped from the grid, resulting
> in rolling blackouts across QLD, NSW, TAS, SA and VIC.
>
>
>
> https://lists.ausnog.net/pipermail/ausnog/2009-July/003222.html
>
>
>
> CH.
> ------------------------------
>
> *From:* AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Nathan
> Brookfield <Nathan.Brookfield at iperium.com.au>
> *Sent:* Tuesday, March 14, 2023 11:40 AM
> *To:* Matt Perkins <matt at spectrum.com.au>; ausnog at lists.ausnog.net <
> ausnog at lists.ausnog.net>
> *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>
>
>
> Have 100% had A and B fail separately at SY1 but never both, thankfully!!!
>
>
>
> Could have been part of the facility though so don?t quote me.
>
>
>
> UK on the other hand, fail!
>
>
>
> *From:* Matt Perkins <matt at spectrum.com.au>
> *Sent:* Tuesday, 14 March 2023 11:30 AM
> *To:* Nathan Brookfield <Nathan.Brookfield at iperium.com.au>;
> ausnog at lists.ausnog.net
> *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>
>
>
> SY1 had A or B fail a few years back if my memory serves me correctly.
> But I was referring  London and Manchester which both had total outages in
> recent years.
>
> Matt.
>
>
>
> On 14/3/2023 11:26 am, Nathan Brookfield wrote:
>
> I?ve never had an A+B power rail failure at Equinix in 18 years I?ve been
> in there facilities, TOUCH WOOD!!!
>
>
>
>
> *Nathan Brookfield | VK2NAB *General Manager
>
> *p*: 1300 592 330 | *m*: 0412 266 008
> *e*: Nathan.brookfield at iperium.com.au | *w*: iperium.com.au
>
> Suite 4.02, 189 Kent Street Sydney NSW 2000
>
> <image001.png>
>
>
> *Your Telco Team*
>
> DISCLAIMER: This document is intended solely for the named addressee. This
> electronic communication, which includes any files or attachments thereto,
> contains proprietary or confidential information and may be privileged and
> otherwise protected under copyright or other applicable intellectual
> property laws. The use, disclosure, copying or distribution of any of the
> information contained in this document, by any person other than the
> addressee, is strictly prohibited. If you received this document in error,
> please contact the sender immediately and delete all the material from any
> computer. Confidentiality and legal privilege are not waived or lost by
> reason of mistaken delivery to you. Any views or opinions presented are
> solely those of the author and do not necessarily represent those of
> Iperium.
>
> WARNING: Computer viruses can be transmitted via email. The recipient
> should check this email and any attachments for the presence of viruses.
> Iperium accepts no liability for any damage caused by any virus transmitted
> by this email.
>
> *From:* AusNOG <ausnog-bounces at lists.ausnog.net>
> <ausnog-bounces at lists.ausnog.net> *On Behalf Of *Matt Perkins
> *Sent:* Tuesday, 14 March 2023 11:18 AM
> *To:* ausnog at lists.ausnog.net
> *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>
>
>
> Yea because Eqinix has never had a power outage. :)
>
> I think it's a little premature to judge at the moment. Doesn't sound good
> based on the current info but we have all had bad days im sure.
>
> Matt
>
>
>
> On 14/3/2023 11:06 am, Bradley Amm wrote:
>
> Friends like let friends put gear in a Vocus dtacenter. Only NextDC or
> Equinix :)
>
> Sent from my iPhone
>
>
>
> On 14 Mar 2023, at 7:08 am, DaZZa <dazzagibbs at gmail.com>
> <dazzagibbs at gmail.com> wrote:
>
> ?
>
> Doesn't say a whole lot about Vocus' data center design methodology, nor
> their periodic contingency testing.
>
>
>
> They're apparently replacing the entire UPS on Friday - guess we can all
> expect more outages then when they stuff that up.
>
>
>
> D
>
>
>
> On Mon, 13 Mar 2023 at 18:30, Chad Kelly <chad at cpkws.com.au> wrote:
>
> Thought I would create a new thread on this, as the issue wasn?t related
> to AussieBB.
>
> It is looking like this Melbourne outage is similar to the Sydney outage
> back in 2018, in that it appears to have been yet again caused by an issue
> with their UPS systems yet again.
>
> I?ll leave this hear
> https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/
> and this article on today?s outage as well.
>
>
> https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987
>
>
>
> Regards Chad.
>
>
>
>
>
>
>
> Chad Kelly
>
> Manager
>
> CPK Web Services
>
> Phone 03 52730246
>
> Web https://www.cpkws.com.au
>
> <image001.jpg>
>
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
> --
>
> veg?e?tar?i?an:
> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
> _______________________________________________
>
> AusNOG mailing list
>
> AusNOG at lists.ausnog.net
>
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
> --
>
> ** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL
>
>
>
> /* Matt Perkins
>
>         Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.
>
>         Office 1300 133 299     matt at spectrum.com.au
>
>         ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000
>
>
>
> */
>
> --
>
> ** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL
>
>
>
> /* Matt Perkins
>
>         Direct 02 8916 8101     Spectrum Networks Ptd. Ltd.
>
>         Office 1300 133 299     matt at spectrum.com.au
>
>         ABN 66 090 112 913      Level 6, 350 George Street Sydney 2000
>
>
>
> */
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230320/7da30d4b/attachment.htm>

From ben_colt at yahoo.com  Mon Mar 20 15:37:41 2023
From: ben_colt at yahoo.com (Ben Colt)
Date: Mon, 20 Mar 2023 04:37:41 +0000 (UTC)
Subject: [AusNOG] =?utf-8?q?Inligo=E2=80=99s_unite_cable_system?=
References: <1468057186.1583238.1679287061659.ref@mail.yahoo.com>
Message-ID: <1468057186.1583238.1679287061659@mail.yahoo.com>

Been following this recently.?Works have commenced in Victoria and NT but?keen to understand dates and if anyone has received pricing for 100g services please contact me off list.?
ThanksBen

Sent from Yahoo Mail for iPhone
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230320/7685b847/attachment.htm>

From andrew at levart.com.au  Tue Mar 21 14:33:25 2023
From: andrew at levart.com.au (Andrew Simmonds)
Date: Tue, 21 Mar 2023 11:33:25 +0800
Subject: [AusNOG] =?utf-8?q?Inligo=E2=80=99s_unite_cable_system?=
In-Reply-To: <1468057186.1583238.1679287061659@mail.yahoo.com>
References: <1468057186.1583238.1679287061659.ref@mail.yahoo.com>
 <1468057186.1583238.1679287061659@mail.yahoo.com>
Message-ID: <2e8d418d-782e-30ff-868d-8401d1a5171a@levart.com.au>

Also add Vocus VDC-PER09 at Murchison Tce, East Perth back in Nov 2014 
into the mix. (May have been Acure/Amcom DC2 back then)

They were performing generator testing at the time.

I was pretty happy when we moved out of there.

<https://www.levart.com.au/>
Andrew Simmonds
Systems Development Manager
*Levart*
1A/18 Gibberd Road
Balcatta WA 6021 <#> 	p:
m:
f:
e:
w: 	1300 538 278 <tel:1300538278>
+61 401 218 463 <tel:+61401218463>
+61 8 9382 8003 <tel:+61893828003>
andrew at levart.com.au
www.levart.com.au <https://www.levart.com.au/>

On 20/03/2023 12:37 pm, Ben Colt wrote:
> Been following this recently.?Works have commenced in Victoria and NT 
> but?keen to understand dates and if anyone has received pricing for 
> 100g services please contact me off list.
>
> Thanks
> Ben
>
> Sent from Yahoo Mail for iPhone 
> <https://overview.mail.yahoo.com/?.src=iOS>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230321/824d5864/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: levart-small.png
Type: image/png
Size: 4936 bytes
Desc: not available
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230321/824d5864/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4228 bytes
Desc: S/MIME Cryptographic Signature
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230321/824d5864/attachment-0001.p7s>

From joseph at goldman.id.au  Wed Mar 22 10:54:35 2023
From: joseph at goldman.id.au (Joseph Goldman)
Date: Tue, 21 Mar 2023 23:54:35 +0000
Subject: [AusNOG] E-waste near SY3
Message-ID: <embcc646cc-4cdc-438f-8508-d122e93f35be@joe-pc>

G'day,

  Doing a good cleanup this weekend of some old servers and switches (old 
enough not worth re-purposing or selling) in Equinix SY3 - any good 
e-waste recyclers close by we can drop them off at once removed?

Thanks,
Joe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230321/71a28947/attachment.htm>

From timc at slowb.ro  Thu Mar 23 19:44:26 2023
From: timc at slowb.ro (Tim C)
Date: Thu, 23 Mar 2023 19:44:26 +1100
Subject: [AusNOG] E-waste near SY3
In-Reply-To: <embcc646cc-4cdc-438f-8508-d122e93f35be@joe-pc>
References: <embcc646cc-4cdc-438f-8508-d122e93f35be@joe-pc>
Message-ID: <1f1a7df4-e172-b3d4-ef20-d24d14cb27aa@slowb.ro>

On 22/3/23 10:54, Joseph Goldman wrote:
> G'day,
> 
>  ?Doing a good cleanup this weekend of some old servers and switches 
> (old enough not worth re-purposing or selling) in Equinix SY3 - any good 
> e-waste recyclers close by we can drop them off at once removed?
> 
> Thanks,
> Joe
> 
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog

Depending how much, your local officeworks may have ewaste facilities. 
https://www.officeworks.com.au/information/about-us/peopleandplanet/recycling

From mike at pineview.net  Thu Mar 23 20:05:45 2023
From: mike at pineview.net (Mike O'Connor)
Date: Thu, 23 Mar 2023 19:35:45 +1030
Subject: [AusNOG] E-waste near SY3
In-Reply-To: <embcc646cc-4cdc-438f-8508-d122e93f35be@joe-pc>
References: <embcc646cc-4cdc-438f-8508-d122e93f35be@joe-pc>
Message-ID: <2725e04d-8e53-be43-af0e-bb4680289881@pineview.net>

Hi,

Bunning's stores take e-waste, but the sort of e-waste your talking 
about might raise eye brows.

Cheers
Mike

On 22/3/2023 10:24 am, Joseph Goldman wrote:
> G'day,
>
> ?Doing a good cleanup this weekend of some old servers and switches 
> (old enough not worth re-purposing or selling) in Equinix SY3 - any 
> good e-waste recyclers close by we can drop them off at once removed?
>
> Thanks,
> Joe
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230323/4cdbaf41/attachment.htm>

From dazzagibbs at gmail.com  Fri Mar 24 13:30:47 2023
From: dazzagibbs at gmail.com (DaZZa)
Date: Fri, 24 Mar 2023 13:30:47 +1100
Subject: [AusNOG] Whups, they did it again..
Message-ID: <CAMpV8Mrh5XWy4OxEquoCFAfMf9bxWhFHpKVuz+-436ZnQNji=Q@mail.gmail.com>

I've just lost my services with Vocus in Victoria again

Is this a case of "Whups, they did it again", or am I just incredibly
unlucky and they've killed all three of my services at once?

Anyone else seeing issues the same as last time they lost the UPS/Power?

D

From m.enger at xi.com.au  Fri Mar 24 13:33:28 2023
From: m.enger at xi.com.au (Matthew Enger)
Date: Fri, 24 Mar 2023 02:33:28 +0000
Subject: [AusNOG] Whups, they did it again..
In-Reply-To: <CAMpV8Mrh5XWy4OxEquoCFAfMf9bxWhFHpKVuz+-436ZnQNji=Q@mail.gmail.com>
References: <CAMpV8Mrh5XWy4OxEquoCFAfMf9bxWhFHpKVuz+-436ZnQNji=Q@mail.gmail.com>
Message-ID: <SY4PR01MB70575640F23F33448484D1C7A6849@SY4PR01MB7057.ausprd01.prod.outlook.com>

I did see the same thing but they have now come back.




[signature_4013076670]
MATTHEW ENGER | Managing Director
T 1300 789 299 D 03 9909 3104 M 0406 532 792
m.enger at xi.com.au<mailto:m.enger at xi.com.au> | www.xi.com.au<http://www.xi.com.au> | Book an appointment<https://calendly.com/m-enger/30min>

This message is confidential. It may also be privileged or otherwise protected by work product immunity or other legal rules. If you have received it by mistake, please let us know by e-mail reply and delete it from your system; you may not copy this message or disclose its contents to anyone. Please send us by fax any message containing deadlines as incoming e-mails are not screened for response deadlines. The integrity and security of this message cannot be guaranteed on the Internet.


From: AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of DaZZa <dazzagibbs at gmail.com>
Date: Friday, 24 March 2023 at 3:32 pm
To: <ausnog at lists.ausnog.net>
Subject: [AusNOG] Whups, they did it again..
I've just lost my services with Vocus in Victoria again

Is this a case of "Whups, they did it again", or am I just incredibly
unlucky and they've killed all three of my services at once?

Anyone else seeing issues the same as last time they lost the UPS/Power?

D
_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog
--
Message  protected by MailGuard: e-mail anti-virus, anti-spam and content filtering.https://www.mailguard.com.au/mg
Click here to report this message as spam:
https://console.mailguard.com.au/ras/253QJadKYi/5BdLBQX1lsih29XWlFuHK2/0.5
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230324/20a355b1/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 7645 bytes
Desc: image001.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230324/20a355b1/attachment.png>

From philip.loenneker at tasmanet.com.au  Fri Mar 24 13:52:12 2023
From: philip.loenneker at tasmanet.com.au (Philip Loenneker)
Date: Fri, 24 Mar 2023 02:52:12 +0000
Subject: [AusNOG] Whups, they did it again..
In-Reply-To: <SY4PR01MB70575640F23F33448484D1C7A6849@SY4PR01MB7057.ausprd01.prod.outlook.com>
References: <CAMpV8Mrh5XWy4OxEquoCFAfMf9bxWhFHpKVuz+-436ZnQNji=Q@mail.gmail.com>
 <SY4PR01MB70575640F23F33448484D1C7A6849@SY4PR01MB7057.ausprd01.prod.outlook.com>
Message-ID: <ME1PR01MB16341F48D33E3DDCFFF4E0BCBA849@ME1PR01MB1634.ausprd01.prod.outlook.com>

It must have been more localised than last time. I can't even see that a single power rail went down for where our gear is in DH1.


From: AusNOG <ausnog-bounces at lists.ausnog.net> On Behalf Of Matthew Enger
Sent: Friday, 24 March 2023 1:33 PM
To: DaZZa <dazzagibbs at gmail.com>; <ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] Whups, they did it again..

You don't often get email from m.enger at xi.com.au. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification>
I did see the same thing but they have now come back.




[signature_4013076670]
MATTHEW ENGER | Managing Director
T 1300 789 299 D 03 9909 3104 M 0406 532 792
m.enger at xi.com.au<mailto:m.enger at xi.com.au> | www.xi.com.au<http://www.xi.com.au> | Book an appointment<https://calendly.com/m-enger/30min>

This message is confidential. It may also be privileged or otherwise protected by work product immunity or other legal rules. If you have received it by mistake, please let us know by e-mail reply and delete it from your system; you may not copy this message or disclose its contents to anyone. Please send us by fax any message containing deadlines as incoming e-mails are not screened for response deadlines. The integrity and security of this message cannot be guaranteed on the Internet.


From: AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of DaZZa <dazzagibbs at gmail.com>
Date: Friday, 24 March 2023 at 3:32 pm
To: <ausnog at lists.ausnog.net>
Subject: [AusNOG] Whups, they did it again..
I've just lost my services with Vocus in Victoria again

Is this a case of "Whups, they did it again", or am I just incredibly
unlucky and they've killed all three of my services at once?

Anyone else seeing issues the same as last time they lost the UPS/Power?

D
_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog
--
Message  protected by MailGuard: e-mail anti-virus, anti-spam and content filtering.https://www.mailguard.com.au/mg
Click here to report this message as spam:
https://console.mailguard.com.au/ras/253QJadKYi/5BdLBQX1lsih29XWlFuHK2/0.5
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230324/8f770dac/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 7645 bytes
Desc: image001.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230324/8f770dac/attachment.png>

From dazzagibbs at gmail.com  Fri Mar 24 14:10:16 2023
From: dazzagibbs at gmail.com (DaZZa)
Date: Fri, 24 Mar 2023 14:10:16 +1100
Subject: [AusNOG] Whups, they did it again..
In-Reply-To: <ME1PR01MB16341F48D33E3DDCFFF4E0BCBA849@ME1PR01MB1634.ausprd01.prod.outlook.com>
References: <CAMpV8Mrh5XWy4OxEquoCFAfMf9bxWhFHpKVuz+-436ZnQNji=Q@mail.gmail.com>
 <SY4PR01MB70575640F23F33448484D1C7A6849@SY4PR01MB7057.ausprd01.prod.outlook.com>
 <ME1PR01MB16341F48D33E3DDCFFF4E0BCBA849@ME1PR01MB1634.ausprd01.prod.outlook.com>
Message-ID: <CAMpV8MqMLUG2T8CPzUtgAp=H-m03BkWLgejhAN-v=nzcfjuOOA@mail.gmail.com>

Guy I spoke to claimed it was an issue with "some of their switches" before
clamming up.

Best guess? Someone made an unsanctioned/untested change and screwed
something up.

D

On Fri, 24 Mar 2023 at 13:52, Philip Loenneker <
philip.loenneker at tasmanet.com.au> wrote:

> It must have been more localised than last time. I can?t even see that a
> single power rail went down for where our gear is in DH1.
>
>
>
>
>
> *From:* AusNOG <ausnog-bounces at lists.ausnog.net> *On Behalf Of *Matthew
> Enger
> *Sent:* Friday, 24 March 2023 1:33 PM
> *To:* DaZZa <dazzagibbs at gmail.com>; <ausnog at lists.ausnog.net> <
> ausnog at lists.ausnog.net>
> *Subject:* Re: [AusNOG] Whups, they did it again..
>
>
>
> You don't often get email from m.enger at xi.com.au. Learn why this is
> important <https://aka.ms/LearnAboutSenderIdentification>
>
> I did see the same thing but they have now come back.
>
>
>
>
>
>
>
>
>
> [image: signature_4013076670]
>
> *MATTHEW ENGER* | Managing Director
> *T *1300 789 299 *D *03 9909 3104 *M *0406 532 792
> m.enger at xi.com.au | www.xi.com.au | Book an appointment
> <https://calendly.com/m-enger/30min>
>
>
>
> This message is confidential. It may also be privileged or otherwise
> protected by work product immunity or other legal rules. If you have
> received it by mistake, please let us know by e-mail reply and delete it
> from your system; you may not copy this message or disclose its contents to
> anyone. Please send us by fax any message containing deadlines as incoming
> e-mails are not screened for response deadlines. The integrity and security
> of this message cannot be guaranteed on the Internet.
>
>
>
>
>
> *From: *AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of DaZZa <
> dazzagibbs at gmail.com>
> *Date: *Friday, 24 March 2023 at 3:32 pm
> *To: *<ausnog at lists.ausnog.net>
> *Subject: *[AusNOG] Whups, they did it again..
>
> I've just lost my services with Vocus in Victoria again
>
> Is this a case of "Whups, they did it again", or am I just incredibly
> unlucky and they've killed all three of my services at once?
>
> Anyone else seeing issues the same as last time they lost the UPS/Power?
>
> D
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
> --
> Message  protected by MailGuard: e-mail anti-virus, anti-spam and content
> filtering.https://www.mailguard.com.au/mg
> Click here to report this message as spam:
> https://console.mailguard.com.au/ras/253QJadKYi/5BdLBQX1lsih29XWlFuHK2/0.5
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>


-- 
veg?e?tar?i?an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230324/8c8ef430/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 7645 bytes
Desc: not available
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230324/8c8ef430/attachment.png>

From james.morgan at vernet.com.au  Fri Mar 24 14:17:29 2023
From: james.morgan at vernet.com.au (James Morgan)
Date: Fri, 24 Mar 2023 03:17:29 +0000
Subject: [AusNOG] Whups, they did it again..
In-Reply-To: <CAMpV8MqMLUG2T8CPzUtgAp=H-m03BkWLgejhAN-v=nzcfjuOOA@mail.gmail.com>
References: <CAMpV8Mrh5XWy4OxEquoCFAfMf9bxWhFHpKVuz+-436ZnQNji=Q@mail.gmail.com>
 <SY4PR01MB70575640F23F33448484D1C7A6849@SY4PR01MB7057.ausprd01.prod.outlook.com>
 <ME1PR01MB16341F48D33E3DDCFFF4E0BCBA849@ME1PR01MB1634.ausprd01.prod.outlook.com>
 <CAMpV8MqMLUG2T8CPzUtgAp=H-m03BkWLgejhAN-v=nzcfjuOOA@mail.gmail.com>
Message-ID: <PS2PR02MB2983B571263DDD52B823AF75D7849@PS2PR02MB2983.apcprd02.prod.outlook.com>

Yeah, we lost a bunch of Vocus P2P L2 circuits for around 10-15 mins (we aren?t in their DCs).

Very nice that they keep the interfaces up on each end for you though?

JM

From: AusNOG <ausnog-bounces at lists.ausnog.net> On Behalf Of DaZZa
Sent: Friday, 24 March 2023 2:10 PM
To: Philip Loenneker <philip.loenneker at tasmanet.com.au>
Cc: <ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] Whups, they did it again..

Guy I spoke to claimed it was an issue with "some of their switches" before clamming up.

Best guess? Someone made an unsanctioned/untested change and screwed something up.

D

On Fri, 24 Mar 2023 at 13:52, Philip Loenneker <philip.loenneker at tasmanet.com.au<mailto:philip.loenneker at tasmanet.com.au>> wrote:
It must have been more localised than last time. I can?t even see that a single power rail went down for where our gear is in DH1.


From: AusNOG <ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>> On Behalf Of Matthew Enger
Sent: Friday, 24 March 2023 1:33 PM
To: DaZZa <dazzagibbs at gmail.com<mailto:dazzagibbs at gmail.com>>; <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>> <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: Re: [AusNOG] Whups, they did it again..

You don't often get email from m.enger at xi.com.au<mailto:m.enger at xi.com.au>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification>
I did see the same thing but they have now come back.




[signature_4013076670]
MATTHEW ENGER | Managing Director
T 1300 789 299 D 03 9909 3104 M 0406 532 792
m.enger at xi.com.au<mailto:m.enger at xi.com.au> | www.xi.com.au<http://www.xi.com.au> | Book an appointment<https://calendly.com/m-enger/30min>

This message is confidential. It may also be privileged or otherwise protected by work product immunity or other legal rules. If you have received it by mistake, please let us know by e-mail reply and delete it from your system; you may not copy this message or disclose its contents to anyone. Please send us by fax any message containing deadlines as incoming e-mails are not screened for response deadlines. The integrity and security of this message cannot be guaranteed on the Internet.


From: AusNOG <ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>> on behalf of DaZZa <dazzagibbs at gmail.com<mailto:dazzagibbs at gmail.com>>
Date: Friday, 24 March 2023 at 3:32 pm
To: <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: [AusNOG] Whups, they did it again..
I've just lost my services with Vocus in Victoria again

Is this a case of "Whups, they did it again", or am I just incredibly
unlucky and they've killed all three of my services at once?

Anyone else seeing issues the same as last time they lost the UPS/Power?

D
_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
https://lists.ausnog.net/mailman/listinfo/ausnog
--
Message  protected by MailGuard: e-mail anti-virus, anti-spam and content filtering.https://www.mailguard.com.au/mg
Click here to report this message as spam:
https://console.mailguard.com.au/ras/253QJadKYi/5BdLBQX1lsih29XWlFuHK2/0.5
_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
https://lists.ausnog.net/mailman/listinfo/ausnog


--
veg?e?tar?i?an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
This email arrived from outside the organization. If the sender claims to be an employee this is likely to be fraudulent.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230324/31ee5ff5/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 7645 bytes
Desc: image001.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230324/31ee5ff5/attachment-0001.png>

From mcdonald.richards at gmail.com  Fri Mar 24 15:04:00 2023
From: mcdonald.richards at gmail.com (McDonald Richards)
Date: Fri, 24 Mar 2023 15:04:00 +1100
Subject: [AusNOG] Whups, they did it again..
In-Reply-To: <PS2PR02MB2983B571263DDD52B823AF75D7849@PS2PR02MB2983.apcprd02.prod.outlook.com>
References: <CAMpV8Mrh5XWy4OxEquoCFAfMf9bxWhFHpKVuz+-436ZnQNji=Q@mail.gmail.com>
 <SY4PR01MB70575640F23F33448484D1C7A6849@SY4PR01MB7057.ausprd01.prod.outlook.com>
 <ME1PR01MB16341F48D33E3DDCFFF4E0BCBA849@ME1PR01MB1634.ausprd01.prod.outlook.com>
 <CAMpV8MqMLUG2T8CPzUtgAp=H-m03BkWLgejhAN-v=nzcfjuOOA@mail.gmail.com>
 <PS2PR02MB2983B571263DDD52B823AF75D7849@PS2PR02MB2983.apcprd02.prod.outlook.com>
Message-ID: <CACrsH-460EBy6twZnDe-gXh8VgFPWfz+mriFo4zQXB5t6X97fg@mail.gmail.com>

It's almost like BFD usage over carrier ethernet is a new thing...

On Fri, Mar 24, 2023 at 2:17?PM James Morgan <james.morgan at vernet.com.au>
wrote:

> Yeah, we lost a bunch of Vocus P2P L2 circuits for around 10-15 mins (we
> aren?t in their DCs).
>
>
>
> Very nice that they keep the interfaces up on each end for you though?
>
>
>
> JM
>
>
>
> *From:* AusNOG <ausnog-bounces at lists.ausnog.net> *On Behalf Of *DaZZa
> *Sent:* Friday, 24 March 2023 2:10 PM
> *To:* Philip Loenneker <philip.loenneker at tasmanet.com.au>
> *Cc:* <ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
> *Subject:* Re: [AusNOG] Whups, they did it again..
>
>
>
> Guy I spoke to claimed it was an issue with "some of their switches"
> before clamming up.
>
>
>
> Best guess? Someone made an unsanctioned/untested change and screwed
> something up.
>
>
>
> D
>
>
>
> On Fri, 24 Mar 2023 at 13:52, Philip Loenneker <
> philip.loenneker at tasmanet.com.au> wrote:
>
> It must have been more localised than last time. I can?t even see that a
> single power rail went down for where our gear is in DH1.
>
>
>
>
>
> *From:* AusNOG <ausnog-bounces at lists.ausnog.net> *On Behalf Of *Matthew
> Enger
> *Sent:* Friday, 24 March 2023 1:33 PM
> *To:* DaZZa <dazzagibbs at gmail.com>; <ausnog at lists.ausnog.net> <
> ausnog at lists.ausnog.net>
> *Subject:* Re: [AusNOG] Whups, they did it again..
>
>
>
> You don't often get email from m.enger at xi.com.au. Learn why this is
> important <https://aka.ms/LearnAboutSenderIdentification>
>
> I did see the same thing but they have now come back.
>
>
>
>
>
>
>
>
>
> [image: signature_4013076670]
>
> *MATTHEW ENGER* | Managing Director
> *T *1300 789 299 *D *03 9909 3104 *M *0406 532 792
> m.enger at xi.com.au | www.xi.com.au | Book an appointment
> <https://calendly.com/m-enger/30min>
>
>
>
> This message is confidential. It may also be privileged or otherwise
> protected by work product immunity or other legal rules. If you have
> received it by mistake, please let us know by e-mail reply and delete it
> from your system; you may not copy this message or disclose its contents to
> anyone. Please send us by fax any message containing deadlines as incoming
> e-mails are not screened for response deadlines. The integrity and security
> of this message cannot be guaranteed on the Internet.
>
>
>
>
>
> *From: *AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of DaZZa <
> dazzagibbs at gmail.com>
> *Date: *Friday, 24 March 2023 at 3:32 pm
> *To: *<ausnog at lists.ausnog.net>
> *Subject: *[AusNOG] Whups, they did it again..
>
> I've just lost my services with Vocus in Victoria again
>
> Is this a case of "Whups, they did it again", or am I just incredibly
> unlucky and they've killed all three of my services at once?
>
> Anyone else seeing issues the same as last time they lost the UPS/Power?
>
> D
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
> --
> Message  protected by MailGuard: e-mail anti-virus, anti-spam and content
> filtering.https://www.mailguard.com.au/mg
> Click here to report this message as spam:
> https://console.mailguard.com.au/ras/253QJadKYi/5BdLBQX1lsih29XWlFuHK2/0.5
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
> --
>
> veg?e?tar?i?an:
> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>
> This email arrived from outside the organization. If the sender claims to
> be an employee this is likely to be fraudulent.
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230324/6accd276/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 7645 bytes
Desc: not available
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230324/6accd276/attachment.png>

From shane at 2000cn.com.au  Fri Mar 24 17:04:33 2023
From: shane at 2000cn.com.au (Shane Chrisp)
Date: Fri, 24 Mar 2023 14:04:33 +0800
Subject: [AusNOG] An urgent call to action - Do your part and protect
 APNIC
In-Reply-To: <SYBP282MB05384010EEFF497A603EFFEDAFA39@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM>
References: <PSAPR06MB4839CA44438D8EAD82BB6DE7AAA29@PSAPR06MB4839.apcprd06.prod.outlook.com>
 <MEYP282MB3163D6A4C6615156E414DFEFADA29@MEYP282MB3163.AUSP282.PROD.OUTLOOK.COM>
 <MEYP282MB31638D5834153CE2603BCAEDADA29@MEYP282MB3163.AUSP282.PROD.OUTLOOK.COM>
 <SYBP282MB05384010EEFF497A603EFFEDAFA39@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM>
Message-ID: <181934fc-a334-5631-48d6-ace6cac3fdda@2000cn.com.au>

On 15/2/23 11:26, Christopher Hawker wrote:

Wow, a bit late, but we just received a letter in the post today with a 
letter and a printed colour multi page pamphlet from NRS pushing their 
members for the EC.

cheers
Shane

> E-mailing execsec at apnic.net yields a "554 5.7.1 Recipient address 
> rejected" bounceback. Is it not designed to receive e-mails from 
> external addresses? All SPF and DMARC checks and balances passed.
>
> Was received by ec at apnic.net though.
>
> CH.
> ------------------------------------------------------------------------
> *From:* AusNOG <ausnog-bounces at ausnog.net> on behalf of Nick Pratley 
> <nick at npratley.net>
> *Sent:* Tuesday, February 14, 2023 8:12 PM
> *To:* Karl Kloppenborg <k at rl.ag>; ausnog at lists.ausnog.net 
> <ausnog at lists.ausnog.net>
> *Cc:* nznog at list.waikato.ac.nz <nznog at list.waikato.ac.nz>
> *Subject:* Re: [AusNOG] An urgent call to action - Do your part and 
> protect APNIC
> Wow!
>
> ec at apnic.net.
>
> I blame the iPhone.
>
> Get Outlook for iOS <https://aka.ms/o0ukef>
> ------------------------------------------------------------------------
> *From:* Nick Pratley <nick at npratley.net>
> *Sent:* Tuesday, February 14, 2023 8:10:25 PM
> *To:* Karl Kloppenborg <k at rl.ag>; ausnog at lists.ausnog.net 
> <ausnog at lists.ausnog.net>
> *Cc:* nznog at list.waikato.ac.nz <nznog at list.waikato.ac.nz>
> *Subject:* Re: An urgent call to action - Do your part and protect APNIC
> Hi Karl, et all.
>
> What a fantastic initiative. Personally, I welcome all the suggested 
> changes to the by-laws.
>
> I encourage everyone to read and help save APNIC from this mess.
>
> For those that want to help, there are two things you can do that will 
> make a difference immediately.
>
>  1. Your vote counts! AU/NZ is very under voted in APNIC elections and
>     this is the best way to stop it!
>  2. Send a note to the APNIC EC as per Karl's template. The best way
>     to contact them is by emailing ec at apjic.net, with a cc to
>     execsec at apnic.net.
>
>
> Karl - thank you for starting this.
>
> Warm Regards
> Nick Pratley
> 0448 379 418
> ------------------------------------------------------------------------
> *From:* AusNOG <ausnog-bounces at ausnog.net> on behalf of Karl 
> Kloppenborg <k at rl.ag>
> *Sent:* Tuesday, February 14, 2023 7:13:37 PM
> *To:* ausnog at lists.ausnog.net <ausnog at lists.ausnog.net>
> *Cc:* nznog at list.waikato.ac.nz <nznog at list.waikato.ac.nz>
> *Subject:* [AusNOG] An urgent call to action - Do your part and 
> protect APNIC
>
> Everyone,
>
> As many are aware, EC Elections are upon us, but few are aware of the 
> dangers currently ahead.
>
> Some of the nominees are attempting to board stack and gain control of 
> the EC.
>
> I implore you to read my attached document and do your part by 
> contacting the EC with the provided template.
>
> By-laws require a very large voter base to enact, but it is possible. 
> Australia and NZ have some of the largest voter block in the APNIC and 
> enough to trigger a by-law change.
>
> I cannot stress this enough; the danger is very real ? at the time of 
> sending this email at least five candidates are either directly 
> endorsed by NRS or the company LARUS.
>
> You can review this yourself here: 
> https://2023.apricot.net/elections/nominations/
>
> Do your part, pass this email, send this template to APNIC members you 
> know and let's garnish the votes necessary to protect our internet and 
> our APNIC.
>
> Thank you,
>
> Karl Kloppenborg.
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230324/bcf5a467/attachment.htm>

From daffy at daffy.za.net  Mon Mar 27 10:31:25 2023
From: daffy at daffy.za.net (Kieran Murphy)
Date: Mon, 27 Mar 2023 10:31:25 +1100
Subject: [AusNOG] Urgently seeking some optics
Message-ID: <CAF_yWZJfPCVobo-OjjJu7FLy=3kShcL8fioz+x6V_0FTAuds6A@mail.gmail.com>

Hi Group,

Time to send up the Ausnog batsignal...

We're looking for a pair of Mellanox (or compatible) 100GBASE-ER4
QSFP28 in the Sydney area.
Just to borrow for a few weeks, or buy if you can spare them.

We've got some on order but delivery times are slippin'

Regards,
Kieran

From chris at thesysadmin.dev  Mon Mar 27 11:21:34 2023
From: chris at thesysadmin.dev (Christopher Hawker)
Date: Mon, 27 Mar 2023 00:21:34 +0000
Subject: [AusNOG] Urgently seeking some optics
In-Reply-To: <CAF_yWZJfPCVobo-OjjJu7FLy=3kShcL8fioz+x6V_0FTAuds6A@mail.gmail.com>
References: <CAF_yWZJfPCVobo-OjjJu7FLy=3kShcL8fioz+x6V_0FTAuds6A@mail.gmail.com>
Message-ID: <SYBP282MB05380FFB160ABB3DE8E241DCAF8B9@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM>

Have a look at 4Cabling in Homebush, they usually keep healthy stock of transceivers. Can program as needed on the spot as well.

CH.

Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Kieran Murphy <daffy at daffy.za.net>
Sent: Monday, March 27, 2023 10:31:25 AM
To: ausnog at ausnog.net <AusNOG at lists.ausnog.net>
Subject: [AusNOG] Urgently seeking some optics

Hi Group,

Time to send up the Ausnog batsignal...

We're looking for a pair of Mellanox (or compatible) 100GBASE-ER4
QSFP28 in the Sydney area.
Just to borrow for a few weeks, or buy if you can spare them.

We've got some on order but delivery times are slippin'

Regards,
Kieran
_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230327/a9ced90b/attachment.htm>

From cphillips at aptient.com  Mon Mar 27 12:57:59 2023
From: cphillips at aptient.com (Chris Phillips)
Date: Mon, 27 Mar 2023 12:57:59 +1100
Subject: [AusNOG] Urgently seeking some optics
In-Reply-To: <CAF_yWZJfPCVobo-OjjJu7FLy=3kShcL8fioz+x6V_0FTAuds6A@mail.gmail.com>
References: <CAF_yWZJfPCVobo-OjjJu7FLy=3kShcL8fioz+x6V_0FTAuds6A@mail.gmail.com>
Message-ID: <33bc64b2-bbf3-c805-b96c-22c7f1387b2c@aptient.com>

I have some, but I'm in Melbourne.  Let me know if you cannot find 
something closer.

On 27/03/2023 10:31 am, Kieran Murphy wrote:
> Hi Group,
> 
> Time to send up the Ausnog batsignal...
> 
> We're looking for a pair of Mellanox (or compatible) 100GBASE-ER4
> QSFP28 in the Sydney area.
> Just to borrow for a few weeks, or buy if you can spare them.
> 
> We've got some on order but delivery times are slippin'
> 
> Regards,
> Kieran
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog

-- 
Chris Phillips
Aptient Consulting Group Inc.
+1 (650) 732-8859 - US & Canada
+61 480 107 120 - Australia
Grab a time in my calendar:
https://calendly.com/chrismphillips/

From dave.taht at gmail.com  Thu Mar 30 02:48:53 2023
From: dave.taht at gmail.com (Dave Taht)
Date: Wed, 29 Mar 2023 08:48:53 -0700
Subject: [AusNOG] curious about lessons learned from your fiber rollouts?
Message-ID: <CAA93jw5osqhLG3FxOc6EyHcaTgEi1mS1bdS7HEDVSYyWPkb7rg@mail.gmail.com>

I am doing an AMA friday, in part about the $70B dollar USA NTIA
broadband and BEAD programs, which are largely targetted at improving
rural access to the internet. The target audience is one with which I
am mostly unfamiliar, the directors of the 50 US states administering
these programs.

I am very interested about what y'all have learned about how to roll
out fiber and fixed wireless right, in your country, so far, and what
could be done better, in mine.

Please let me know what you think here, (links to studies would be
great, too)

and/or come heckle!

-- 
AMA March 31: https://www.broadband.io/c/broadband-grant-events/dave-taht
Dave T?ht CEO, TekLibre, LLC

From jaedwards at gmail.com  Thu Mar 30 11:22:05 2023
From: jaedwards at gmail.com (John Edwards)
Date: Thu, 30 Mar 2023 10:52:05 +1030
Subject: [AusNOG] curious about lessons learned from your fiber rollouts?
In-Reply-To: <CAA93jw5osqhLG3FxOc6EyHcaTgEi1mS1bdS7HEDVSYyWPkb7rg@mail.gmail.com>
References: <CAA93jw5osqhLG3FxOc6EyHcaTgEi1mS1bdS7HEDVSYyWPkb7rg@mail.gmail.com>
Message-ID: <CAOSsYkqBftyJiQa42+GAncZku5c-BwRT9seBWcDbjfpsRO74gA@mail.gmail.com>

My tips:

Rural networks are economically tentative at the best of times, they are
low density and the majority of the demographic that they serve don't have
disposable income for tech entertainment. Government funding likes to be
directed to announceables, and nobody announces a maintenance plan. Budget
accordingly.

Driving time for maintenance personnel is going to become the majority of
your operating budget. Build networks to be bulletproof and have no need
for maintenance, and consider that the added expense of a ring topology or
diverse path might still cost less than a single unscheduled callout by a
contractor. For a rural network, redundant systems buy time to attend to
outages on more relaxed timelines.

Fixed wireless
- is all about waterproofing, be it customer rooftops, cabling joints or
especially equipment shelters.
- rigging teams are at the mercy of safety standards, weather and personal
leave, so you can't expect more than 100 days/year of productive work
- a certain type of person wants to climb towers all day, and they may not
be productive if you try and force them into a soft office environment on
the other days of the year

Fibre Networks
- use a conduit that is a size larger than you need. If you have any kind
of success you're going to need a second cable in that conduit at some
point, even if it's just to do a graceful upgrade
- always use a fibre cleaning kit
- with single-fibre optics (ie: BX), put the transceiver that sends at a
lower frequency at the far end. This gives you a chance to spot the
difference between a fibre fault and a power fault

John



On Thu, 30 Mar 2023 at 02:19, Dave Taht <dave.taht at gmail.com> wrote:

> I am doing an AMA friday, in part about the $70B dollar USA NTIA
> broadband and BEAD programs, which are largely targetted at improving
> rural access to the internet. The target audience is one with which I
> am mostly unfamiliar, the directors of the 50 US states administering
> these programs.
>
> I am very interested about what y'all have learned about how to roll
> out fiber and fixed wireless right, in your country, so far, and what
> could be done better, in mine.
>
> Please let me know what you think here, (links to studies would be
> great, too)
>
> and/or come heckle!
>
> --
> AMA March 31: https://www.broadband.io/c/broadband-grant-events/dave-taht
> Dave T?ht CEO, TekLibre, LLC
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/f98fef7a/attachment.htm>

From joe at ausnog.net  Thu Mar 30 12:37:23 2023
From: joe at ausnog.net (Joe Wooller)
Date: Thu, 30 Mar 2023 09:37:23 +0800
Subject: [AusNOG] Call for paper - AusNOG 2023 Conference
Message-ID: <CAFcm0Pf00eCjTdc=qoY+_GtXBT=gJBDrc+rperQGvOv3U4KC7Q@mail.gmail.com>

Hey AusNOG People,

AusNOG 2023 will be held at the Sea World Conference Centre on the Gold
Coast  (Yes this is not a typo, it is not in Sydney or Melbourne this year)
on the 7th & 8th of September, 2023. The Program (was Programme) Committee
is seeking submissions from people wishing to present at the conference.

Please note that AusNOG 2023 is a technical conference so presentations
must be of a technical nature (i.e. no sales or marketing presentations).
Preference will be given to presentations that result in actual operational
outcomes.

We plan on recording presentations for release after the event for those
who are unable to attend, please don?t let that deter you from submitting,
if your presentation can not be published let us know and we will ensure to
not record it.

For further information, details of important dates,  or to make a
submission please see the website at the URL
https://www.ausnog.net/events/ausnog-2023

Lastly a big thank you to the PC committee; Jocelyn (Program Chair),
Belynda, Mark and Phil.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/bce84f85/attachment.htm>

From xrobau at gmail.com  Thu Mar 30 14:54:59 2023
From: xrobau at gmail.com (Rob Thomas)
Date: Thu, 30 Mar 2023 13:54:59 +1000
Subject: [AusNOG] Critical 3CX Windows/Mac hack.
Message-ID: <CAC5zRjOB28=xU_qwGoNqdLq_ruqjQi19E1JG5Ls8fS-4fsznqw@mail.gmail.com>

As no-one's mentioned it here yet, I just thought I'd bring up the
zero-day, in the wild, active RIGHT NOW, trojan 3CX Windows and Mac apps.

If you, or you have clients, running 3CX, make sure they ARE NOT using the
app. If they are, their machines are probably already owned, and all their
stored credentials and session cookies have been leaked.

https://www.bleepingcomputer.com/news/security/hackers-compromise-3cx-desktop-app-in-a-supply-chain-attack/amp/

This is really bad. Sorry 8-(

--Rob
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/b71dd32d/attachment.htm>

From glipschitz at summitinternet.com.au  Thu Mar 30 15:09:45 2023
From: glipschitz at summitinternet.com.au (Greg Lipschitz)
Date: Thu, 30 Mar 2023 04:09:45 +0000
Subject: [AusNOG] Critical 3CX Windows/Mac hack.
In-Reply-To: <CAC5zRjOB28=xU_qwGoNqdLq_ruqjQi19E1JG5Ls8fS-4fsznqw@mail.gmail.com>
References: <CAC5zRjOB28=xU_qwGoNqdLq_ruqjQi19E1JG5Ls8fS-4fsznqw@mail.gmail.com>
Message-ID: <MEYP282MB25976995206A1A35030FC3D4828E9@MEYP282MB2597.AUSP282.PROD.OUTLOOK.COM>

Here is a list of commands (or make a shell script) to stop it phoning home and getting more payload.

# Disable 3CX Unattended-Upgrades Service

systemctl stop unattended-upgrades

# Collect the version of 3CX Desktop Apps on the Server

cd /var/lib/3cxpbx/Instance1/Data/Http/electron
ls -la * > /root/3cx-desktop-versions.log

# Remove the files

rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/osx/*.dmg
rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/osx/*.zip
rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/windows/*.msi
rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/windows/*.nupkg


https://www.3cx.com/community/threads/threat-alerts-from-sentinelone-for-desktop-update-initiated-from-desktop-client.119806/page-5


Sadly, 3CX haven't even acknowledged this yet.
It would seem that their whole CI-CD pipeline has been compromised

Greg.


Greg Lipschitz | Founder & CEO | Summit Internet
glipschitz at summitinternet.com.au
summitinternet.com.au
1300 049 749
Unit 2, 31-39 Norcal Road, Nunawading VIC 3131
Summit Internet
________________________________
From: AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Rob Thomas <xrobau at gmail.com>
Sent: 30 March 2023 14:54
To: <ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
Subject: [AusNOG] Critical 3CX Windows/Mac hack.

As no-one's mentioned it here yet, I just thought I'd bring up the zero-day, in the wild, active RIGHT NOW, trojan 3CX Windows and Mac apps.

If you, or you have clients, running 3CX, make sure they ARE NOT using the app. If they are, their machines are probably already owned, and all their stored credentials and session cookies have been leaked.

https://www.bleepingcomputer.com/news/security/hackers-compromise-3cx-desktop-app-in-a-supply-chain-attack/amp/<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.bleepingcomputer.com%2Fnews%2Fsecurity%2Fhackers-compromise-3cx-desktop-app-in-a-supply-chain-attack%2Famp%2F&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C5134fed0ee3f4dbc894808db30d2a12f%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157453430051909%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UvNTww7E05nvQnaDQ25Qc8XytZFC%2FhIseT3MHYckCNM%3D&reserved=0>

This is really bad. Sorry 8-(

--Rob

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/aaf3bf22/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image587573.png
Type: image/png
Size: 984 bytes
Desc: image587573.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/aaf3bf22/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image244471.png
Type: image/png
Size: 10728 bytes
Desc: image244471.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/aaf3bf22/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image947901.png
Type: image/png
Size: 1930 bytes
Desc: image947901.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/aaf3bf22/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image891048.png
Type: image/png
Size: 3004 bytes
Desc: image891048.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/aaf3bf22/attachment-0007.png>

From chris at thesysadmin.dev  Thu Mar 30 15:31:14 2023
From: chris at thesysadmin.dev (Christopher Hawker)
Date: Thu, 30 Mar 2023 04:31:14 +0000
Subject: [AusNOG] Critical 3CX Windows/Mac hack.
In-Reply-To: <MEYP282MB25976995206A1A35030FC3D4828E9@MEYP282MB2597.AUSP282.PROD.OUTLOOK.COM>
References: <CAC5zRjOB28=xU_qwGoNqdLq_ruqjQi19E1JG5Ls8fS-4fsznqw@mail.gmail.com>
 <MEYP282MB25976995206A1A35030FC3D4828E9@MEYP282MB2597.AUSP282.PROD.OUTLOOK.COM>
Message-ID: <SYBP282MB0538C9BA5014A95976B5496BAF8E9@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM>

It appears their sales team have no info regarding this. Just rang our Senior AM at 3CX and they've advised that they have no information, and that they are referring anyone who calls to their technical teams via support tickets in the 3CX portal.

Not a good look for them.

CH

Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Greg Lipschitz <glipschitz at summitinternet.com.au>
Sent: Thursday, March 30, 2023 3:09:45 PM
To: Rob Thomas <xrobau at gmail.com>; <ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] Critical 3CX Windows/Mac hack.

Here is a list of commands (or make a shell script) to stop it phoning home and getting more payload.

# Disable 3CX Unattended-Upgrades Service

systemctl stop unattended-upgrades

# Collect the version of 3CX Desktop Apps on the Server

cd /var/lib/3cxpbx/Instance1/Data/Http/electron
ls -la * > /root/3cx-desktop-versions.log

# Remove the files

rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/osx/*.dmg
rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/osx/*.zip
rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/windows/*.msi
rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/windows/*.nupkg


https://www.3cx.com/community/threads/threat-alerts-from-sentinelone-for-desktop-update-initiated-from-desktop-client.119806/page-5


Sadly, 3CX haven't even acknowledged this yet.
It would seem that their whole CI-CD pipeline has been compromised

Greg.


Greg Lipschitz?  |      Founder & CEO    |      Summit Internet
glipschitz at summitinternet.com.au<mailto:glipschitz at summitinternet.com.au>
summitinternet.com.au<http://summitinternet.com.au>
1300 049 749<tel:1300%20049%20749>
Unit 2, 31-39 Norcal Road, Nunawading VIC 3131<https://www.google.com/maps?cid=12522583051503623677&_ga=2.149009334.1057584350.1554770858-1081443428.1554770858>
[cid:image587573.png at 9D56C314.E6D7594C]
[cid:image244471.png at C75A89D0.8905FE7F]
[Summit Internet]<http://summitinternet.com.au> [cid:image891048.png at 51ACA080.56DCC416]

________________________________
From: AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Rob Thomas <xrobau at gmail.com>
Sent: 30 March 2023 14:54
To: <ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
Subject: [AusNOG] Critical 3CX Windows/Mac hack.

As no-one's mentioned it here yet, I just thought I'd bring up the zero-day, in the wild, active RIGHT NOW, trojan 3CX Windows and Mac apps.

If you, or you have clients, running 3CX, make sure they ARE NOT using the app. If they are, their machines are probably already owned, and all their stored credentials and session cookies have been leaked.

https://www.bleepingcomputer.com/news/security/hackers-compromise-3cx-desktop-app-in-a-supply-chain-attack/amp/<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.bleepingcomputer.com%2Fnews%2Fsecurity%2Fhackers-compromise-3cx-desktop-app-in-a-supply-chain-attack%2Famp%2F&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C5134fed0ee3f4dbc894808db30d2a12f%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157453430051909%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UvNTww7E05nvQnaDQ25Qc8XytZFC%2FhIseT3MHYckCNM%3D&reserved=0>

This is really bad. Sorry 8-(

--Rob

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/209a49ee/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image587573.png
Type: image/png
Size: 984 bytes
Desc: image587573.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/209a49ee/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image244471.png
Type: image/png
Size: 10728 bytes
Desc: image244471.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/209a49ee/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image947901.png
Type: image/png
Size: 1930 bytes
Desc: image947901.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/209a49ee/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image891048.png
Type: image/png
Size: 3004 bytes
Desc: image891048.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/209a49ee/attachment-0003.png>

From yaleman at ricetek.net  Thu Mar 30 15:33:24 2023
From: yaleman at ricetek.net (James Hodgkinson)
Date: Thu, 30 Mar 2023 14:33:24 +1000
Subject: [AusNOG] Critical 3CX Windows/Mac hack.
In-Reply-To: <MEYP282MB25976995206A1A35030FC3D4828E9@MEYP282MB2597.AUSP282.PROD.OUTLOOK.COM>
References: <CAC5zRjOB28=xU_qwGoNqdLq_ruqjQi19E1JG5Ls8fS-4fsznqw@mail.gmail.com>
 <MEYP282MB25976995206A1A35030FC3D4828E9@MEYP282MB2597.AUSP282.PROD.OUTLOOK.COM>
Message-ID: <f7eb80e1-5349-49eb-918d-ce7f257d563a@app.fastmail.com>

They've pulled the installers from their website and refer people to the web client...which is not much of a start...


On 2023-03-30 14:09 Greg Lipschitz wrote:
> Here is a list of commands (or make a shell script) to stop it phoning home and getting more payload.
> 
> # Disable 3CX Unattended-Upgrades Service
> 
> systemctl stop unattended-upgrades
> 
> # Collect the version of 3CX Desktop Apps on the Server
> 
> cd /var/lib/3cxpbx/Instance1/Data/Http/electron
> ls -la * > /root/3cx-desktop-versions.log
> 
> # Remove the files
> 
> rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/osx/*.dmg
> rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/osx/*.zip
> rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/windows/*.msi
> rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/windows/*.nupkg
> 
> 
> https://www.3cx.com/community/threads/threat-alerts-from-sentinelone-for-desktop-update-initiated-from-desktop-client.119806/page-5
> 
> 
> Sadly, 3CX haven't even acknowledged this yet.
> It would seem that their whole CI-CD pipeline has been compromised
> 
> Greg. 
> 
> 
> 
> Greg Lipschitz
>  | 
> Founder & CEO
>  | 
> Summit Internet
> *glipschitz at summitinternet.com.au*
> *summitinternet.com.au*
> *1300 049 749* <tel:1300%20049%20749>
> *Unit 2, 31-39 Norcal Road, Nunawading VIC 3131* <https://www.google.com/maps?cid=12522583051503623677&_ga=2.149009334.1057584350.1554770858-1081443428.1554770858>
> 
> 
> Summit Internet <http://summitinternet.com.au/>
> 
> 
> 
> *From:* AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Rob Thomas <xrobau at gmail.com>
> *Sent:* 30 March 2023 14:54
> *To:* <ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
> *Subject:* [AusNOG] Critical 3CX Windows/Mac hack.
>  
> As no-one's mentioned it here yet, I just thought I'd bring up the zero-day, in the wild, active RIGHT NOW, trojan 3CX Windows and Mac apps.
> 
> If you, or you have clients, running 3CX, make sure they ARE NOT using the app. If they are, their machines are probably already owned, and all their stored credentials and session cookies have been leaked.
> 
> https://www.bleepingcomputer.com/news/security/hackers-compromise-3cx-desktop-app-in-a-supply-chain-attack/amp/ <https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.bleepingcomputer.com%2Fnews%2Fsecurity%2Fhackers-compromise-3cx-desktop-app-in-a-supply-chain-attack%2Famp%2F&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C5134fed0ee3f4dbc894808db30d2a12f%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157453430051909%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UvNTww7E05nvQnaDQ25Qc8XytZFC%2FhIseT3MHYckCNM%3D&reserved=0>
> 
> This is really bad. Sorry 8-(
> 
> --Rob
> 
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/dfa68267/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image587573.png
Type: image/png
Size: 984 bytes
Desc: not available
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/dfa68267/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image244471.png
Type: image/png
Size: 10728 bytes
Desc: not available
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/dfa68267/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image947901.png
Type: image/png
Size: 1930 bytes
Desc: not available
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/dfa68267/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image891048.png
Type: image/png
Size: 3004 bytes
Desc: not available
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/dfa68267/attachment-0003.png>

From Nathan.Brookfield at iperium.com.au  Thu Mar 30 15:51:08 2023
From: Nathan.Brookfield at iperium.com.au (Nathan Brookfield)
Date: Thu, 30 Mar 2023 04:51:08 +0000
Subject: [AusNOG] Critical 3CX Windows/Mac hack.
In-Reply-To: <SYBP282MB0538C9BA5014A95976B5496BAF8E9@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM>
References: <CAC5zRjOB28=xU_qwGoNqdLq_ruqjQi19E1JG5Ls8fS-4fsznqw@mail.gmail.com>
 <MEYP282MB25976995206A1A35030FC3D4828E9@MEYP282MB2597.AUSP282.PROD.OUTLOOK.COM>
 <SYBP282MB0538C9BA5014A95976B5496BAF8E9@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM>
Message-ID: <MEYP282MB252094D1B586FBB7580B401EDB8E9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>

To be fair, they likely don?t know much yet and things are probably pretty hectic?. Give them time, crisis management is probably only kicking in now.

From: AusNOG <ausnog-bounces at lists.ausnog.net> On Behalf Of Christopher Hawker
Sent: Thursday, March 30, 2023 3:31 PM
To: Greg Lipschitz <glipschitz at summitinternet.com.au>; Rob Thomas <xrobau at gmail.com>; <ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] Critical 3CX Windows/Mac hack.

It appears their sales team have no info regarding this. Just rang our Senior AM at 3CX and they've advised that they have no information, and that they are referring anyone who calls to their technical teams via support tickets in the 3CX portal.

Not a good look for them.

CH

Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: AusNOG <ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>> on behalf of Greg Lipschitz <glipschitz at summitinternet.com.au<mailto:glipschitz at summitinternet.com.au>>
Sent: Thursday, March 30, 2023 3:09:45 PM
To: Rob Thomas <xrobau at gmail.com<mailto:xrobau at gmail.com>>; <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>> <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: Re: [AusNOG] Critical 3CX Windows/Mac hack.

Here is a list of commands (or make a shell script) to stop it phoning home and getting more payload.

# Disable 3CX Unattended-Upgrades Service
systemctl stop unattended-upgrades

# Collect the version of 3CX Desktop Apps on the Server

cd /var/lib/3cxpbx/Instance1/Data/Http/electron
ls -la * > /root/3cx-desktop-versions.log

# Remove the files

rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/osx/*.dmg
rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/osx/*.zip
rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/windows/*.msi
rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/windows/*.nupkg


https://www.3cx.com/community/threads/threat-alerts-from-sentinelone-for-desktop-update-initiated-from-desktop-client.119806/page-5


Sadly, 3CX haven't even acknowledged this yet.
It would seem that their whole CI-CD pipeline has been compromised

Greg.


Greg Lipschitz?
 |
Founder & CEO
 |
Summit Internet
glipschitz at summitinternet.com.au<mailto:glipschitz at summitinternet.com.au>
summitinternet.com.au<http://summitinternet.com.au>
1300 049 749<tel:1300%20049%20749>
Unit 2, 31-39 Norcal Road, Nunawading VIC 3131<https://www.google.com/maps?cid=12522583051503623677&_ga=2.149009334.1057584350.1554770858-1081443428.1554770858>
[cid:image001.png at 01D9631F.70B4E6A0]
[cid:image002.png at 01D9631F.70B4E6A0]
[Summit Internet]<http://summitinternet.com.au/>
[cid:image004.png at 01D9631F.70B4E6A0]

________________________________
From: AusNOG <ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>> on behalf of Rob Thomas <xrobau at gmail.com<mailto:xrobau at gmail.com>>
Sent: 30 March 2023 14:54
To: <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>> <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: [AusNOG] Critical 3CX Windows/Mac hack.

As no-one's mentioned it here yet, I just thought I'd bring up the zero-day, in the wild, active RIGHT NOW, trojan 3CX Windows and Mac apps.

If you, or you have clients, running 3CX, make sure they ARE NOT using the app. If they are, their machines are probably already owned, and all their stored credentials and session cookies have been leaked.

https://www.bleepingcomputer.com/news/security/hackers-compromise-3cx-desktop-app-in-a-supply-chain-attack/amp/<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.bleepingcomputer.com%2Fnews%2Fsecurity%2Fhackers-compromise-3cx-desktop-app-in-a-supply-chain-attack%2Famp%2F&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C5134fed0ee3f4dbc894808db30d2a12f%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157453430051909%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UvNTww7E05nvQnaDQ25Qc8XytZFC%2FhIseT3MHYckCNM%3D&reserved=0>

This is really bad. Sorry 8-(

--Rob

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/3cb5288d/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 984 bytes
Desc: image001.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/3cb5288d/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 10728 bytes
Desc: image002.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/3cb5288d/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 1930 bytes
Desc: image003.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/3cb5288d/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 3004 bytes
Desc: image004.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/3cb5288d/attachment-0007.png>

From matthew at htsol.com.au  Thu Mar 30 15:57:01 2023
From: matthew at htsol.com.au (Matthew Mace)
Date: Thu, 30 Mar 2023 04:57:01 +0000
Subject: [AusNOG] Critical 3CX Windows/Mac hack.
In-Reply-To: <MEYP282MB252094D1B586FBB7580B401EDB8E9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
References: <CAC5zRjOB28=xU_qwGoNqdLq_ruqjQi19E1JG5Ls8fS-4fsznqw@mail.gmail.com>
 <MEYP282MB25976995206A1A35030FC3D4828E9@MEYP282MB2597.AUSP282.PROD.OUTLOOK.COM>
 <SYBP282MB0538C9BA5014A95976B5496BAF8E9@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM>
 <MEYP282MB252094D1B586FBB7580B401EDB8E9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
Message-ID: <SY4P282MB351717C698442CD63C8BF0E6958E9@SY4P282MB3517.AUSP282.PROD.OUTLOOK.COM>

Can anyone definitively confirm that they?ve personally seen it get picked up by anything else than S1?

In addition to  this anyone that has had it installed at a site and also run a premium DNS filtering service (Umbrella, DNS Filter etc.) and/or premium routers with DPI (Sonicwall, Firebox etc.), do you know if they picked up this traffic and stopped it? I would be hoping so.

Definitely curious to know either way.



Matthew Mace


From: AusNOG <ausnog-bounces at lists.ausnog.net> On Behalf Of Nathan Brookfield
Sent: Thursday, March 30, 2023 2:51 PM
To: Christopher Hawker <chris at thesysadmin.dev>; Greg Lipschitz <glipschitz at summitinternet.com.au>; Rob Thomas <xrobau at gmail.com>; <ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] Critical 3CX Windows/Mac hack.

To be fair, they likely don?t know much yet and things are probably pretty hectic?. Give them time, crisis management is probably only kicking in now.

From: AusNOG <ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>> On Behalf Of Christopher Hawker
Sent: Thursday, March 30, 2023 3:31 PM
To: Greg Lipschitz <glipschitz at summitinternet.com.au<mailto:glipschitz at summitinternet.com.au>>; Rob Thomas <xrobau at gmail.com<mailto:xrobau at gmail.com>>; <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>> <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: Re: [AusNOG] Critical 3CX Windows/Mac hack.

It appears their sales team have no info regarding this. Just rang our Senior AM at 3CX and they've advised that they have no information, and that they are referring anyone who calls to their technical teams via support tickets in the 3CX portal.

Not a good look for them.

CH

Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: AusNOG <ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>> on behalf of Greg Lipschitz <glipschitz at summitinternet.com.au<mailto:glipschitz at summitinternet.com.au>>
Sent: Thursday, March 30, 2023 3:09:45 PM
To: Rob Thomas <xrobau at gmail.com<mailto:xrobau at gmail.com>>; <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>> <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: Re: [AusNOG] Critical 3CX Windows/Mac hack.

Here is a list of commands (or make a shell script) to stop it phoning home and getting more payload.

# Disable 3CX Unattended-Upgrades Service
systemctl stop unattended-upgrades

# Collect the version of 3CX Desktop Apps on the Server

cd /var/lib/3cxpbx/Instance1/Data/Http/electron
ls -la * > /root/3cx-desktop-versions.log

# Remove the files

rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/osx/*.dmg
rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/osx/*.zip
rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/windows/*.msi
rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/windows/*.nupkg


https://www.3cx.com/community/threads/threat-alerts-from-sentinelone-for-desktop-update-initiated-from-desktop-client.119806/page-5


Sadly, 3CX haven't even acknowledged this yet.
It would seem that their whole CI-CD pipeline has been compromised

Greg.


Greg Lipschitz?
 |
Founder & CEO
 |
Summit Internet
glipschitz at summitinternet.com.au<mailto:glipschitz at summitinternet.com.au>
summitinternet.com.au<http://summitinternet.com.au>
1300 049 749<tel:1300%20049%20749>
Unit 2, 31-39 Norcal Road, Nunawading VIC 3131<https://www.google.com/maps?cid=12522583051503623677&_ga=2.149009334.1057584350.1554770858-1081443428.1554770858>
[cid:image001.png at 01D96317.E105C670]
[cid:image002.png at 01D96317.E105C670]
[Summit Internet]<http://summitinternet.com.au/>
[cid:image004.png at 01D96317.E105C670]

________________________________
From: AusNOG <ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>> on behalf of Rob Thomas <xrobau at gmail.com<mailto:xrobau at gmail.com>>
Sent: 30 March 2023 14:54
To: <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>> <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: [AusNOG] Critical 3CX Windows/Mac hack.

As no-one's mentioned it here yet, I just thought I'd bring up the zero-day, in the wild, active RIGHT NOW, trojan 3CX Windows and Mac apps.

If you, or you have clients, running 3CX, make sure they ARE NOT using the app. If they are, their machines are probably already owned, and all their stored credentials and session cookies have been leaked.

https://www.bleepingcomputer.com/news/security/hackers-compromise-3cx-desktop-app-in-a-supply-chain-attack/amp/<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.bleepingcomputer.com%2Fnews%2Fsecurity%2Fhackers-compromise-3cx-desktop-app-in-a-supply-chain-attack%2Famp%2F&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C5134fed0ee3f4dbc894808db30d2a12f%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157453430051909%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UvNTww7E05nvQnaDQ25Qc8XytZFC%2FhIseT3MHYckCNM%3D&reserved=0>

This is really bad. Sorry 8-(

--Rob

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/456d3086/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 984 bytes
Desc: image001.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/456d3086/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 10728 bytes
Desc: image002.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/456d3086/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 1930 bytes
Desc: image003.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/456d3086/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 3004 bytes
Desc: image004.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/456d3086/attachment-0003.png>

From alexander at neilson.net.nz  Thu Mar 30 16:08:10 2023
From: alexander at neilson.net.nz (Alexander Neilson)
Date: Thu, 30 Mar 2023 18:08:10 +1300
Subject: [AusNOG] Critical 3CX Windows/Mac hack.
In-Reply-To: <SY4P282MB351717C698442CD63C8BF0E6958E9@SY4P282MB3517.AUSP282.PROD.OUTLOOK.COM>
References: <CAC5zRjOB28=xU_qwGoNqdLq_ruqjQi19E1JG5Ls8fS-4fsznqw@mail.gmail.com>
 <MEYP282MB25976995206A1A35030FC3D4828E9@MEYP282MB2597.AUSP282.PROD.OUTLOOK.COM>
 <SYBP282MB0538C9BA5014A95976B5496BAF8E9@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM>
 <MEYP282MB252094D1B586FBB7580B401EDB8E9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <SY4P282MB351717C698442CD63C8BF0E6958E9@SY4P282MB3517.AUSP282.PROD.OUTLOOK.COM>
Message-ID: <CAHYTDkSuWv5zFarSTDvLicWw9A+ZuP6t-Kj_hPUwfFUkKf-K8g@mail.gmail.com>

I haven't seen it personally

However others are reporting it as separate investigations they have seen
the loader execute:
https://www.todyl.com/blog/post/threat-advisory-3cx-softphone-telephony-campaign
https://www.3cx.com/community/threads/3cx-desktop-app-vulnerability-security-group-contact.119930/
- Reports ESET detected it - possibly using signature / hash from S1
https://www.3cx.com/community/threads/threat-alerts-from-sentinelone-for-desktop-update-initiated-from-desktop-client.119806/post-558449
-  Cortex xdr Paloalto
https://www.3cx.com/community/threads/threat-alerts-from-sentinelone-for-desktop-update-initiated-from-desktop-client.119806/post-558708
- CrowdStrike
https://www.bleepingcomputer.com/news/security/hackers-compromise-3cx-desktop-app-in-a-supply-chain-attack/
- References Sophos


I am pretty confident that if this isn't a malicious actor doing this then
3CX has performed the mother of all response tests on its customers over
the past week and should have had a better reply than silence when they
were asked about it.

Regards
Alexander

Alexander Neilson
Neilson Productions Limited

alexander at neilson.net.nz
021 329 681
022 456 2326


On Thu, 30 Mar 2023 at 17:57, Matthew Mace <matthew at htsol.com.au> wrote:

> Can anyone definitively confirm that they?ve personally seen it get picked
> up by anything else than S1?
>
>
>
> In addition to  this anyone that has had it installed at a site and also
> run a premium DNS filtering service (Umbrella, DNS Filter etc.) and/or
> premium routers with DPI (Sonicwall, Firebox etc.), do you know if they
> picked up this traffic and stopped it? I would be hoping so.
>
>
>
> Definitely curious to know either way.
>
>
>
>
>
>
>
> *Matthew Mace*
>
>
>
>
>
> *From:* AusNOG <ausnog-bounces at lists.ausnog.net> *On Behalf Of *Nathan
> Brookfield
> *Sent:* Thursday, March 30, 2023 2:51 PM
> *To:* Christopher Hawker <chris at thesysadmin.dev>; Greg Lipschitz <
> glipschitz at summitinternet.com.au>; Rob Thomas <xrobau at gmail.com>; <
> ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
> *Subject:* Re: [AusNOG] Critical 3CX Windows/Mac hack.
>
>
>
> To be fair, they likely don?t know much yet and things are probably pretty
> hectic?. Give them time, crisis management is probably only kicking in now.
>
>
>
> *From:* AusNOG <ausnog-bounces at lists.ausnog.net> *On Behalf Of *Christopher
> Hawker
> *Sent:* Thursday, March 30, 2023 3:31 PM
> *To:* Greg Lipschitz <glipschitz at summitinternet.com.au>; Rob Thomas <
> xrobau at gmail.com>; <ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
> *Subject:* Re: [AusNOG] Critical 3CX Windows/Mac hack.
>
>
>
> It appears their sales team have no info regarding this. Just rang our
> Senior AM at 3CX and they've advised that they have no information, and
> that they are referring anyone who calls to their technical teams via
> support tickets in the 3CX portal.
>
>
>
> Not a good look for them.
>
>
>
> CH
>
>
>
> Get Outlook for Android <https://aka.ms/AAb9ysg>
> ------------------------------
>
> *From:* AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Greg
> Lipschitz <glipschitz at summitinternet.com.au>
> *Sent:* Thursday, March 30, 2023 3:09:45 PM
> *To:* Rob Thomas <xrobau at gmail.com>; <ausnog at lists.ausnog.net> <
> ausnog at lists.ausnog.net>
> *Subject:* Re: [AusNOG] Critical 3CX Windows/Mac hack.
>
>
>
> Here is a list of commands (or make a shell script) to stop it phoning
> home and getting more payload.
>
>
>
> # Disable 3CX Unattended-Upgrades Service
>
> systemctl stop unattended-upgrades
>
>
>
> # Collect the version of 3CX Desktop Apps on the Server
>
>
>
> cd /var/lib/3cxpbx/Instance1/Data/Http/electron
>
> ls -la * > /root/3cx-desktop-versions.log
>
>
>
> # Remove the files
>
>
>
> rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/osx/*.dmg
>
> rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/osx/*.zip
>
> rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/windows/*.msi
>
> rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/windows/*.nupkg
>
>
>
>
>
>
> https://www.3cx.com/community/threads/threat-alerts-from-sentinelone-for-desktop-update-initiated-from-desktop-client.119806/page-5
>
>
>
>
>
> Sadly, 3CX haven't even acknowledged this yet.
>
> It would seem that their whole CI-CD pipeline has been compromised
>
>
>
> Greg.
>
>
>
>
>
> *Greg Lipschitz**?*
>
>  |
>
> *Founder & CEO*
>
>  |
>
> *Summit Internet*
>
> *glipschitz at summitinternet.com.au* <glipschitz at summitinternet.com.au>
>
> *summitinternet.com.au* <http://summitinternet.com.au>
>
> *1300 049 749* <1300%20049%20749>
>
> *Unit 2, 31-39 Norcal Road, Nunawading VIC 3131*
> <https://www.google.com/maps?cid=12522583051503623677&_ga=2.149009334.1057584350.1554770858-1081443428.1554770858>
>
> [image: Summit Internet] <http://summitinternet.com.au/>
>
>
> ------------------------------
>
> *From:* AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Rob Thomas <
> xrobau at gmail.com>
> *Sent:* 30 March 2023 14:54
> *To:* <ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
> *Subject:* [AusNOG] Critical 3CX Windows/Mac hack.
>
>
>
> As no-one's mentioned it here yet, I just thought I'd bring up the
> zero-day, in the wild, active RIGHT NOW, trojan 3CX Windows and Mac apps.
>
>
>
> If you, or you have clients, running 3CX, make sure they ARE NOT using the
> app. If they are, their machines are probably already owned, and all their
> stored credentials and session cookies have been leaked.
>
>
>
>
> https://www.bleepingcomputer.com/news/security/hackers-compromise-3cx-desktop-app-in-a-supply-chain-attack/amp/
> <https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.bleepingcomputer.com%2Fnews%2Fsecurity%2Fhackers-compromise-3cx-desktop-app-in-a-supply-chain-attack%2Famp%2F&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C5134fed0ee3f4dbc894808db30d2a12f%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157453430051909%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UvNTww7E05nvQnaDQ25Qc8XytZFC%2FhIseT3MHYckCNM%3D&reserved=0>
>
>
>
> This is really bad. Sorry 8-(
>
>
>
> --Rob
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/eed1c008/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 984 bytes
Desc: not available
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/eed1c008/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 10728 bytes
Desc: not available
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/eed1c008/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 1930 bytes
Desc: not available
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/eed1c008/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 3004 bytes
Desc: not available
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/eed1c008/attachment-0003.png>

From tim.jago at cbros.co.nz  Thu Mar 30 16:12:14 2023
From: tim.jago at cbros.co.nz (Tim Jago)
Date: Thu, 30 Mar 2023 18:12:14 +1300
Subject: [AusNOG] Critical 3CX Windows/Mac hack.
In-Reply-To: <CAHYTDkSuWv5zFarSTDvLicWw9A+ZuP6t-Kj_hPUwfFUkKf-K8g@mail.gmail.com>
References: <CAC5zRjOB28=xU_qwGoNqdLq_ruqjQi19E1JG5Ls8fS-4fsznqw@mail.gmail.com>
 <MEYP282MB25976995206A1A35030FC3D4828E9@MEYP282MB2597.AUSP282.PROD.OUTLOOK.COM>
 <SYBP282MB0538C9BA5014A95976B5496BAF8E9@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM>
 <MEYP282MB252094D1B586FBB7580B401EDB8E9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <SY4P282MB351717C698442CD63C8BF0E6958E9@SY4P282MB3517.AUSP282.PROD.OUTLOOK.COM>
 <CAHYTDkSuWv5zFarSTDvLicWw9A+ZuP6t-Kj_hPUwfFUkKf-K8g@mail.gmail.com>
Message-ID: <06bb84ae-c352-743c-bcb7-b7fb43bf9013@cbros.co.nz>

Confirmed now at least... 
https://www.3cx.com/community/threads/threat-alerts-from-sentinelone-for-desktop-update-initiated-from-desktop-client.119806/post-558899

From dazzagibbs at gmail.com  Thu Mar 30 16:42:10 2023
From: dazzagibbs at gmail.com (DaZZa)
Date: Thu, 30 Mar 2023 16:42:10 +1100
Subject: [AusNOG] Critical 3CX Windows/Mac hack.
In-Reply-To: <CAHYTDkSuWv5zFarSTDvLicWw9A+ZuP6t-Kj_hPUwfFUkKf-K8g@mail.gmail.com>
References: <CAC5zRjOB28=xU_qwGoNqdLq_ruqjQi19E1JG5Ls8fS-4fsznqw@mail.gmail.com>
 <MEYP282MB25976995206A1A35030FC3D4828E9@MEYP282MB2597.AUSP282.PROD.OUTLOOK.COM>
 <SYBP282MB0538C9BA5014A95976B5496BAF8E9@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM>
 <MEYP282MB252094D1B586FBB7580B401EDB8E9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <SY4P282MB351717C698442CD63C8BF0E6958E9@SY4P282MB3517.AUSP282.PROD.OUTLOOK.COM>
 <CAHYTDkSuWv5zFarSTDvLicWw9A+ZuP6t-Kj_hPUwfFUkKf-K8g@mail.gmail.com>
Message-ID: <CAMpV8MpTYGwrUw6sZ9MHLgw6bnXGqYHJh658-8wyteZfUJPceg@mail.gmail.com>

>From a security perspective, the utterly terrifying part of most of these
responses boils down to "Oh, must be a glitch in the AV, I'll *whitelist*
it so it doesn;t get caught".

Jesus Wept. I'd be bashing heads if anyone in my company even suggested
that without a much more thorough investigation!

D

On Thu, 30 Mar 2023 at 16:08, Alexander Neilson <alexander at neilson.net.nz>
wrote:

> I haven't seen it personally
>
> However others are reporting it as separate investigations they have seen
> the loader execute:
>
> https://www.todyl.com/blog/post/threat-advisory-3cx-softphone-telephony-campaign
>
> https://www.3cx.com/community/threads/3cx-desktop-app-vulnerability-security-group-contact.119930/
> - Reports ESET detected it - possibly using signature / hash from S1
>
> https://www.3cx.com/community/threads/threat-alerts-from-sentinelone-for-desktop-update-initiated-from-desktop-client.119806/post-558449
> -  Cortex xdr Paloalto
>
> https://www.3cx.com/community/threads/threat-alerts-from-sentinelone-for-desktop-update-initiated-from-desktop-client.119806/post-558708
> - CrowdStrike
>
> https://www.bleepingcomputer.com/news/security/hackers-compromise-3cx-desktop-app-in-a-supply-chain-attack/
> - References Sophos
>
>
> I am pretty confident that if this isn't a malicious actor doing this then
> 3CX has performed the mother of all response tests on its customers over
> the past week and should have had a better reply than silence when they
> were asked about it.
>
> Regards
> Alexander
>
> Alexander Neilson
> Neilson Productions Limited
>
> alexander at neilson.net.nz
> 021 329 681
> 022 456 2326
>
>
> On Thu, 30 Mar 2023 at 17:57, Matthew Mace <matthew at htsol.com.au> wrote:
>
>> Can anyone definitively confirm that they?ve personally seen it get
>> picked up by anything else than S1?
>>
>>
>>
>> In addition to  this anyone that has had it installed at a site and also
>> run a premium DNS filtering service (Umbrella, DNS Filter etc.) and/or
>> premium routers with DPI (Sonicwall, Firebox etc.), do you know if they
>> picked up this traffic and stopped it? I would be hoping so.
>>
>>
>>
>> Definitely curious to know either way.
>>
>>
>>
>>
>>
>>
>>
>> *Matthew Mace*
>>
>>
>>
>>
>>
>> *From:* AusNOG <ausnog-bounces at lists.ausnog.net> *On Behalf Of *Nathan
>> Brookfield
>> *Sent:* Thursday, March 30, 2023 2:51 PM
>> *To:* Christopher Hawker <chris at thesysadmin.dev>; Greg Lipschitz <
>> glipschitz at summitinternet.com.au>; Rob Thomas <xrobau at gmail.com>; <
>> ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
>> *Subject:* Re: [AusNOG] Critical 3CX Windows/Mac hack.
>>
>>
>>
>> To be fair, they likely don?t know much yet and things are probably
>> pretty hectic?. Give them time, crisis management is probably only kicking
>> in now.
>>
>>
>>
>> *From:* AusNOG <ausnog-bounces at lists.ausnog.net> *On Behalf Of *Christopher
>> Hawker
>> *Sent:* Thursday, March 30, 2023 3:31 PM
>> *To:* Greg Lipschitz <glipschitz at summitinternet.com.au>; Rob Thomas <
>> xrobau at gmail.com>; <ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
>> *Subject:* Re: [AusNOG] Critical 3CX Windows/Mac hack.
>>
>>
>>
>> It appears their sales team have no info regarding this. Just rang our
>> Senior AM at 3CX and they've advised that they have no information, and
>> that they are referring anyone who calls to their technical teams via
>> support tickets in the 3CX portal.
>>
>>
>>
>> Not a good look for them.
>>
>>
>>
>> CH
>>
>>
>>
>> Get Outlook for Android <https://aka.ms/AAb9ysg>
>> ------------------------------
>>
>> *From:* AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Greg
>> Lipschitz <glipschitz at summitinternet.com.au>
>> *Sent:* Thursday, March 30, 2023 3:09:45 PM
>> *To:* Rob Thomas <xrobau at gmail.com>; <ausnog at lists.ausnog.net> <
>> ausnog at lists.ausnog.net>
>> *Subject:* Re: [AusNOG] Critical 3CX Windows/Mac hack.
>>
>>
>>
>> Here is a list of commands (or make a shell script) to stop it phoning
>> home and getting more payload.
>>
>>
>>
>> # Disable 3CX Unattended-Upgrades Service
>>
>> systemctl stop unattended-upgrades
>>
>>
>>
>> # Collect the version of 3CX Desktop Apps on the Server
>>
>>
>>
>> cd /var/lib/3cxpbx/Instance1/Data/Http/electron
>>
>> ls -la * > /root/3cx-desktop-versions.log
>>
>>
>>
>> # Remove the files
>>
>>
>>
>> rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/osx/*.dmg
>>
>> rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/osx/*.zip
>>
>> rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/windows/*.msi
>>
>> rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/windows/*.nupkg
>>
>>
>>
>>
>>
>>
>> https://www.3cx.com/community/threads/threat-alerts-from-sentinelone-for-desktop-update-initiated-from-desktop-client.119806/page-5
>>
>>
>>
>>
>>
>> Sadly, 3CX haven't even acknowledged this yet.
>>
>> It would seem that their whole CI-CD pipeline has been compromised
>>
>>
>>
>> Greg.
>>
>>
>>
>>
>>
>> *Greg Lipschitz**?*
>>
>>  |
>>
>> *Founder & CEO*
>>
>>  |
>>
>> *Summit Internet*
>>
>> *glipschitz at summitinternet.com.au* <glipschitz at summitinternet.com.au>
>>
>> *summitinternet.com.au* <http://summitinternet.com.au>
>>
>> *1300 049 749* <1300%20049%20749>
>>
>> *Unit 2, 31-39 Norcal Road, Nunawading VIC 3131*
>> <https://www.google.com/maps?cid=12522583051503623677&_ga=2.149009334.1057584350.1554770858-1081443428.1554770858>
>>
>> [image: Summit Internet] <http://summitinternet.com.au/>
>>
>>
>> ------------------------------
>>
>> *From:* AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Rob Thomas
>> <xrobau at gmail.com>
>> *Sent:* 30 March 2023 14:54
>> *To:* <ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
>> *Subject:* [AusNOG] Critical 3CX Windows/Mac hack.
>>
>>
>>
>> As no-one's mentioned it here yet, I just thought I'd bring up the
>> zero-day, in the wild, active RIGHT NOW, trojan 3CX Windows and Mac apps.
>>
>>
>>
>> If you, or you have clients, running 3CX, make sure they ARE NOT using
>> the app. If they are, their machines are probably already owned, and all
>> their stored credentials and session cookies have been leaked.
>>
>>
>>
>>
>> https://www.bleepingcomputer.com/news/security/hackers-compromise-3cx-desktop-app-in-a-supply-chain-attack/amp/
>> <https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.bleepingcomputer.com%2Fnews%2Fsecurity%2Fhackers-compromise-3cx-desktop-app-in-a-supply-chain-attack%2Famp%2F&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C5134fed0ee3f4dbc894808db30d2a12f%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157453430051909%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UvNTww7E05nvQnaDQ25Qc8XytZFC%2FhIseT3MHYckCNM%3D&reserved=0>
>>
>>
>>
>> This is really bad. Sorry 8-(
>>
>>
>>
>> --Rob
>>
>>
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>


-- 
veg?e?tar?i?an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/c1afe638/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 984 bytes
Desc: not available
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/c1afe638/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 10728 bytes
Desc: not available
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/c1afe638/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 1930 bytes
Desc: not available
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/c1afe638/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 3004 bytes
Desc: not available
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/c1afe638/attachment-0007.png>

From glipschitz at summitinternet.com.au  Thu Mar 30 16:48:26 2023
From: glipschitz at summitinternet.com.au (Greg Lipschitz)
Date: Thu, 30 Mar 2023 05:48:26 +0000
Subject: [AusNOG] Critical 3CX Windows/Mac hack.
In-Reply-To: <SY4P282MB351717C698442CD63C8BF0E6958E9@SY4P282MB3517.AUSP282.PROD.OUTLOOK.COM>
References: <CAC5zRjOB28=xU_qwGoNqdLq_ruqjQi19E1JG5Ls8fS-4fsznqw@mail.gmail.com>
 <MEYP282MB25976995206A1A35030FC3D4828E9@MEYP282MB2597.AUSP282.PROD.OUTLOOK.COM>
 <SYBP282MB0538C9BA5014A95976B5496BAF8E9@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM>
 <MEYP282MB252094D1B586FBB7580B401EDB8E9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <SY4P282MB351717C698442CD63C8BF0E6958E9@SY4P282MB3517.AUSP282.PROD.OUTLOOK.COM>
Message-ID: <MEYP282MB2597D4036A8C7496B8A2B292828E9@MEYP282MB2597.AUSP282.PROD.OUTLOOK.COM>

Windows Defender picked it up too.

Greg Lipschitz | Founder & CEO | Summit Internet
glipschitz at summitinternet.com.au
summitinternet.com.au
1300 049 749
Unit 2, 31-39 Norcal Road, Nunawading VIC 3131
Summit Internet
________________________________
From: Matthew Mace <matthew at htsol.com.au>
Sent: 30 March 2023 15:57
To: Nathan Brookfield <Nathan.Brookfield at iperium.com.au>; Christopher Hawker <chris at thesysadmin.dev>; Greg Lipschitz <glipschitz at summitinternet.com.au>; Rob Thomas <xrobau at gmail.com>; <ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
Subject: RE: [AusNOG] Critical 3CX Windows/Mac hack.

You don't often get email from matthew at htsol.com.au. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification>

Can anyone definitively confirm that they?ve personally seen it get picked up by anything else than S1?



In addition to  this anyone that has had it installed at a site and also run a premium DNS filtering service (Umbrella, DNS Filter etc.) and/or premium routers with DPI (Sonicwall, Firebox etc.), do you know if they picked up this traffic and stopped it? I would be hoping so.



Definitely curious to know either way.







Matthew Mace





From: AusNOG <ausnog-bounces at lists.ausnog.net> On Behalf Of Nathan Brookfield
Sent: Thursday, March 30, 2023 2:51 PM
To: Christopher Hawker <chris at thesysadmin.dev>; Greg Lipschitz <glipschitz at summitinternet.com.au>; Rob Thomas <xrobau at gmail.com>; <ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] Critical 3CX Windows/Mac hack.



To be fair, they likely don?t know much yet and things are probably pretty hectic?. Give them time, crisis management is probably only kicking in now.



From: AusNOG <ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>> On Behalf Of Christopher Hawker
Sent: Thursday, March 30, 2023 3:31 PM
To: Greg Lipschitz <glipschitz at summitinternet.com.au<mailto:glipschitz at summitinternet.com.au>>; Rob Thomas <xrobau at gmail.com<mailto:xrobau at gmail.com>>; <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>> <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: Re: [AusNOG] Critical 3CX Windows/Mac hack.



It appears their sales team have no info regarding this. Just rang our Senior AM at 3CX and they've advised that they have no information, and that they are referring anyone who calls to their technical teams via support tickets in the 3CX portal.



Not a good look for them.



CH



Get Outlook for Android<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C28ab046f470a4f19932b08db30db3490%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157490273890466%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=LeXZsLN4cUATWMtiKqKRMgdDICBmFCmTUpQ3wZ%2FBGK4%3D&reserved=0>

________________________________

From: AusNOG <ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>> on behalf of Greg Lipschitz <glipschitz at summitinternet.com.au<mailto:glipschitz at summitinternet.com.au>>
Sent: Thursday, March 30, 2023 3:09:45 PM
To: Rob Thomas <xrobau at gmail.com<mailto:xrobau at gmail.com>>; <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>> <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: Re: [AusNOG] Critical 3CX Windows/Mac hack.



Here is a list of commands (or make a shell script) to stop it phoning home and getting more payload.



# Disable 3CX Unattended-Upgrades Service

systemctl stop unattended-upgrades



# Collect the version of 3CX Desktop Apps on the Server



cd /var/lib/3cxpbx/Instance1/Data/Http/electron

ls -la * > /root/3cx-desktop-versions.log



# Remove the files



rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/osx/*.dmg

rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/osx/*.zip

rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/windows/*.msi

rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/windows/*.nupkg





https://www.3cx.com/community/threads/threat-alerts-from-sentinelone-for-desktop-update-initiated-from-desktop-client.119806/page-5<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.3cx.com%2Fcommunity%2Fthreads%2Fthreat-alerts-from-sentinelone-for-desktop-update-initiated-from-desktop-client.119806%2Fpage-5&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C28ab046f470a4f19932b08db30db3490%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157490273890466%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=gBvUcr9Pkokg10aeP864qVSsMZDg6KE%2FuuDUDi2imtE%3D&reserved=0>





Sadly, 3CX haven't even acknowledged this yet.

It would seem that their whole CI-CD pipeline has been compromised



Greg.





Greg Lipschitz?

 |

Founder & CEO

 |

Summit Internet

glipschitz at summitinternet.com.au<mailto:glipschitz at summitinternet.com.au>

summitinternet.com.au<https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsummitinternet.com.au%2F&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C28ab046f470a4f19932b08db30db3490%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157490273890466%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zO2mC4vUjaIW7YOuPf7i45hrukQ5fRzyFikE1GWDZjQ%3D&reserved=0>

1300 049 749<tel:1300%20049%20749>

Unit 2, 31-39 Norcal Road, Nunawading VIC 3131<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.google.com%2Fmaps%3Fcid%3D12522583051503623677%26_ga%3D2.149009334.1057584350.1554770858-1081443428.1554770858&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C28ab046f470a4f19932b08db30db3490%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157490273890466%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=VkgdWHriLjc1epWzbMAJ6U7ksoRzTvp6An1JpnbAt1k%3D&reserved=0>

[cid:image001.png at 01D96317.E105C670]

[cid:image002.png at 01D96317.E105C670]

[Summit Internet]<https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsummitinternet.com.au%2F&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C28ab046f470a4f19932b08db30db3490%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157490273890466%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zO2mC4vUjaIW7YOuPf7i45hrukQ5fRzyFikE1GWDZjQ%3D&reserved=0>

[cid:image004.png at 01D96317.E105C670]



________________________________

From: AusNOG <ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>> on behalf of Rob Thomas <xrobau at gmail.com<mailto:xrobau at gmail.com>>
Sent: 30 March 2023 14:54
To: <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>> <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: [AusNOG] Critical 3CX Windows/Mac hack.



As no-one's mentioned it here yet, I just thought I'd bring up the zero-day, in the wild, active RIGHT NOW, trojan 3CX Windows and Mac apps.



If you, or you have clients, running 3CX, make sure they ARE NOT using the app. If they are, their machines are probably already owned, and all their stored credentials and session cookies have been leaked.



https://www.bleepingcomputer.com/news/security/hackers-compromise-3cx-desktop-app-in-a-supply-chain-attack/amp/<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.bleepingcomputer.com%2Fnews%2Fsecurity%2Fhackers-compromise-3cx-desktop-app-in-a-supply-chain-attack%2Famp%2F&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C28ab046f470a4f19932b08db30db3490%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157490273890466%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=NUiZuLaVjcCKlwNPMt1Np84joAu8dkLObF0L7rfce6s%3D&reserved=0>



This is really bad. Sorry 8-(



--Rob


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/eb13696e/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 984 bytes
Desc: image001.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/eb13696e/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 10728 bytes
Desc: image002.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/eb13696e/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 1930 bytes
Desc: image003.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/eb13696e/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 3004 bytes
Desc: image004.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/eb13696e/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image106381.png
Type: image/png
Size: 984 bytes
Desc: image106381.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/eb13696e/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image224032.png
Type: image/png
Size: 10728 bytes
Desc: image224032.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/eb13696e/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image723620.png
Type: image/png
Size: 1930 bytes
Desc: image723620.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/eb13696e/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image974825.png
Type: image/png
Size: 3004 bytes
Desc: image974825.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/eb13696e/attachment-0007.png>

From jrandombob at darkglade.com  Thu Mar 30 16:53:52 2023
From: jrandombob at darkglade.com (Jrandombob)
Date: Thu, 30 Mar 2023 16:53:52 +1100
Subject: [AusNOG] Critical 3CX Windows/Mac hack.
In-Reply-To: <CAMpV8MpTYGwrUw6sZ9MHLgw6bnXGqYHJh658-8wyteZfUJPceg@mail.gmail.com>
References: <CAC5zRjOB28=xU_qwGoNqdLq_ruqjQi19E1JG5Ls8fS-4fsznqw@mail.gmail.com>
 <MEYP282MB25976995206A1A35030FC3D4828E9@MEYP282MB2597.AUSP282.PROD.OUTLOOK.COM>
 <SYBP282MB0538C9BA5014A95976B5496BAF8E9@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM>
 <MEYP282MB252094D1B586FBB7580B401EDB8E9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <SY4P282MB351717C698442CD63C8BF0E6958E9@SY4P282MB3517.AUSP282.PROD.OUTLOOK.COM>
 <CAHYTDkSuWv5zFarSTDvLicWw9A+ZuP6t-Kj_hPUwfFUkKf-K8g@mail.gmail.com>
 <CAMpV8MpTYGwrUw6sZ9MHLgw6bnXGqYHJh658-8wyteZfUJPceg@mail.gmail.com>
Message-ID: <CAGhfpxoMhpLEH5K4Nrn6=mSJ-vL-gy4PNP6dzVzc+01E-6koYg@mail.gmail.com>

Yeah, some of those forum threads are IMPRESSIVELY trainwrecky, I think the
most succinct evaluation I've seen is this one;
"Seriously. Your EDR tells you that your phone client is behaving like a C2
talking to North Korea, and your response is to put it in the whitelist?
Wow..."

On Thu, Mar 30, 2023 at 4:42?PM DaZZa <dazzagibbs at gmail.com> wrote:

> From a security perspective, the utterly terrifying part of most of these
> responses boils down to "Oh, must be a glitch in the AV, I'll *whitelist*
> it so it doesn;t get caught".
>
> Jesus Wept. I'd be bashing heads if anyone in my company even suggested
> that without a much more thorough investigation!
>
> D
>
> On Thu, 30 Mar 2023 at 16:08, Alexander Neilson <alexander at neilson.net.nz>
> wrote:
>
>> I haven't seen it personally
>>
>> However others are reporting it as separate investigations they have seen
>> the loader execute:
>>
>> https://www.todyl.com/blog/post/threat-advisory-3cx-softphone-telephony-campaign
>>
>> https://www.3cx.com/community/threads/3cx-desktop-app-vulnerability-security-group-contact.119930/
>> - Reports ESET detected it - possibly using signature / hash from S1
>>
>> https://www.3cx.com/community/threads/threat-alerts-from-sentinelone-for-desktop-update-initiated-from-desktop-client.119806/post-558449
>> -  Cortex xdr Paloalto
>>
>> https://www.3cx.com/community/threads/threat-alerts-from-sentinelone-for-desktop-update-initiated-from-desktop-client.119806/post-558708
>> - CrowdStrike
>>
>> https://www.bleepingcomputer.com/news/security/hackers-compromise-3cx-desktop-app-in-a-supply-chain-attack/
>> - References Sophos
>>
>>
>> I am pretty confident that if this isn't a malicious actor doing this
>> then 3CX has performed the mother of all response tests on its customers
>> over the past week and should have had a better reply than silence when
>> they were asked about it.
>>
>> Regards
>> Alexander
>>
>> Alexander Neilson
>> Neilson Productions Limited
>>
>> alexander at neilson.net.nz
>> 021 329 681
>> 022 456 2326
>>
>>
>> On Thu, 30 Mar 2023 at 17:57, Matthew Mace <matthew at htsol.com.au> wrote:
>>
>>> Can anyone definitively confirm that they?ve personally seen it get
>>> picked up by anything else than S1?
>>>
>>>
>>>
>>> In addition to  this anyone that has had it installed at a site and also
>>> run a premium DNS filtering service (Umbrella, DNS Filter etc.) and/or
>>> premium routers with DPI (Sonicwall, Firebox etc.), do you know if they
>>> picked up this traffic and stopped it? I would be hoping so.
>>>
>>>
>>>
>>> Definitely curious to know either way.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *Matthew Mace*
>>>
>>>
>>>
>>>
>>>
>>> *From:* AusNOG <ausnog-bounces at lists.ausnog.net> *On Behalf Of *Nathan
>>> Brookfield
>>> *Sent:* Thursday, March 30, 2023 2:51 PM
>>> *To:* Christopher Hawker <chris at thesysadmin.dev>; Greg Lipschitz <
>>> glipschitz at summitinternet.com.au>; Rob Thomas <xrobau at gmail.com>; <
>>> ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
>>> *Subject:* Re: [AusNOG] Critical 3CX Windows/Mac hack.
>>>
>>>
>>>
>>> To be fair, they likely don?t know much yet and things are probably
>>> pretty hectic?. Give them time, crisis management is probably only kicking
>>> in now.
>>>
>>>
>>>
>>> *From:* AusNOG <ausnog-bounces at lists.ausnog.net> *On Behalf Of *Christopher
>>> Hawker
>>> *Sent:* Thursday, March 30, 2023 3:31 PM
>>> *To:* Greg Lipschitz <glipschitz at summitinternet.com.au>; Rob Thomas <
>>> xrobau at gmail.com>; <ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
>>> *Subject:* Re: [AusNOG] Critical 3CX Windows/Mac hack.
>>>
>>>
>>>
>>> It appears their sales team have no info regarding this. Just rang our
>>> Senior AM at 3CX and they've advised that they have no information, and
>>> that they are referring anyone who calls to their technical teams via
>>> support tickets in the 3CX portal.
>>>
>>>
>>>
>>> Not a good look for them.
>>>
>>>
>>>
>>> CH
>>>
>>>
>>>
>>> Get Outlook for Android <https://aka.ms/AAb9ysg>
>>> ------------------------------
>>>
>>> *From:* AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Greg
>>> Lipschitz <glipschitz at summitinternet.com.au>
>>> *Sent:* Thursday, March 30, 2023 3:09:45 PM
>>> *To:* Rob Thomas <xrobau at gmail.com>; <ausnog at lists.ausnog.net> <
>>> ausnog at lists.ausnog.net>
>>> *Subject:* Re: [AusNOG] Critical 3CX Windows/Mac hack.
>>>
>>>
>>>
>>> Here is a list of commands (or make a shell script) to stop it phoning
>>> home and getting more payload.
>>>
>>>
>>>
>>> # Disable 3CX Unattended-Upgrades Service
>>>
>>> systemctl stop unattended-upgrades
>>>
>>>
>>>
>>> # Collect the version of 3CX Desktop Apps on the Server
>>>
>>>
>>>
>>> cd /var/lib/3cxpbx/Instance1/Data/Http/electron
>>>
>>> ls -la * > /root/3cx-desktop-versions.log
>>>
>>>
>>>
>>> # Remove the files
>>>
>>>
>>>
>>> rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/osx/*.dmg
>>>
>>> rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/osx/*.zip
>>>
>>> rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/windows/*.msi
>>>
>>> rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/windows/*.nupkg
>>>
>>>
>>>
>>>
>>>
>>>
>>> https://www.3cx.com/community/threads/threat-alerts-from-sentinelone-for-desktop-update-initiated-from-desktop-client.119806/page-5
>>>
>>>
>>>
>>>
>>>
>>> Sadly, 3CX haven't even acknowledged this yet.
>>>
>>> It would seem that their whole CI-CD pipeline has been compromised
>>>
>>>
>>>
>>> Greg.
>>>
>>>
>>>
>>>
>>>
>>> *Greg Lipschitz**?*
>>>
>>>  |
>>>
>>> *Founder & CEO*
>>>
>>>  |
>>>
>>> *Summit Internet*
>>>
>>> *glipschitz at summitinternet.com.au* <glipschitz at summitinternet.com.au>
>>>
>>> *summitinternet.com.au* <http://summitinternet.com.au>
>>>
>>> *1300 049 749* <1300%20049%20749>
>>>
>>> *Unit 2, 31-39 Norcal Road, Nunawading VIC 3131*
>>> <https://www.google.com/maps?cid=12522583051503623677&_ga=2.149009334.1057584350.1554770858-1081443428.1554770858>
>>>
>>> [image: Summit Internet] <http://summitinternet.com.au/>
>>>
>>>
>>> ------------------------------
>>>
>>> *From:* AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Rob
>>> Thomas <xrobau at gmail.com>
>>> *Sent:* 30 March 2023 14:54
>>> *To:* <ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
>>> *Subject:* [AusNOG] Critical 3CX Windows/Mac hack.
>>>
>>>
>>>
>>> As no-one's mentioned it here yet, I just thought I'd bring up the
>>> zero-day, in the wild, active RIGHT NOW, trojan 3CX Windows and Mac apps.
>>>
>>>
>>>
>>> If you, or you have clients, running 3CX, make sure they ARE NOT using
>>> the app. If they are, their machines are probably already owned, and all
>>> their stored credentials and session cookies have been leaked.
>>>
>>>
>>>
>>>
>>> https://www.bleepingcomputer.com/news/security/hackers-compromise-3cx-desktop-app-in-a-supply-chain-attack/amp/
>>> <https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.bleepingcomputer.com%2Fnews%2Fsecurity%2Fhackers-compromise-3cx-desktop-app-in-a-supply-chain-attack%2Famp%2F&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C5134fed0ee3f4dbc894808db30d2a12f%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157453430051909%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UvNTww7E05nvQnaDQ25Qc8XytZFC%2FhIseT3MHYckCNM%3D&reserved=0>
>>>
>>>
>>>
>>> This is really bad. Sorry 8-(
>>>
>>>
>>>
>>> --Rob
>>>
>>>
>>> _______________________________________________
>>> AusNOG mailing list
>>> AusNOG at lists.ausnog.net
>>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>>
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
>
>
> --
> veg?e?tar?i?an:
> Ancient tribal slang for the village idiot who can't hunt, fish or ride
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/2f846e4b/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 984 bytes
Desc: not available
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/2f846e4b/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 10728 bytes
Desc: not available
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/2f846e4b/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 1930 bytes
Desc: not available
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/2f846e4b/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 3004 bytes
Desc: not available
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/2f846e4b/attachment-0007.png>

From matthew at htsol.com.au  Thu Mar 30 16:58:59 2023
From: matthew at htsol.com.au (Matthew Mace)
Date: Thu, 30 Mar 2023 05:58:59 +0000
Subject: [AusNOG] Critical 3CX Windows/Mac hack.
In-Reply-To: <MEYP282MB2597D4036A8C7496B8A2B292828E9@MEYP282MB2597.AUSP282.PROD.OUTLOOK.COM>
References: <CAC5zRjOB28=xU_qwGoNqdLq_ruqjQi19E1JG5Ls8fS-4fsznqw@mail.gmail.com>
 <MEYP282MB25976995206A1A35030FC3D4828E9@MEYP282MB2597.AUSP282.PROD.OUTLOOK.COM>
 <SYBP282MB0538C9BA5014A95976B5496BAF8E9@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM>
 <MEYP282MB252094D1B586FBB7580B401EDB8E9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <SY4P282MB351717C698442CD63C8BF0E6958E9@SY4P282MB3517.AUSP282.PROD.OUTLOOK.COM>
 <MEYP282MB2597D4036A8C7496B8A2B292828E9@MEYP282MB2597.AUSP282.PROD.OUTLOOK.COM>
Message-ID: <SY4P282MB3517E53142C0F936438119F0958E9@SY4P282MB3517.AUSP282.PROD.OUTLOOK.COM>

Interesting!

How long ago did it start seeing it and was It standard defender or Endpoint Business?



Matthew Mace
Director
Honest Technology Solutions
P: 07 3188 7244
E: matthew at htsol.com.au<mailto:matthew at htsol.com.au>

www.htsol.com.au<http://www.htsol.com.au/>
"Keeping IT Honest"

[cid:image001.png at 01D96320.88ED0BC0]


From: Greg Lipschitz <glipschitz at summitinternet.com.au>
Sent: Thursday, March 30, 2023 3:48 PM
To: Matthew Mace <matthew at htsol.com.au>; Nathan Brookfield <Nathan.Brookfield at iperium.com.au>; Christopher Hawker <chris at thesysadmin.dev>; Rob Thomas <xrobau at gmail.com>; <ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] Critical 3CX Windows/Mac hack.

Windows Defender picked it up too.

Greg Lipschitz?
 |
Founder & CEO
 |
Summit Internet
glipschitz at summitinternet.com.au<mailto:glipschitz at summitinternet.com.au>
summitinternet.com.au<http://summitinternet.com.au>
1300 049 749<tel:1300%20049%20749>
Unit 2, 31-39 Norcal Road, Nunawading VIC 3131<https://www.google.com/maps?cid=12522583051503623677&_ga=2.149009334.1057584350.1554770858-1081443428.1554770858>
[cid:image002.png at 01D96320.88ED0BC0]
[cid:image003.png at 01D96320.88ED0BC0]
[Summit Internet]<http://summitinternet.com.au/>
[cid:image005.png at 01D96320.88ED0BC0]

________________________________
From: Matthew Mace <matthew at htsol.com.au>
Sent: 30 March 2023 15:57
To: Nathan Brookfield <Nathan.Brookfield at iperium.com.au>; Christopher Hawker <chris at thesysadmin.dev>; Greg Lipschitz <glipschitz at summitinternet.com.au>; Rob Thomas <xrobau at gmail.com>; <ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
Subject: RE: [AusNOG] Critical 3CX Windows/Mac hack.

You don't often get email from matthew at htsol.com.au. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification>

Can anyone definitively confirm that they?ve personally seen it get picked up by anything else than S1?



In addition to  this anyone that has had it installed at a site and also run a premium DNS filtering service (Umbrella, DNS Filter etc.) and/or premium routers with DPI (Sonicwall, Firebox etc.), do you know if they picked up this traffic and stopped it? I would be hoping so.



Definitely curious to know either way.







Matthew Mace





From: AusNOG <ausnog-bounces at lists.ausnog.net> On Behalf Of Nathan Brookfield
Sent: Thursday, March 30, 2023 2:51 PM
To: Christopher Hawker <chris at thesysadmin.dev>; Greg Lipschitz <glipschitz at summitinternet.com.au>; Rob Thomas <xrobau at gmail.com>; <ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] Critical 3CX Windows/Mac hack.



To be fair, they likely don?t know much yet and things are probably pretty hectic?. Give them time, crisis management is probably only kicking in now.



From: AusNOG <ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>> On Behalf Of Christopher Hawker
Sent: Thursday, March 30, 2023 3:31 PM
To: Greg Lipschitz <glipschitz at summitinternet.com.au<mailto:glipschitz at summitinternet.com.au>>; Rob Thomas <xrobau at gmail.com<mailto:xrobau at gmail.com>>; <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>> <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: Re: [AusNOG] Critical 3CX Windows/Mac hack.



It appears their sales team have no info regarding this. Just rang our Senior AM at 3CX and they've advised that they have no information, and that they are referring anyone who calls to their technical teams via support tickets in the 3CX portal.



Not a good look for them.



CH



Get Outlook for Android<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C28ab046f470a4f19932b08db30db3490%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157490273890466%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=LeXZsLN4cUATWMtiKqKRMgdDICBmFCmTUpQ3wZ%2FBGK4%3D&reserved=0>

________________________________

From: AusNOG <ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>> on behalf of Greg Lipschitz <glipschitz at summitinternet.com.au<mailto:glipschitz at summitinternet.com.au>>
Sent: Thursday, March 30, 2023 3:09:45 PM
To: Rob Thomas <xrobau at gmail.com<mailto:xrobau at gmail.com>>; <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>> <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: Re: [AusNOG] Critical 3CX Windows/Mac hack.



Here is a list of commands (or make a shell script) to stop it phoning home and getting more payload.



# Disable 3CX Unattended-Upgrades Service

systemctl stop unattended-upgrades



# Collect the version of 3CX Desktop Apps on the Server



cd /var/lib/3cxpbx/Instance1/Data/Http/electron

ls -la * > /root/3cx-desktop-versions.log



# Remove the files



rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/osx/*.dmg

rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/osx/*.zip

rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/windows/*.msi

rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/windows/*.nupkg





https://www.3cx.com/community/threads/threat-alerts-from-sentinelone-for-desktop-update-initiated-from-desktop-client.119806/page-5<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.3cx.com%2Fcommunity%2Fthreads%2Fthreat-alerts-from-sentinelone-for-desktop-update-initiated-from-desktop-client.119806%2Fpage-5&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C28ab046f470a4f19932b08db30db3490%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157490273890466%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=gBvUcr9Pkokg10aeP864qVSsMZDg6KE%2FuuDUDi2imtE%3D&reserved=0>





Sadly, 3CX haven't even acknowledged this yet.

It would seem that their whole CI-CD pipeline has been compromised



Greg.





Greg Lipschitz?

 |

Founder & CEO

 |

Summit Internet

glipschitz at summitinternet.com.au<mailto:glipschitz at summitinternet.com.au>

summitinternet.com.au<https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsummitinternet.com.au%2F&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C28ab046f470a4f19932b08db30db3490%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157490273890466%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zO2mC4vUjaIW7YOuPf7i45hrukQ5fRzyFikE1GWDZjQ%3D&reserved=0>

1300 049 749<tel:1300%20049%20749>

Unit 2, 31-39 Norcal Road, Nunawading VIC 3131<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.google.com%2Fmaps%3Fcid%3D12522583051503623677%26_ga%3D2.149009334.1057584350.1554770858-1081443428.1554770858&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C28ab046f470a4f19932b08db30db3490%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157490273890466%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=VkgdWHriLjc1epWzbMAJ6U7ksoRzTvp6An1JpnbAt1k%3D&reserved=0>

[cid:image002.png at 01D96320.88ED0BC0]

[cid:image003.png at 01D96320.88ED0BC0]

[Summit Internet]<https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsummitinternet.com.au%2F&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C28ab046f470a4f19932b08db30db3490%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157490273890466%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zO2mC4vUjaIW7YOuPf7i45hrukQ5fRzyFikE1GWDZjQ%3D&reserved=0>

[cid:image005.png at 01D96320.88ED0BC0]



________________________________

From: AusNOG <ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>> on behalf of Rob Thomas <xrobau at gmail.com<mailto:xrobau at gmail.com>>
Sent: 30 March 2023 14:54
To: <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>> <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: [AusNOG] Critical 3CX Windows/Mac hack.



As no-one's mentioned it here yet, I just thought I'd bring up the zero-day, in the wild, active RIGHT NOW, trojan 3CX Windows and Mac apps.



If you, or you have clients, running 3CX, make sure they ARE NOT using the app. If they are, their machines are probably already owned, and all their stored credentials and session cookies have been leaked.



https://www.bleepingcomputer.com/news/security/hackers-compromise-3cx-desktop-app-in-a-supply-chain-attack/amp/<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.bleepingcomputer.com%2Fnews%2Fsecurity%2Fhackers-compromise-3cx-desktop-app-in-a-supply-chain-attack%2Famp%2F&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C28ab046f470a4f19932b08db30db3490%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157490273890466%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=NUiZuLaVjcCKlwNPMt1Np84joAu8dkLObF0L7rfce6s%3D&reserved=0>



This is really bad. Sorry 8-(



--Rob


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/de259af3/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 17808 bytes
Desc: image001.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/de259af3/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 984 bytes
Desc: image002.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/de259af3/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 10728 bytes
Desc: image003.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/de259af3/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 1930 bytes
Desc: image004.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/de259af3/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image005.png
Type: image/png
Size: 3004 bytes
Desc: image005.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/de259af3/attachment-0004.png>

From glipschitz at summitinternet.com.au  Thu Mar 30 17:04:09 2023
From: glipschitz at summitinternet.com.au (Greg Lipschitz)
Date: Thu, 30 Mar 2023 06:04:09 +0000
Subject: [AusNOG] Critical 3CX Windows/Mac hack.
In-Reply-To: <SY4P282MB3517E53142C0F936438119F0958E9@SY4P282MB3517.AUSP282.PROD.OUTLOOK.COM>
References: <CAC5zRjOB28=xU_qwGoNqdLq_ruqjQi19E1JG5Ls8fS-4fsznqw@mail.gmail.com>
 <MEYP282MB25976995206A1A35030FC3D4828E9@MEYP282MB2597.AUSP282.PROD.OUTLOOK.COM>
 <SYBP282MB0538C9BA5014A95976B5496BAF8E9@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM>
 <MEYP282MB252094D1B586FBB7580B401EDB8E9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <SY4P282MB351717C698442CD63C8BF0E6958E9@SY4P282MB3517.AUSP282.PROD.OUTLOOK.COM>
 <MEYP282MB2597D4036A8C7496B8A2B292828E9@MEYP282MB2597.AUSP282.PROD.OUTLOOK.COM>
 <SY4P282MB3517E53142C0F936438119F0958E9@SY4P282MB3517.AUSP282.PROD.OUTLOOK.COM>
Message-ID: <MEYP282MB2597CAC625ABC0E699F0E06D828E9@MEYP282MB2597.AUSP282.PROD.OUTLOOK.COM>

We have the paid whiz bang M365 version (you can tell I use Windows, right!?)
When we removed it and then reinstalled it, it grabbed it straight away.

Greg Lipschitz | Founder & CEO | Summit Internet
glipschitz at summitinternet.com.au
summitinternet.com.au
1300 049 749
Unit 2, 31-39 Norcal Road, Nunawading VIC 3131
Summit Internet
________________________________
From: Matthew Mace <matthew at htsol.com.au>
Sent: 30 March 2023 16:58
To: Greg Lipschitz <glipschitz at summitinternet.com.au>; Nathan Brookfield <Nathan.Brookfield at iperium.com.au>; Christopher Hawker <chris at thesysadmin.dev>; Rob Thomas <xrobau at gmail.com>; <ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
Subject: RE: [AusNOG] Critical 3CX Windows/Mac hack.

You don't often get email from matthew at htsol.com.au. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification>

Interesting!



How long ago did it start seeing it and was It standard defender or Endpoint Business?







Matthew Mace

Director

Honest Technology Solutions

P: 07 3188 7244

E: matthew at htsol.com.au<mailto:matthew at htsol.com.au>



www.htsol.com.au<https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.htsol.com.au%2F&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C3e53ff8877ba461ef70508db30e3dcce%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157527457979236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tvenO4p9FtY%2BA2pdX5DphxarqK7chYbN1Yhlrpm2Tg0%3D&reserved=0>

"Keeping IT Honest"



[cid:image001.png at 01D96320.88ED0BC0]





From: Greg Lipschitz <glipschitz at summitinternet.com.au>
Sent: Thursday, March 30, 2023 3:48 PM
To: Matthew Mace <matthew at htsol.com.au>; Nathan Brookfield <Nathan.Brookfield at iperium.com.au>; Christopher Hawker <chris at thesysadmin.dev>; Rob Thomas <xrobau at gmail.com>; <ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] Critical 3CX Windows/Mac hack.



Windows Defender picked it up too.



Greg Lipschitz?

 |

Founder & CEO

 |

Summit Internet

glipschitz at summitinternet.com.au<mailto:glipschitz at summitinternet.com.au>

summitinternet.com.au<https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsummitinternet.com.au%2F&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C3e53ff8877ba461ef70508db30e3dcce%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157527457979236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3EyM3Pd5pii4zx9Xm0aquhVe6cEiWS0sOOmaWXitziM%3D&reserved=0>

1300 049 749<tel:1300%20049%20749>

Unit 2, 31-39 Norcal Road, Nunawading VIC 3131<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.google.com%2Fmaps%3Fcid%3D12522583051503623677%26_ga%3D2.149009334.1057584350.1554770858-1081443428.1554770858&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C3e53ff8877ba461ef70508db30e3dcce%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157527457979236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=d2Lii8liwdK2oyNLUXSbR7C%2BaXm0TmW7dWqFByZyWpU%3D&reserved=0>

[cid:image002.png at 01D96320.88ED0BC0]

[cid:image003.png at 01D96320.88ED0BC0]

[Summit Internet]<https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsummitinternet.com.au%2F&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C3e53ff8877ba461ef70508db30e3dcce%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157527457979236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3EyM3Pd5pii4zx9Xm0aquhVe6cEiWS0sOOmaWXitziM%3D&reserved=0>

[cid:image005.png at 01D96320.88ED0BC0]



________________________________

From: Matthew Mace <matthew at htsol.com.au>
Sent: 30 March 2023 15:57
To: Nathan Brookfield <Nathan.Brookfield at iperium.com.au>; Christopher Hawker <chris at thesysadmin.dev>; Greg Lipschitz <glipschitz at summitinternet.com.au>; Rob Thomas <xrobau at gmail.com>; <ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
Subject: RE: [AusNOG] Critical 3CX Windows/Mac hack.



You don't often get email from matthew at htsol.com.au. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification>

Can anyone definitively confirm that they?ve personally seen it get picked up by anything else than S1?



In addition to  this anyone that has had it installed at a site and also run a premium DNS filtering service (Umbrella, DNS Filter etc.) and/or premium routers with DPI (Sonicwall, Firebox etc.), do you know if they picked up this traffic and stopped it? I would be hoping so.



Definitely curious to know either way.







Matthew Mace





From: AusNOG <ausnog-bounces at lists.ausnog.net> On Behalf Of Nathan Brookfield
Sent: Thursday, March 30, 2023 2:51 PM
To: Christopher Hawker <chris at thesysadmin.dev>; Greg Lipschitz <glipschitz at summitinternet.com.au>; Rob Thomas <xrobau at gmail.com>; <ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] Critical 3CX Windows/Mac hack.



To be fair, they likely don?t know much yet and things are probably pretty hectic?. Give them time, crisis management is probably only kicking in now.



From: AusNOG <ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>> On Behalf Of Christopher Hawker
Sent: Thursday, March 30, 2023 3:31 PM
To: Greg Lipschitz <glipschitz at summitinternet.com.au<mailto:glipschitz at summitinternet.com.au>>; Rob Thomas <xrobau at gmail.com<mailto:xrobau at gmail.com>>; <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>> <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: Re: [AusNOG] Critical 3CX Windows/Mac hack.



It appears their sales team have no info regarding this. Just rang our Senior AM at 3CX and they've advised that they have no information, and that they are referring anyone who calls to their technical teams via support tickets in the 3CX portal.



Not a good look for them.



CH



Get Outlook for Android<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C3e53ff8877ba461ef70508db30e3dcce%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157527457979236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=wx6B%2ByjV5nWfRSpFXl2H%2F6XN4%2BgsRcdgCkIB6o05M8M%3D&reserved=0>

________________________________

From: AusNOG <ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>> on behalf of Greg Lipschitz <glipschitz at summitinternet.com.au<mailto:glipschitz at summitinternet.com.au>>
Sent: Thursday, March 30, 2023 3:09:45 PM
To: Rob Thomas <xrobau at gmail.com<mailto:xrobau at gmail.com>>; <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>> <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: Re: [AusNOG] Critical 3CX Windows/Mac hack.



Here is a list of commands (or make a shell script) to stop it phoning home and getting more payload.



# Disable 3CX Unattended-Upgrades Service

systemctl stop unattended-upgrades



# Collect the version of 3CX Desktop Apps on the Server



cd /var/lib/3cxpbx/Instance1/Data/Http/electron

ls -la * > /root/3cx-desktop-versions.log



# Remove the files



rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/osx/*.dmg

rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/osx/*.zip

rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/windows/*.msi

rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/windows/*.nupkg





https://www.3cx.com/community/threads/threat-alerts-from-sentinelone-for-desktop-update-initiated-from-desktop-client.119806/page-5<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.3cx.com%2Fcommunity%2Fthreads%2Fthreat-alerts-from-sentinelone-for-desktop-update-initiated-from-desktop-client.119806%2Fpage-5&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C3e53ff8877ba461ef70508db30e3dcce%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157527457979236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=2PuQU0clb%2Fth3Wx3bPdig9M2GcB4pthDar2xMEg4lR4%3D&reserved=0>





Sadly, 3CX haven't even acknowledged this yet.

It would seem that their whole CI-CD pipeline has been compromised



Greg.





Greg Lipschitz?

 |

Founder & CEO

 |

Summit Internet

glipschitz at summitinternet.com.au<mailto:glipschitz at summitinternet.com.au>

summitinternet.com.au<https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsummitinternet.com.au%2F&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C3e53ff8877ba461ef70508db30e3dcce%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157527457979236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3EyM3Pd5pii4zx9Xm0aquhVe6cEiWS0sOOmaWXitziM%3D&reserved=0>

1300 049 749<tel:1300%20049%20749>

Unit 2, 31-39 Norcal Road, Nunawading VIC 3131<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.google.com%2Fmaps%3Fcid%3D12522583051503623677%26_ga%3D2.149009334.1057584350.1554770858-1081443428.1554770858&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C3e53ff8877ba461ef70508db30e3dcce%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157527457979236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=d2Lii8liwdK2oyNLUXSbR7C%2BaXm0TmW7dWqFByZyWpU%3D&reserved=0>

[cid:image002.png at 01D96320.88ED0BC0]

[cid:image003.png at 01D96320.88ED0BC0]

[Summit Internet]<https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsummitinternet.com.au%2F&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C3e53ff8877ba461ef70508db30e3dcce%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157527457979236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3EyM3Pd5pii4zx9Xm0aquhVe6cEiWS0sOOmaWXitziM%3D&reserved=0>

[cid:image005.png at 01D96320.88ED0BC0]



________________________________

From: AusNOG <ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>> on behalf of Rob Thomas <xrobau at gmail.com<mailto:xrobau at gmail.com>>
Sent: 30 March 2023 14:54
To: <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>> <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: [AusNOG] Critical 3CX Windows/Mac hack.



As no-one's mentioned it here yet, I just thought I'd bring up the zero-day, in the wild, active RIGHT NOW, trojan 3CX Windows and Mac apps.



If you, or you have clients, running 3CX, make sure they ARE NOT using the app. If they are, their machines are probably already owned, and all their stored credentials and session cookies have been leaked.



https://www.bleepingcomputer.com/news/security/hackers-compromise-3cx-desktop-app-in-a-supply-chain-attack/amp/<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.bleepingcomputer.com%2Fnews%2Fsecurity%2Fhackers-compromise-3cx-desktop-app-in-a-supply-chain-attack%2Famp%2F&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C3e53ff8877ba461ef70508db30e3dcce%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157527457979236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=0gosoVz1GPGxL9o%2BGxsa7TsWLhKWPuZqU6buGcj%2BO4U%3D&reserved=0>



This is really bad. Sorry 8-(



--Rob


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/1e159128/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 17808 bytes
Desc: image001.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/1e159128/attachment-0009.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 984 bytes
Desc: image002.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/1e159128/attachment-0010.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 10728 bytes
Desc: image003.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/1e159128/attachment-0011.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 1930 bytes
Desc: image004.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/1e159128/attachment-0012.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image005.png
Type: image/png
Size: 3004 bytes
Desc: image005.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/1e159128/attachment-0013.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image341512.png
Type: image/png
Size: 984 bytes
Desc: image341512.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/1e159128/attachment-0014.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image534930.png
Type: image/png
Size: 10728 bytes
Desc: image534930.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/1e159128/attachment-0015.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image634373.png
Type: image/png
Size: 1930 bytes
Desc: image634373.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/1e159128/attachment-0016.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image377621.png
Type: image/png
Size: 3004 bytes
Desc: image377621.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/1e159128/attachment-0017.png>

From ross at zrx.me  Thu Mar 30 18:00:33 2023
From: ross at zrx.me (Ross Fawcett)
Date: Thu, 30 Mar 2023 07:00:33 +0000
Subject: [AusNOG] Critical 3CX Windows/Mac hack.
In-Reply-To: <MEYP282MB2597CAC625ABC0E699F0E06D828E9@MEYP282MB2597.AUSP282.PROD.OUTLOOK.COM>
References: <CAC5zRjOB28=xU_qwGoNqdLq_ruqjQi19E1JG5Ls8fS-4fsznqw@mail.gmail.com>
 <MEYP282MB25976995206A1A35030FC3D4828E9@MEYP282MB2597.AUSP282.PROD.OUTLOOK.COM>
 <SYBP282MB0538C9BA5014A95976B5496BAF8E9@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM>
 <MEYP282MB252094D1B586FBB7580B401EDB8E9@MEYP282MB2520.AUSP282.PROD.OUTLOOK.COM>
 <SY4P282MB351717C698442CD63C8BF0E6958E9@SY4P282MB3517.AUSP282.PROD.OUTLOOK.COM>
 <MEYP282MB2597D4036A8C7496B8A2B292828E9@MEYP282MB2597.AUSP282.PROD.OUTLOOK.COM>
 <SY4P282MB3517E53142C0F936438119F0958E9@SY4P282MB3517.AUSP282.PROD.OUTLOOK.COM>
 <MEYP282MB2597CAC625ABC0E699F0E06D828E9@MEYP282MB2597.AUSP282.PROD.OUTLOOK.COM>
Message-ID: <SY4PR01MB65373F8CD212A2199F94C32BB18E9@SY4PR01MB6537.ausprd01.prod.outlook.com>

Their CEO has posted an update confirming it.
https://www.3cx.com/community/threads/3cx-desktopapp-security-alert.119951/

From: AusNOG <ausnog-bounces at lists.ausnog.net> On Behalf Of Greg Lipschitz
Sent: Thursday, March 30, 2023 2:04 PM
To: Matthew Mace <matthew at htsol.com.au>; Nathan Brookfield <Nathan.Brookfield at iperium.com.au>; Christopher Hawker <chris at thesysadmin.dev>; Rob Thomas <xrobau at gmail.com>; <ausnog at lists.ausnog.net> <ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] Critical 3CX Windows/Mac hack.

We have the paid whiz bang M365 version (you can tell I use Windows, right!?)
When we removed it and then reinstalled it, it grabbed it straight away.

Greg Lipschitz?
 |
Founder & CEO
 |
Summit Internet
glipschitz at summitinternet.com.au<mailto:glipschitz at summitinternet.com.au>
summitinternet.com.au<http://summitinternet.com.au>
1300 049 749<tel:1300%20049%20749>
Unit 2, 31-39 Norcal Road, Nunawading VIC 3131<https://www.google.com/maps?cid=12522583051503623677&_ga=2.149009334.1057584350.1554770858-1081443428.1554770858>
[cid:image002.png at 01D96318.5E590EC0]
[cid:image003.png at 01D96318.5E590EC0]
[Summit Internet]<http://summitinternet.com.au/>
[cid:image005.png at 01D96318.5E590EC0]

________________________________
From: Matthew Mace <matthew at htsol.com.au<mailto:matthew at htsol.com.au>>
Sent: 30 March 2023 16:58
To: Greg Lipschitz <glipschitz at summitinternet.com.au<mailto:glipschitz at summitinternet.com.au>>; Nathan Brookfield <Nathan.Brookfield at iperium.com.au<mailto:Nathan.Brookfield at iperium.com.au>>; Christopher Hawker <chris at thesysadmin.dev<mailto:chris at thesysadmin.dev>>; Rob Thomas <xrobau at gmail.com<mailto:xrobau at gmail.com>>; <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>> <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: RE: [AusNOG] Critical 3CX Windows/Mac hack.

You don't often get email from matthew at htsol.com.au<mailto:matthew at htsol.com.au>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification>

Interesting!



How long ago did it start seeing it and was It standard defender or Endpoint Business?







Matthew Mace

Director

Honest Technology Solutions

P: 07 3188 7244

E: matthew at htsol.com.au<mailto:matthew at htsol.com.au>



www.htsol.com.au<https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.htsol.com.au%2F&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C3e53ff8877ba461ef70508db30e3dcce%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157527457979236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tvenO4p9FtY%2BA2pdX5DphxarqK7chYbN1Yhlrpm2Tg0%3D&reserved=0>

"Keeping IT Honest"



[cid:image006.png at 01D96318.5E590EC0]





From: Greg Lipschitz <glipschitz at summitinternet.com.au<mailto:glipschitz at summitinternet.com.au>>
Sent: Thursday, March 30, 2023 3:48 PM
To: Matthew Mace <matthew at htsol.com.au<mailto:matthew at htsol.com.au>>; Nathan Brookfield <Nathan.Brookfield at iperium.com.au<mailto:Nathan.Brookfield at iperium.com.au>>; Christopher Hawker <chris at thesysadmin.dev<mailto:chris at thesysadmin.dev>>; Rob Thomas <xrobau at gmail.com<mailto:xrobau at gmail.com>>; <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>> <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: Re: [AusNOG] Critical 3CX Windows/Mac hack.



Windows Defender picked it up too.



Greg Lipschitz?

 |

Founder & CEO

 |

Summit Internet

glipschitz at summitinternet.com.au<mailto:glipschitz at summitinternet.com.au>

summitinternet.com.au<https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsummitinternet.com.au%2F&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C3e53ff8877ba461ef70508db30e3dcce%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157527457979236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3EyM3Pd5pii4zx9Xm0aquhVe6cEiWS0sOOmaWXitziM%3D&reserved=0>

1300 049 749<tel:1300%20049%20749>

Unit 2, 31-39 Norcal Road, Nunawading VIC 3131<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.google.com%2Fmaps%3Fcid%3D12522583051503623677%26_ga%3D2.149009334.1057584350.1554770858-1081443428.1554770858&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C3e53ff8877ba461ef70508db30e3dcce%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157527457979236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=d2Lii8liwdK2oyNLUXSbR7C%2BaXm0TmW7dWqFByZyWpU%3D&reserved=0>

[cid:image002.png at 01D96318.5E590EC0]

[cid:image003.png at 01D96318.5E590EC0]

[Summit Internet]<https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsummitinternet.com.au%2F&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C3e53ff8877ba461ef70508db30e3dcce%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157527457979236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3EyM3Pd5pii4zx9Xm0aquhVe6cEiWS0sOOmaWXitziM%3D&reserved=0>

[cid:image005.png at 01D96318.5E590EC0]



________________________________

From: Matthew Mace <matthew at htsol.com.au<mailto:matthew at htsol.com.au>>
Sent: 30 March 2023 15:57
To: Nathan Brookfield <Nathan.Brookfield at iperium.com.au<mailto:Nathan.Brookfield at iperium.com.au>>; Christopher Hawker <chris at thesysadmin.dev<mailto:chris at thesysadmin.dev>>; Greg Lipschitz <glipschitz at summitinternet.com.au<mailto:glipschitz at summitinternet.com.au>>; Rob Thomas <xrobau at gmail.com<mailto:xrobau at gmail.com>>; <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>> <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: RE: [AusNOG] Critical 3CX Windows/Mac hack.



You don't often get email from matthew at htsol.com.au<mailto:matthew at htsol.com.au>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification>

Can anyone definitively confirm that they?ve personally seen it get picked up by anything else than S1?



In addition to  this anyone that has had it installed at a site and also run a premium DNS filtering service (Umbrella, DNS Filter etc.) and/or premium routers with DPI (Sonicwall, Firebox etc.), do you know if they picked up this traffic and stopped it? I would be hoping so.



Definitely curious to know either way.







Matthew Mace





From: AusNOG <ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>> On Behalf Of Nathan Brookfield
Sent: Thursday, March 30, 2023 2:51 PM
To: Christopher Hawker <chris at thesysadmin.dev<mailto:chris at thesysadmin.dev>>; Greg Lipschitz <glipschitz at summitinternet.com.au<mailto:glipschitz at summitinternet.com.au>>; Rob Thomas <xrobau at gmail.com<mailto:xrobau at gmail.com>>; <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>> <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: Re: [AusNOG] Critical 3CX Windows/Mac hack.



To be fair, they likely don?t know much yet and things are probably pretty hectic?. Give them time, crisis management is probably only kicking in now.



From: AusNOG <ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>> On Behalf Of Christopher Hawker
Sent: Thursday, March 30, 2023 3:31 PM
To: Greg Lipschitz <glipschitz at summitinternet.com.au<mailto:glipschitz at summitinternet.com.au>>; Rob Thomas <xrobau at gmail.com<mailto:xrobau at gmail.com>>; <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>> <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: Re: [AusNOG] Critical 3CX Windows/Mac hack.



It appears their sales team have no info regarding this. Just rang our Senior AM at 3CX and they've advised that they have no information, and that they are referring anyone who calls to their technical teams via support tickets in the 3CX portal.



Not a good look for them.



CH



Get Outlook for Android<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C3e53ff8877ba461ef70508db30e3dcce%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157527457979236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=wx6B%2ByjV5nWfRSpFXl2H%2F6XN4%2BgsRcdgCkIB6o05M8M%3D&reserved=0>

________________________________

From: AusNOG <ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>> on behalf of Greg Lipschitz <glipschitz at summitinternet.com.au<mailto:glipschitz at summitinternet.com.au>>
Sent: Thursday, March 30, 2023 3:09:45 PM
To: Rob Thomas <xrobau at gmail.com<mailto:xrobau at gmail.com>>; <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>> <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: Re: [AusNOG] Critical 3CX Windows/Mac hack.



Here is a list of commands (or make a shell script) to stop it phoning home and getting more payload.



# Disable 3CX Unattended-Upgrades Service

systemctl stop unattended-upgrades



# Collect the version of 3CX Desktop Apps on the Server



cd /var/lib/3cxpbx/Instance1/Data/Http/electron

ls -la * > /root/3cx-desktop-versions.log



# Remove the files



rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/osx/*.dmg

rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/osx/*.zip

rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/windows/*.msi

rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/windows/*.nupkg





https://www.3cx.com/community/threads/threat-alerts-from-sentinelone-for-desktop-update-initiated-from-desktop-client.119806/page-5<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.3cx.com%2Fcommunity%2Fthreads%2Fthreat-alerts-from-sentinelone-for-desktop-update-initiated-from-desktop-client.119806%2Fpage-5&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C3e53ff8877ba461ef70508db30e3dcce%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157527457979236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=2PuQU0clb%2Fth3Wx3bPdig9M2GcB4pthDar2xMEg4lR4%3D&reserved=0>





Sadly, 3CX haven't even acknowledged this yet.

It would seem that their whole CI-CD pipeline has been compromised



Greg.





Greg Lipschitz?

 |

Founder & CEO

 |

Summit Internet

glipschitz at summitinternet.com.au<mailto:glipschitz at summitinternet.com.au>

summitinternet.com.au<https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsummitinternet.com.au%2F&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C3e53ff8877ba461ef70508db30e3dcce%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157527457979236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3EyM3Pd5pii4zx9Xm0aquhVe6cEiWS0sOOmaWXitziM%3D&reserved=0>

1300 049 749<tel:1300%20049%20749>

Unit 2, 31-39 Norcal Road, Nunawading VIC 3131<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.google.com%2Fmaps%3Fcid%3D12522583051503623677%26_ga%3D2.149009334.1057584350.1554770858-1081443428.1554770858&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C3e53ff8877ba461ef70508db30e3dcce%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157527457979236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=d2Lii8liwdK2oyNLUXSbR7C%2BaXm0TmW7dWqFByZyWpU%3D&reserved=0>

[cid:image002.png at 01D96318.5E590EC0]

[cid:image003.png at 01D96318.5E590EC0]

[Summit Internet]<https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsummitinternet.com.au%2F&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C3e53ff8877ba461ef70508db30e3dcce%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157527457979236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3EyM3Pd5pii4zx9Xm0aquhVe6cEiWS0sOOmaWXitziM%3D&reserved=0>

[cid:image005.png at 01D96318.5E590EC0]



________________________________

From: AusNOG <ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>> on behalf of Rob Thomas <xrobau at gmail.com<mailto:xrobau at gmail.com>>
Sent: 30 March 2023 14:54
To: <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>> <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: [AusNOG] Critical 3CX Windows/Mac hack.



As no-one's mentioned it here yet, I just thought I'd bring up the zero-day, in the wild, active RIGHT NOW, trojan 3CX Windows and Mac apps.



If you, or you have clients, running 3CX, make sure they ARE NOT using the app. If they are, their machines are probably already owned, and all their stored credentials and session cookies have been leaked.



https://www.bleepingcomputer.com/news/security/hackers-compromise-3cx-desktop-app-in-a-supply-chain-attack/amp/<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.bleepingcomputer.com%2Fnews%2Fsecurity%2Fhackers-compromise-3cx-desktop-app-in-a-supply-chain-attack%2Famp%2F&data=05%7C01%7Cglipschitz%40summitinternet.com.au%7C3e53ff8877ba461ef70508db30e3dcce%7C0838a12f226e43dfa6e4bb63d2643a7e%7C0%7C0%7C638157527457979236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=0gosoVz1GPGxL9o%2BGxsa7TsWLhKWPuZqU6buGcj%2BO4U%3D&reserved=0>



This is really bad. Sorry 8-(



--Rob


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/2097b055/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 984 bytes
Desc: image002.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/2097b055/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 10728 bytes
Desc: image003.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/2097b055/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 1930 bytes
Desc: image004.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/2097b055/attachment-0007.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image005.png
Type: image/png
Size: 3004 bytes
Desc: image005.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/2097b055/attachment-0008.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image006.png
Type: image/png
Size: 17808 bytes
Desc: image006.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230330/2097b055/attachment-0009.png>

From matt.g.walker at outlook.com  Fri Mar 31 12:30:54 2023
From: matt.g.walker at outlook.com (Matt Walker)
Date: Fri, 31 Mar 2023 01:30:54 +0000
Subject: [AusNOG] WebCentral Contact
Message-ID: <SL2P216MB1216441DEB477A92663F1C38B18F9@SL2P216MB1216.KORP216.PROD.OUTLOOK.COM>

Hi Noggers,

Have been struggling with WebCentral DNS for the last 3 weeks where we (like a lot of people) publish a CNAME for autodiscover.domain to autodiscover.outlook.com

With WebCentral something is a miss as it is not resolving correctly, I have tried escalating but this doesn?t seem to be getting traction

If anyone on the list is part of WebCentral or knows someone who could help, please contact me off list and I will be incredibly grateful 

Kind Regards
Matt Walker
Peninsula Health

From mitchkelly24 at gmail.com  Fri Mar 31 13:23:02 2023
From: mitchkelly24 at gmail.com (Mitch Kelly)
Date: Fri, 31 Mar 2023 10:23:02 +0800
Subject: [AusNOG] WebCentral Contact
In-Reply-To: <SL2P216MB1216441DEB477A92663F1C38B18F9@SL2P216MB1216.KORP216.PROD.OUTLOOK.COM>
References: <SL2P216MB1216441DEB477A92663F1C38B18F9@SL2P216MB1216.KORP216.PROD.OUTLOOK.COM>
Message-ID: <CAKZcER6aM3oTEk8pZ5_pKShb=kwZYpeavpMZE3oXA1jJWWSd4w@mail.gmail.com>

Hi Matt,

I often have issues with this issue, Are you finding it not working, Or
unable to be added?

On Fri, Mar 31, 2023 at 9:31?AM Matt Walker <matt.g.walker at outlook.com>
wrote:

> Hi Noggers,
>
> Have been struggling with WebCentral DNS for the last 3 weeks where we
> (like a lot of people) publish a CNAME for autodiscover.domain to
> autodiscover.outlook.com
>
> With WebCentral something is a miss as it is not resolving correctly, I
> have tried escalating but this doesn?t seem to be getting traction
>
> If anyone on the list is part of WebCentral or knows someone who could
> help, please contact me off list and I will be incredibly grateful
>
> Kind Regards
> Matt Walker
> Peninsula Health
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230331/9fad3f9e/attachment.htm>

From matt.g.walker at outlook.com  Fri Mar 31 13:42:10 2023
From: matt.g.walker at outlook.com (Matt Walker)
Date: Fri, 31 Mar 2023 02:42:10 +0000
Subject: [AusNOG] WebCentral Contact
In-Reply-To: <CAKZcER6aM3oTEk8pZ5_pKShb=kwZYpeavpMZE3oXA1jJWWSd4w@mail.gmail.com>
References: <SL2P216MB1216441DEB477A92663F1C38B18F9@SL2P216MB1216.KORP216.PROD.OUTLOOK.COM>
 <CAKZcER6aM3oTEk8pZ5_pKShb=kwZYpeavpMZE3oXA1jJWWSd4w@mail.gmail.com>
Message-ID: <SL2P216MB12164D20083CA01B2A3D01BCB18F9@SL2P216MB1216.KORP216.PROD.OUTLOOK.COM>

Gday Mitch,

It has been added (and re-added as a test) through the DNS Manager, however external users are unable to resolve and as such many services are not working as expected

Someone has contacted me off list also, so crossing fingers we can get this resolved quickly

Kind Regards
Matt

On 31 Mar 2023, at 13:23, Mitch Kelly <mitchkelly24 at gmail.com> wrote:

?
Hi Matt,

I often have issues with this issue, Are you finding it not working, Or unable to be added?

On Fri, Mar 31, 2023 at 9:31?AM Matt Walker <matt.g.walker at outlook.com<mailto:matt.g.walker at outlook.com>> wrote:
Hi Noggers,

Have been struggling with WebCentral DNS for the last 3 weeks where we (like a lot of people) publish a CNAME for autodiscover.domain to autodiscover.outlook.com<http://autodiscover.outlook.com>

With WebCentral something is a miss as it is not resolving correctly, I have tried escalating but this doesn?t seem to be getting traction

If anyone on the list is part of WebCentral or knows someone who could help, please contact me off list and I will be incredibly grateful

Kind Regards
Matt Walker
Peninsula Health
_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
https://lists.ausnog.net/mailman/listinfo/ausnog
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230331/ff2225a9/attachment.htm>

From m.enger at xi.com.au  Fri Mar 31 16:07:32 2023
From: m.enger at xi.com.au (Matthew Enger)
Date: Fri, 31 Mar 2023 05:07:32 +0000
Subject: [AusNOG] Coles.com.au contact
Message-ID: <SY4PR01MB70579C8FAFBE5DC89E98A84CA68F9@SY4PR01MB7057.ausprd01.prod.outlook.com>

Hello,

Does anyone have a contact for coles.com.au who can help with their website and getting the below when connecting via IPv6 (works on other ipv6 sources):

You are not authorized to view this page.
Firewall Rule:NotAllowed

I have tried the contacting their support team but they are struggling to grasp the issue. Even phoned them, no luck.





[signature_4013076670]
MATTHEW ENGER | Managing Director
T 1300 789 299 D 03 9909 3104 M 0406 532 792
m.enger at xi.com.au<mailto:m.enger at xi.com.au> | www.xi.com.au<http://www.xi.com.au> | Book an appointment<https://calendly.com/m-enger/30min>

This message is confidential. It may also be privileged or otherwise protected by work product immunity or other legal rules. If you have received it by mistake, please let us know by e-mail reply and delete it from your system; you may not copy this message or disclose its contents to anyone. Please send us by fax any message containing deadlines as incoming e-mails are not screened for response deadlines. The integrity and security of this message cannot be guaranteed on the Internet.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230331/cfd2e519/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 7645 bytes
Desc: image001.png
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230331/cfd2e519/attachment.png>

From jenn at jenn.id.au  Fri Mar 31 16:13:21 2023
From: jenn at jenn.id.au (Jennifer Sims)
Date: Fri, 31 Mar 2023 16:13:21 +1100
Subject: [AusNOG] Coles.com.au contact
In-Reply-To: <SY4PR01MB70579C8FAFBE5DC89E98A84CA68F9@SY4PR01MB7057.ausprd01.prod.outlook.com>
References: <SY4PR01MB70579C8FAFBE5DC89E98A84CA68F9@SY4PR01MB7057.ausprd01.prod.outlook.com>
Message-ID: <CA+k2KW1AsQUWp5o6T-9WG1YiSyt-mnFC+GuQo_r84jY7R6Qrrg@mail.gmail.com>

I reached out to a contact of mine, they were struggling to identify the
right team (they aren't in direct IT) as we proved with their ABB IPv6 that
the coles site was blocking them too. But the minute we disabled IPv6, the
site was working normally.

I verified the same problem as well.

And also hit the same struggles trying to get it to the right team
internally from their outside customer support teams.

Jen

On Fri, Mar 31, 2023 at 4:08?PM Matthew Enger <m.enger at xi.com.au> wrote:

> Hello,
>
>
>
> Does anyone have a contact for coles.com.au who can help with their
> website and getting the below when connecting via IPv6 (works on other ipv6
> sources):
>
> You are not authorized to view this page.
>
> Firewall Rule:NotAllowed
>
>
>
> I have tried the contacting their support team but they are struggling to
> grasp the issue. Even phoned them, no luck.
>
>
>
>
>
>
>
>
>
>
>
> [image: signature_4013076670]
>
> *MATTHEW ENGER* | Managing Director
> *T *1300 789 299 *D *03 9909 3104 *M *0406 532 792
> m.enger at xi.com.au | www.xi.com.au | Book an appointment
> <https://calendly.com/m-enger/30min>
>
>
>
> This message is confidential. It may also be privileged or otherwise
> protected by work product immunity or other legal rules. If you have
> received it by mistake, please let us know by e-mail reply and delete it
> from your system; you may not copy this message or disclose its contents to
> anyone. Please send us by fax any message containing deadlines as incoming
> e-mails are not screened for response deadlines. The integrity and security
> of this message cannot be guaranteed on the Internet.
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230331/4aedaa19/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 7645 bytes
Desc: not available
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230331/4aedaa19/attachment.png>

From luke.t at tncrew.com.au  Fri Mar 31 16:19:15 2023
From: luke.t at tncrew.com.au (Luke Thompson)
Date: Fri, 31 Mar 2023 16:19:15 +1100
Subject: [AusNOG] Coles.com.au contact
In-Reply-To: <CA+k2KW1AsQUWp5o6T-9WG1YiSyt-mnFC+GuQo_r84jY7R6Qrrg@mail.gmail.com>
References: <SY4PR01MB70579C8FAFBE5DC89E98A84CA68F9@SY4PR01MB7057.ausprd01.prod.outlook.com>
 <CA+k2KW1AsQUWp5o6T-9WG1YiSyt-mnFC+GuQo_r84jY7R6Qrrg@mail.gmail.com>
Message-ID: <e3b79889-3e9c-39c1-99dd-c1c1dfe7aa54@tncrew.com.au>

You might have joy with the Head of Digital Engineering:

Steven Corral <steven.corral at coles.com.au>

Just a hunch going off the Whois records. Registrant is DomainAdmin@, 
though name against it is his.

The above is confirmed to receive OK moments ago.

Good luck,
Luke

On 31/3/2023 4:13 pm, Jennifer Sims wrote:
> I reached out to a contact of mine, they were struggling to identify 
> the right team (they aren't in direct IT) as we proved with their ABB 
> IPv6 that the coles site was blocking them too. But the minute we 
> disabled IPv6, the site was working normally.
>
> I verified the same problem as well.
>
> And also hit the same struggles trying to get it to the right team 
> internally from their outside customer support teams.
>
> Jen
>
> On Fri, Mar 31, 2023 at 4:08?PM Matthew Enger <m.enger at xi.com.au> wrote:
>
>     Hello,
>
>     Does anyone have a contact for coles.com.au <http://coles.com.au>
>     who can help with their website and getting the below when
>     connecting via IPv6 (works on other ipv6 sources):
>
>     You are not authorized to view this page.
>
>     Firewall Rule:NotAllowed
>
>     I have tried the contacting their support team but they are
>     struggling to grasp the issue. Even phoned them, no luck.
>
>     	
>
>     signature_4013076670
>
>     	
>
>     *MATTHEW ENGER*?| Managing Director
>     *T *1300 789 299 *D *03 9909 3104 *M *0406 532 792
>     m.enger at xi.com.au <mailto:m.enger at xi.com.au>?| www.xi.com.au
>     <http://www.xi.com.au> | Book an appointment
>     <https://calendly.com/m-enger/30min>
>
>     	
>
>     This message is confidential. It may also be privileged or
>     otherwise protected by work product immunity or other legal rules.
>     If you have received it by mistake, please let us know by e-mail
>     reply and delete it from your system; you may not copy this
>     message or disclose its contents to anyone. Please send us by fax
>     any message containing deadlines as incoming e-mails are not
>     screened for response deadlines. The integrity and security of
>     this message cannot be guaranteed on the Internet.
>
>
>     	
>     	
>
>     _______________________________________________
>     AusNOG mailing list
>     AusNOG at lists.ausnog.net
>     https://lists.ausnog.net/mailman/listinfo/ausnog
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230331/a26d3d26/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 7645 bytes
Desc: not available
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230331/a26d3d26/attachment.png>

From luke.t at tncrew.com.au  Fri Mar 31 16:55:29 2023
From: luke.t at tncrew.com.au (Luke Thompson)
Date: Fri, 31 Mar 2023 16:55:29 +1100
Subject: [AusNOG] Microsoft 365 - Junk Policy (last 2 weeks)
Message-ID: <4d3620db-7699-7288-9b01-660c30eb922e@tncrew.com.au>

Hi all,

Strange question though curious if anyone's seen the same.

We've had a range of clients reporting very intense junking behaviour 
with M365, whether sending or receiving, if there's a M365 tenant 
involved it seems to be hitting Junk for reasons that aren't clear. This 
is with SPF/DKIM/DMARC/etc all passing.

I'm wondering if potentially a new policy has shipped which is 
over-reaching somewhat, as no bounce-backs are being received so it's a 
case of delivered-but-not. This is across a range of clients and 
clients' clients.

If we'd only heard of this from a single client we'd not think much of 
it, though this is both ways and across a range of tenants. Would be 
good to know if there's any substance to it.

As we don't run M365 nor have responsibility for impacted tenants, the 
request here is to check with the IT community. Thank you.

Cheers,
Luke


From djr at pdconsec.net  Fri Mar 31 18:10:26 2023
From: djr at pdconsec.net (David Rawling)
Date: Fri, 31 Mar 2023 18:10:26 +1100
Subject: [AusNOG] Microsoft 365 - Junk Policy (last 2 weeks)
In-Reply-To: <4d3620db-7699-7288-9b01-660c30eb922e@tncrew.com.au>
References: <4d3620db-7699-7288-9b01-660c30eb922e@tncrew.com.au>
Message-ID: <64beea24fced00028abd230d558ab5c70972eff2.camel@pdconsec.net>

Hi Luke

I can confirm that we've been seeing quite a lot of this across a
number of Office 365 tenants; both our own and several we manage (I'll
note that I am subscribed to AusNOG from my personal account not the
affected tenant).

As we're one of the affected tenants, we've been able to dig quite
deeply into the behaviour - in our case, it seems to be that almost all
"new" messages in a thread are successfully delivered to us, but once
it becomes a reply-fest it gets junked. There's some thought that it
could be related to one or more links in our signature - not that it's
a particularly large or "spammy" signature, though. No SPF, DMARC or
DKIM failures here. No obvious reasons it would be marked junk - the
SCL is below our threshold, etc.

What's frustrating is that so far, it's only affecting some people in
our tenant. I've not had the problem with messages I send and receive,
but there are others where it's 90% failure. I do have a different
signature file - very different formatting though it looks the same to
the recipient. I was thinking that might be related, but no dice so
far.

So you're not alone but we have absolutely no idea what's breaking.

Dave.
-- 
David Rawling - Principal Consultant
PD Consulting and Security

t: +61 41 213 5513? |? e:?djr at pdconsec.net

Please note that whilst we take all care, neither PD Consulting and
Security nor the sender accepts any responsibility for viruses and it
is your responsibility to scan for viruses. The contents are intended
only for use by the addressee and may contain confidential and/or
privileged material. If you received this in error, we request that you
please inform the sender and/or addressee immediately and delete the
material.

On Fri, 2023-03-31 at 16:55 +1100, Luke Thompson wrote:
> Hi all,
> 
> Strange question though curious if anyone's seen the same.
> 
> We've had a range of clients reporting very intense junking behaviour
> with M365, whether sending or receiving, if there's a M365 tenant 
> involved it seems to be hitting Junk for reasons that aren't clear.
> This 
> is with SPF/DKIM/DMARC/etc all passing.
> 
> I'm wondering if potentially a new policy has shipped which is 
> over-reaching somewhat, as no bounce-backs are being received so it's
> a 
> case of delivered-but-not. This is across a range of clients and 
> clients' clients.
> 
> If we'd only heard of this from a single client we'd not think much
> of 
> it, though this is both ways and across a range of tenants. Would be 
> good to know if there's any substance to it.
> 
> As we don't run M365 nor have responsibility for impacted tenants,
> the 
> request here is to check with the IT community. Thank you.
> 
> Cheers,
> Luke
> 
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20230331/6d8012d9/attachment.htm>