[AusNOG] NBN FTTC services and power hits

Tue Jan 4 01:29:38 AEDT 2022

FTTC NCDs and DPUs are known to be pretty weak and die easily during storms. 
A. Ensure your legacy phone line was cut out of the CIU in the pit/pole - and also the other three neighbors connected to the DPU. 
B. Put your NCD and phone line on a surge protector. 
C. Try and get the more hardy Adtrans version of the NCD as they are/were meant to replace the Netcomms as the thousands of them get replaced every-time there is a storm. 
D. Get a spare/backup NCD. 
E. If budget allows, get on an eSLA via your provider - so you get priority techs.
On 3/01/2022 8:56:55 PM, ausnog-request at lists.ausnog.net <ausnog-request at lists.ausnog.net> wrote:
Send AusNOG mailing list submissions to
ausnog at lists.ausnog.net

To subscribe or unsubscribe via the World Wide Web, visit
http://lists.ausnog.net/mailman/listinfo/ausnog
or, via email, send a message with subject or body 'help' to
ausnog-request at lists.ausnog.net

You can reach the person managing the list at
ausnog-owner at lists.ausnog.net

When replying, please edit your Subject line so it is more specific
than "Re: Contents of AusNOG digest..."

Today's Topics:

1. On Prem Exchange issues - y2k22 bug. (Mark Dignam)
2. Re: On Prem Exchange issues - y2k22 bug. (DaZZa)
3. Telstra NBN Outage NSW (Simon Pearce)
4. NBN FTTC services and power hits (Chris Chaundy)
5. Re: NBN FTTC services and power hits (Julien Goodwin)
6. Re: NBN FTTC services and power hits (Andy Blume)
7. Re: NBN FTTC services and power hits (Greg Lipschitz)
8. Re: NBN FTTC services and power hits (Chris Chaundy)

----------------------------------------------------------------------

Message: 1
Date: Sat, 1 Jan 2022 15:20:25 +0800
From: "Mark Dignam"
To:
Subject: [AusNOG] On Prem Exchange issues - y2k22 bug.
Message-ID: <000b01d7fee0$0bb61250$232236f0$@innaloo.net>
Content-Type: text/plain; charset="us-ascii"

People.

There appears to a y2k22 bug in Exchange servers - and its fairly nasty. As
of midnight UTC, the Filtering Service stopped passing emails, email goes in
- and never gets delivered to the mailbox. Easiest thing to look for in the
event logs:

The FIP-FS Filtering Management Service was unable to acquire a scanner
within the specified timeout.

or

The FIP-FS "Microsoft" Scan Engine failed to load. PID: 12348, Error Code:
0x80004005. Error Description: Can't convert "2201010001" to long.

Office365 appears NOT affected at this stage. (as expected, with MS's
current "move to our cloud" strategy)

https://twitter.com/JRoosen/status/1477120097747677184

Quick fix for a 2016 system, open a Exchange Management Power Shell screen.

cd $exscripts

.\Disable-AntimalwareScanning.ps1

Restart-Service MSExchangeTRansport

And anything waiting in the submission will get delivered in the next few
mins.

-------------- next part --------------
An HTML attachment was scrubbed...
URL:

------------------------------

Message: 2
Date: Sat, 1 Jan 2022 18:52:32 +1100
From: DaZZa
To: Mark Dignam
Cc: AusNOG Mailing List
Subject: Re: [AusNOG] On Prem Exchange issues - y2k22 bug.
Message-ID:

Content-Type: text/plain; charset="utf-8"

Probably related Reddit thread

https://www.reddit.com/r/sysadmin/comments/rt91z6/exchange_2019_antimalware_bad_update

Appears to be related to the long signed integer value for version numbers
allowing values only up to 2,147,483,647 - so when version numbers for the
malware/inline virus scanner were in 2021 - the first two numbers were "21"
and it worked.

Now it's 2022 - the first two numbers are "22", and it b0rks the update

It's almost comical that a company as big as Microsoft - especially after
the hype around Y2K bugs all those years ago - has missed this so blatantly
- but I suppose the company that we should be used to that by now.

DaZZa

On Sat, 1 Jan 2022, 6:20 pm Mark Dignam, wrote:

> People.
>
>
>
> There appears to a y2k22 bug in Exchange servers - and its fairly nasty.
> As of midnight UTC, the Filtering Service stopped passing emails, email
> goes in - and never gets delivered to the mailbox. Easiest thing to look
> for in the event logs:
>
> *The FIP-FS Filtering Management Service was unable to acquire a scanner
> within the specified timeout.*
>
> or
>
> *The FIP-FS "Microsoft" Scan Engine failed to load. PID: 12348, Error
> Code: 0x80004005. Error Description: Can't convert "2201010001" to long.*
>
>
>
> Office365 appears NOT affected at this stage. (as expected, with MS's
> current "move to our cloud" strategy)
>
> https://twitter.com/JRoosen/status/1477120097747677184
>
>
>
> Quick fix for a 2016 system, open a Exchange Management Power Shell
> screen?
>
>
>
> cd $exscripts
>
> .\Disable-AntimalwareScanning.ps1
>
> Restart-Service MSExchangeTRansport
>
>
>
> And anything waiting in the submission will get delivered in the next few
> mins.
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:

------------------------------

Message: 3
Date: Sun, 2 Jan 2022 18:01:23 +1100
From: Simon Pearce
To: ausnog at lists.ausnog.net
Subject: [AusNOG] Telstra NBN Outage NSW
Message-ID:

Content-Type: text/plain; charset="utf-8"

Hi all

Apologies for the noise. We saw multiple Telstra backed NBN services go
offline in NSW and ACT around an hour ago. The Call centres seem backed up
with long wait times.

Cheers
Simon

AS146963
-------------- next part --------------
An HTML attachment was scrubbed...
URL:

------------------------------

Message: 4
Date: Mon, 3 Jan 2022 09:26:00 +1100
From: Chris Chaundy
To: "ausnog at ausnog.net"
Subject: [AusNOG] NBN FTTC services and power hits
Message-ID:
Content-Type: text/plain; charset=utf-8

Is there anyone from NBNco on this list willing to discuss this with me?

It seems that every time there is a power hit in our locality (which with Ausnet is pretty regularly), the FTTC services die and don?t come back until you log a fault through the provider (in my case, Telstra) and then wait for them to go through the hoops of dealing with NBNco.

This has happened four times in the last couple of months (thankfully I have 4G backup). There must be some aggregator kit that has no power backup or cannot recover from power outages - pretty crap really. :-|

Sorry if this is a little OT for the list but I?d be interested if other FTTC areas see the same problem.

Sent from my iPhone

------------------------------

Message: 5
Date: Mon, 3 Jan 2022 18:13:04 +1100
From: Julien Goodwin
To: "ausnog at ausnog.net"
Subject: Re: [AusNOG] NBN FTTC services and power hits
Message-ID:
Content-Type: text/plain; charset=UTF-8; format=flowed

On 3/1/22 9:26 am, Chris Chaundy wrote:
> It seems that every time there is a power hit in our locality (which with Ausnet is pretty regularly), the FTTC services die and don?t come back until you log a fault through the provider (in my case, Telstra) and then wait for them to go through the hoops of dealing with NBNco.

For *my* FTTC I've had occasional cases where an NTD reboot was needed
(~1/year), however much more frequent appears to be cases where one of
my neighbours bounce their NTD (for whatever reason) the micro-DSLAM
seems to reboot for a multi-minute outage.

I'm seriously right about at the point where I might put in an order to
get someone else to dig fibre in[1] since there's enough evidence that
NBN simply choose not to care about multiple multi-minute outages a day
so I've not even bothered trying to call a fault on it.

1: Hi sales people, no I'm not interested in talking to you, I know who
has fibre near me, I'll reach out to contacts I already have if I end up
deciding to try.

------------------------------

Message: 6
Date: Mon, 3 Jan 2022 08:13:42 +0000
From: Andy Blume
Cc: "ausnog at ausnog.net"
Subject: Re: [AusNOG] NBN FTTC services and power hits
Message-ID: <6FA9015A-CB3C-40F0-AF59-0F5637B3FF94 at andyblu.me>
Content-Type: text/plain; charset="utf-8"

On 3 Jan 2022, at 09:26, Chris Chaundy <>> wrote:

It seems that every time there is a power hit in our locality (which with Ausnet is pretty regularly), the FTTC services die and don?t come back until you log a fault through the provider

Happened twice here; first in July and second just two weeks ago. Both involved a call to Aussie Broadband to organise an NBN tech to come out. Thankfully in both cases someone was able to come out very quickly, however I am concerned that every time there's a power outage or interruption this is going to be the standard procedure.

-------------- next part --------------
An HTML attachment was scrubbed...
URL:

------------------------------

Message: 7
Date: Mon, 3 Jan 2022 08:54:58 +0000
From: Greg Lipschitz
To: Chris Chaundy , "ausnog at ausnog.net"

Subject: Re: [AusNOG] NBN FTTC services and power hits
Message-ID:

Content-Type: text/plain; charset="utf-8"

Hi Chris,

The nbn DPU (the "mini-vdsl/gfast dslam" which lives in the pit) is reverse powered (about 6-14W) from the NCD (the nbn box inside your house). nbn use a mix of Netcomm and more recently Nokia DPU's. The DPU only needs 1 customer NCD connected to it to supply the power.

The DPU can connect up to 4 premises and the interesting thing is that any of the RSP's who have a customer connected to the DPU can initiate a DPU reset via the nbn API.

So, you might be with Telstra, and your neighbours with Aussie Broadband, TPG and Optus and any one of them can essentially cause your service to drop momentarily if they contact their RSP who initiates the DPU reset as part of the troubleshooting.

We've found that the DPU's often suffer the same fate as the old ADSL copper services when the pit fills up with water, the DPU goes off to la-la land (who would have thunk power and water didn't mix!) or the copper between the DPU and the customer gets corroded and needs to be cleaned up or replaced.

I've not heard of power issues causing the DPU to have issues. It sounds like the DPU is perhaps losing its config when there's a power interruption (we've certainly seen that with the Telstra EA MRV OS904 NTU's over the years).

If you're interested how it all comes together, nbn publish their design docs - https://www.nbnco.com.au/content/dam/nbn/documents/sell/sau/network-design-rules-20210630.pdf

Cheers
Greg

________________________________

Greg Lipschitz | Founder & CEO | Summit Internet
glipschitz at summitinternet.com.au
summitinternet.com.au
1300 049 749
Unit 2, 31-39 Norcal Road, Nunawading VIC 3131
Summit Internet
From: AusNOG on behalf of Chris Chaundy
Sent: 03 January 2022 09:26
To: ausnog at ausnog.net
Subject: [AusNOG] NBN FTTC services and power hits

Is there anyone from NBNco on this list willing to discuss this with me?

It seems that every time there is a power hit in our locality (which with Ausnet is pretty regularly), the FTTC services die and don?t come back until you log a fault through the provider (in my case, Telstra) and then wait for them to go through the hoops of dealing with NBNco.

This has happened four times in the last couple of months (thankfully I have 4G backup). There must be some aggregator kit that has no power backup or cannot recover from power outages - pretty crap really. :-|

Sorry if this is a little OT for the list but I?d be interested if other FTTC areas see the same problem.

Sent from my iPhone
_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ausnog.net%2Fmailman%2Flistinfo%2Fausnog&data=04%7C01%7Cglipschitz%40summitinternet.com.au%7C7d3675defb8a4be9a6ab08d9ce3ef1a0%7C0838a12f226e43dfa6e4bb63d2643a7e%7C1%7C0%7C637767592012923616%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=eaYK8fpeVKh%2FCKcTW5KgKBQOB32TYug2Q5GLkCN7xLo%3D&reserved=0
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image793377.png
Type: image/png
Size: 984 bytes
Desc: image793377.png
URL:
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image192808.png
Type: image/png
Size: 10728 bytes
Desc: image192808.png
URL:
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image042550.png
Type: image/png
Size: 1930 bytes
Desc: image042550.png
URL:
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image612353.png
Type: image/png
Size: 3004 bytes
Desc: image612353.png
URL:

------------------------------

Message: 8
Date: Mon, 3 Jan 2022 20:55:38 +1100
From: Chris Chaundy
To: Greg Lipschitz
Cc: "ausnog at ausnog.net"
Subject: Re: [AusNOG] NBN FTTC services and power hits
Message-ID:

Content-Type: text/plain; charset="utf-8"

Hi Greg,

Thanks for the info (a fair bit I was already aware of though) but up where
I am, pretty much everyone has their own DPU (big properties along a dirt
road. No concern with pits full of water though - everything is on power
poles including the DPUs and the fibre that connects them back to the
network (plus the copper feed into my house)! It does go underground
further down the mountain near the township but I'm not sure where the next
'active' kit is located (there is a lot of reading in the document the link
you sent :-).

These are not brief hits but outages that have lasted for days. Your
comment about the DPU losing its configuration is interesting though and
may well be the issue. I hope they replace it when NBNco visits me
tomorrow!

Cheers, Chris

Sent from my iPhone

On 3 Jan 2022, at 19:55, Greg Lipschitz
wrote:

?
Hi Chris,

The nbn DPU (the "mini-vdsl/gfast dslam" which lives in the pit) is reverse
powered (about 6-14W) from the NCD (the nbn box inside your house). nbn use
a mix of Netcomm and more recently Nokia DPU's. The DPU only needs 1
customer NCD connected to it to supply the power.

The DPU can connect up to 4 premises and the interesting thing is that any
of the RSP's who have a customer connected to the DPU can initiate a DPU
reset via the nbn API.

So, you might be with Telstra, and your neighbours with Aussie Broadband,
TPG and Optus and any one of them can essentially cause your service to
drop momentarily if they contact their RSP who initiates the DPU reset as
part of the troubleshooting.

We've found that the DPU's often suffer the same fate as the old ADSL
copper services when the pit fills up with water, the DPU goes off to la-la
land (who would have thunk power and water didn't mix!) or the copper
between the DPU and the customer gets corroded and needs to be cleaned up
or replaced.

I've not heard of power issues causing the DPU to have issues. It sounds
like the DPU is perhaps losing its config when there's a power interruption
(we've certainly seen that with the Telstra EA MRV OS904 NTU's over the
years).

If you're interested how it all comes together, nbn publish their design
docs -
https://www.nbnco.com.au/content/dam/nbn/documents/sell/sau/network-design-rules-20210630.pdf

Cheers
Greg

Greg Lipschitz | Founder & CEO | Summit Internet
*glipschitz at summitinternet.com.au*
*summitinternet.com.au*
*1300 049 749* <1300%20049%20749>
*Unit 2, 31-39 Norcal Road, Nunawading VIC 3131*

[image: Summit Internet]
------------------------------
*From:* AusNOG on behalf of Chris Chaundy

*Sent:* 03 January 2022 09:26
*To:* ausnog at ausnog.net
*Subject:* [AusNOG] NBN FTTC services and power hits

Is there anyone from NBNco on this list willing to discuss this with me?

It seems that every time there is a power hit in our locality (which with
Ausnet is pretty regularly), the FTTC services die and don?t come back
until you log a fault through the provider (in my case, Telstra) and then
wait for them to go through the hoops of dealing with NBNco.

This has happened four times in the last couple of months (thankfully I
have 4G backup). There must be some aggregator kit that has no power
backup or cannot recover from power outages - pretty crap really. :-|

Sorry if this is a little OT for the list but I?d be interested if other
FTTC areas see the same problem.

Sent from my iPhone
_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ausnog.net%2Fmailman%2Flistinfo%2Fausnog&data=04%7C01%7Cglipschitz%40summitinternet.com.au%7C7d3675defb8a4be9a6ab08d9ce3ef1a0%7C0838a12f226e43dfa6e4bb63d2643a7e%7C1%7C0%7C637767592012923616%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=eaYK8fpeVKh%2FCKcTW5KgKBQOB32TYug2Q5GLkCN7xLo%3D&reserved=0
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image192808.png
Type: image/png
Size: 10728 bytes
Desc: not available
URL:
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image612353.png
Type: image/png
Size: 3004 bytes
Desc: not available
URL:
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image793377.png
Type: image/png
Size: 984 bytes
Desc: not available
URL:
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image042550.png
Type: image/png
Size: 1930 bytes
Desc: not available
URL:

------------------------------

Subject: Digest Footer

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
http://lists.ausnog.net/mailman/listinfo/ausnog

------------------------------

End of AusNOG Digest, Vol 119, Issue 2
**************************************

[074055ca-183a-4963-a3b4-493d89700511]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20220104/a286be6f/attachment-0001.htm>