[AusNOG] Telstra outage?

[Robert Hudson]

I'd suggest VMs are fine as a time source as long as they also sync from
something else AND you're aware of how your hypervisor impacts time keeping
and allow for that.

On Tue, 14 July 2026, 7:56 am Mark Smith, <markzzzsmith at gmail.com> wrote:

> Going from the news reports they only had two.
>
> Two services ir devices for redundancy is usually enough.
>
> That's what I thought about NTP, until I read the following BCP.
>
> The ideal minimum number of NTP time sources for a client is *5*, due to
> the way time sources are compared to try to identify the best and most
> accurate one to follow, and then to have redundancy, which comes with
> having an odd number of NTP sources.
>
> RFC8633, "Network Time Protocol Best Current Practices"
> https://datatracker.ietf.org/doc/html/rfc8633
>
>
> Also, don't use VMs as NTP time servers. The underlying virtualisation
> layer can cause the VM's virtual timer interrupt to drift about.
>
> You want a hardware timer interrupt to be driving the OS's clock.
>
> If Telstra had had 5 Stratacom NTP servers, even if they were all old and
> had the 20 years ago time fault, it's possible that due to different power
> on times, the first one that failed and turned time back 20 years would
> have been ignored by the others and clients because it became a bad time
> source, and they may have been lucky enough to have time to mitigate the
> remaining ones failing before it had an impact.
>
>
> Regards,
> Mark.
>
> On Tue, 14 July 2026, 15:30 Luke Thompson, <luke.t at tnc.works> wrote:
>
>> "Telstra had a nation-wide network outage last week that affected
>> emergency services.
>>
>> The outage has been pinned on an obsolete Symmetricom SyncServer S300
>> node, which manages time on the network but resets its 10-bit week counter
>> to zero every 1024 weeks (just under 20 years) in a common and well
>> understood GPS rollover bug that caused the device to reset to 2006.
>>
>> The SyncServer S300 was discontinued in 2016."
>>
>> Cheers,
>>
>> Luke Thompson, CTO
>> The Network Crew P/L
>>
>> E: luke.t at tnc.works
>> https://tnc.works
>>
>> On 9 July 2026 7:21:52 am Greg Price <greg at lakemountain.com> wrote:
>>
>>> The stuff I've read and heard is that it was a software update across
>>> multiple systems that provide the reference sources from GNSS that went
>>> wrong causing the the reference date/time to jump back decades, and the
>>> wheels fall off.  This is from the patchwork of sources, but I'd like to
>>> hear it definitively from engineers.
>>>
>>> On 9 Jul 2026, at 7:12 am, Phillip Grasso <phillip.grasso at gmail.com>
>>> wrote:
>>>
>>> Do you think it was related to software patching breaking time services?
>>> (NTP?)
>>>
>>> On Wed, 8 July 2026, 2:48 pm DaZZa, <dazzagibbs at gmail.com> wrote:
>>>
>>>> To paraphrase an oft-repeated meme
>>>>
>>>> It's not NTP
>>>> There's no way it's NTP
>>>> It was NTP
>>>>
>>>> Bets on this just being a speculator to take the heat off until they
>>>> find out what really went wrong?
>>>>
>>>> D
>>>>
>>>> On Wed, 8 Jul 2026 at 14:34, Luke Thompson <luke.t at tnc.works> wrote:
>>>> >
>>>> > Telstra's CFO has pointed to time / NTP:
>>>> >
>>>> >
>>>> https://www.theguardian.com/australia-news/2026/jul/08/telstra-outage-down-network-mobile-outages-today
>>>> >
>>>> > Seems just over 4 hours of impact, and so far they're unsure what
>>>> caused it.
>>>> >
>>>> > Cheers,
>>>> >
>>>> > Luke Thompson, CTO
>>>> > The Network Crew P/L
>>>> >
>>>> > E: luke.t at tnc.works
>>>> > https://tnc.works
>>>> >
>>>> > On 8 July 2026 8:43:01 am Luke Thompson <luke.t at tnc.works> wrote:
>>>> >
>>>> >> & likewise, service restored at the moment.
>>>> >>
>>>> >> Cheers,
>>>> >>
>>>> >> Luke Thompson, CTO
>>>> >> The Network Crew P/L
>>>> >>
>>>> >> luke.t at tnc.works
>>>> >> https://tnc.works
>>>> >>
>>>> >>
>>>> >> On 8/7/2026 8:40 am, lauricat at fastmail.fm wrote:
>>>> >>>
>>>> >>> Good morning
>>>> >>>
>>>> >>> Back here. Regional Vic  Boost and T prepaid mobile data.
>>>> >>>
>>>> >>> ----- Original message -----
>>>> >>> From: lauricat at fastmail.fm
>>>> >>> To: ausnog at lists.ausnog.net
>>>> >>> Subject: [AusNOG] Telstra outage?
>>>> >>> Date: Wednesday, 8 July 2026 7:44 AM
>>>> >>>
>>>> >>> G'day.
>>>> >>>
>>>> >>> How is T where you are?
>>>> >>>
>>>> >>> Regional Vic Boost network no calls other than emergency can be
>>>> made.
>>>> >>>
>>>> >>> Cheers
>>>> >>> Laurie
>>>> >>>
>>>> >> _______________________________________________
>>>> >> AusNOG mailing list
>>>> >> AusNOG at lists.ausnog.net
>>>> >> https://lists.ausnog.net/mailman/listinfo/ausnog
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > AusNOG mailing list
>>>> > AusNOG at lists.ausnog.net
>>>> > https://lists.ausnog.net/mailman/listinfo/ausnog
>>>> _______________________________________________
>>>> AusNOG mailing list
>>>> AusNOG at lists.ausnog.net
>>>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>>>
>>> _______________________________________________
>>> AusNOG mailing list
>>> AusNOG at lists.ausnog.net
>>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>>
>>>
>>>
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20260714/1a539589/attachment.htm>