<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Tue, 14 Jul 2026 at 17:09, Michael Junek <<a href="mailto:michael@juneks.com.au">michael@juneks.com.au</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-1574154300606724377">
<div dir="ltr">
<div id="m_-1574154300606724377divtagdefaultwrapper" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif" dir="ltr">
<p><span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:16px">If they have only two devices, that's not enough for redundancy.
You need a minimum of three. With only two upstream sources, and NTP client cannot accurately determine which is the correct time - whereas with 3, you'll know that 'one' is wrong.<br></span></p></div></div></div></blockquote><div><br></div><div>And if one of your 3 fail, you now only have two, and similar to if you have only two watches on your wrist, you can't tell which one is accurate.</div><div><br></div><div>Therefore with 3 as minimum you have a single point of failure.</div><div><br></div><div>That's why the BCP says a minimum of four, and ideally it's 5 because you want an odd number under normal circumstances.</div><div><br></div><div>In addition to the <a href="https://www.ntppool.org/">https://www.ntppool.org/</a> NTP time servers make available, I've found plenty of freely available NTP time sources on the Internet that are run by highly funded and should able to trusted as a time sources like Microsoft, Apple, Coudflare, Facebook, as well as many others run by government agencies and universities over both IPv4 and IPv6. (Google don't use leap seconds, they smear time, and I understand that's incompatible with other public NTP time sources.)</div><div><br></div><div>Somebody might argue you shouldn't trust time you get off the Internet, which in some cases would be valid. However, if you have a lot of time sources from diverse organisations around the world, your local NTP implementation is inherently measuring them all against each other, which is therefore inherently building more trust in the time accuracy across them all.</div><div><br></div><div>One case however where I wouldn't use "random" publicly available NTP time servers is if financial transactions were being timestamped with time derived from those publicly available NTP time sources. I could see a lawyer in court arguing that financial transaction timestamps can't be trusted if the timestamp was sourced from the Internet. </div><div><br></div><div>You could use the Australian Government's NMI time servers for financial transactions.</div><div><br></div><div> <a href="https://www.industry.gov.au/national-measurement-institute/nmi-services/physical-measurement-services/time-and-frequency-services">https://www.industry.gov.au/national-measurement-institute/nmi-services/physical-measurement-services/time-and-frequency-services</a></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-1574154300606724377"><div dir="ltr"><div id="m_-1574154300606724377divtagdefaultwrapper" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif" dir="ltr"><p><span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:16px">
</span>That being said if all the Stratum 1 devices rolled back their clocks simultaneously, then all downstream clients would have started to assume that as correct time; once their polling intervals came around (~17 min).<br></p></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-1574154300606724377"><div dir="ltr"><div id="m_-1574154300606724377divtagdefaultwrapper" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif" dir="ltr"><p><br>
Usually the GPS hardware has the GPS "base week" written in as part of the firmware, to calculate the real time when they report their time back. It's likely that if the devices were all manufactured at the same time, that they would have all died relatively
simultaneously.</p></div></div></div></blockquote><div><br></div><div>That's probably the case. I was thinking of clock source uptime being a factor, and with likely different uptimes there may be a lucky opportunity to stop other clocks having an effect.</div><div><br></div><div>So this is really an example of the debt collector coming for the technical debt of not replacing obsolete equipment.</div><div><br></div><div>Regards,</div><div>Mark.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-1574154300606724377"><div dir="ltr"><div id="m_-1574154300606724377divtagdefaultwrapper" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif" dir="ltr">
<p><br>
</p>
<div>
<div style="color:rgb(0,0,0)">
<hr style="display:inline-block;width:98%">
<div id="m_-1574154300606724377divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> AusNOG <<a href="mailto:ausnog-bounces@lists.ausnog.net" target="_blank">ausnog-bounces@lists.ausnog.net</a>> on behalf of Mark Smith <<a href="mailto:markzzzsmith@gmail.com" target="_blank">markzzzsmith@gmail.com</a>><br>
<b>Sent:</b> Tuesday, 14 July 2026 15:55<br>
<b>Cc:</b> Ausnog; Greg Price<br>
<b>Subject:</b> Re: [AusNOG] Telstra outage?</font>
<div> </div>
</div>
<div>
<div dir="auto">Going from the news reports they only had two.
<div dir="auto"><br>
</div>
<div dir="auto">Two services ir devices for redundancy is usually enough.</div>
<div dir="auto"><br>
</div>
<div dir="auto">That's what I thought about NTP, until I read the following BCP.</div>
<div dir="auto"><br>
</div>
<div dir="auto">The ideal minimum number of NTP time sources for a client is *5*, due to the way time sources are compared to try to identify the best and most accurate one to follow, and then to have redundancy, which comes with having an odd number of NTP
sources.</div>
<div dir="auto"><br>
</div>
<div dir="auto">RFC8633, "Network Time Protocol Best Current Practices"</div>
<div dir="auto"><a href="https://datatracker.ietf.org/doc/html/rfc8633" id="m_-1574154300606724377LPlnk160115" target="_blank">https://datatracker.ietf.org/doc/html/rfc8633</a></div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">Also, don't use VMs as NTP time servers. The underlying virtualisation layer can cause the VM's virtual timer interrupt to drift about.</div>
<div dir="auto"><br>
</div>
<div dir="auto">You want a hardware timer interrupt to be driving the OS's clock.</div>
<div dir="auto"><br>
</div>
<div dir="auto">If Telstra had had 5 Stratacom NTP servers, even if they were all old and had the 20 years ago time fault, it's possible that due to different power on times, the first one that failed and turned time back 20 years would have been ignored by
the others and clients because it became a bad time source, and they may have been lucky enough to have time to mitigate the remaining ones failing before it had an impact.</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">Regards,</div>
<div dir="auto">Mark.</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, 14 July 2026, 15:30 Luke Thompson, <luke.t@tnc.works> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<u></u>
<div>
<div dir="auto">
<div dir="auto"><span style="font-size:12pt">"Telstra had a nation-wide network outage last week that affected emergency services. </span></div>
<div dir="auto"><br>
</div>
<div dir="auto">The outage has been pinned on an obsolete Symmetricom SyncServer S300 node, which manages time on the network but resets its 10-bit week counter to zero every 1024 weeks (just under 20 years) in a common and well understood GPS rollover bug
that caused the device to reset to 2006.</div>
<div dir="auto"><br>
</div>
<div dir="auto">The SyncServer S300 was discontinued in 2016."</div>
<div dir="auto"><br>
</div>
<div dir="auto">Cheers, </div>
<div dir="auto"><br>
</div>
<div id="m_-1574154300606724377m_8693697543794731986m_4443338909368167326aqm-signature" dir="auto" style="color:black">
<div dir="auto">Luke Thompson, CTO</div>
<div dir="auto">The Network Crew P/L</div>
<div dir="auto"><br>
</div>
<div dir="auto">E: luke.t@tnc.works</div>
<div dir="auto"><a href="https://tnc.works" rel="noreferrer noreferrer" id="m_-1574154300606724377LPlnk217193" target="_blank">https://tnc.works</a></div>
</div>
<div dir="auto"><br>
</div>
<div id="m_-1574154300606724377m_8693697543794731986m_4443338909368167326aqm-original" style="color:black">
<div>
<div style="color:black">
<p style="color:black;font-size:10pt;font-family:sans-serif;margin:8pt 0px">On 9 July 2026 7:21:52 am Greg Price <<a href="mailto:greg@lakemountain.com" rel="noreferrer noreferrer" target="_blank">greg@lakemountain.com</a>> wrote:</p>
<blockquote type="cite" class="gmail_quote" style="margin:0px 0px 0px 0.75ex;border-left:1px solid rgb(128,128,128);padding-left:0.75ex">
The stuff I've read and heard is that it was a software update across multiple systems that provide the reference sources from GNSS that went wrong causing the the reference date/time to jump back decades, and the wheels fall off. This is from the patchwork
of sources, but I'd like to hear it definitively from engineers.<br id="m_-1574154300606724377m_8693697543794731986m_4443338909368167326lineBreakAtBeginningOfMessage">
<div><br>
<blockquote type="cite">
<div>On 9 Jul 2026, at 7:12 am, Phillip Grasso <<a href="mailto:phillip.grasso@gmail.com" rel="noreferrer noreferrer" target="_blank">phillip.grasso@gmail.com</a>> wrote:</div>
<br>
<div>
<div dir="auto">Do you think it was related to software patching breaking time services? (NTP?)</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, 8 July 2026, 2:48 pm DaZZa, <<a href="mailto:dazzagibbs@gmail.com" rel="noreferrer noreferrer" target="_blank">dazzagibbs@gmail.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
To paraphrase an oft-repeated meme<br>
<br>
It's not NTP<br>
There's no way it's NTP<br>
It was NTP<br>
<br>
Bets on this just being a speculator to take the heat off until they<br>
find out what really went wrong?<br>
<br>
D<br>
<br>
On Wed, 8 Jul 2026 at 14:34, Luke Thompson <luke.t@tnc.works> wrote:<br>
><br>
> Telstra's CFO has pointed to time / NTP:<br>
><br>
> <a href="https://www.theguardian.com/australia-news/2026/jul/08/telstra-outage-down-network-mobile-outages-today" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank">
https://www.theguardian.com/australia-news/2026/jul/08/telstra-outage-down-network-mobile-outages-today</a><br>
><br>
> Seems just over 4 hours of impact, and so far they're unsure what caused it.<br>
><br>
> Cheers,<br>
><br>
> Luke Thompson, CTO<br>
> The Network Crew P/L<br>
><br>
> E: luke.t@tnc.works<br>
> <a href="https://tnc.works/" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank">
https://tnc.works</a><br>
><br>
> On 8 July 2026 8:43:01 am Luke Thompson <luke.t@tnc.works> wrote:<br>
><br>
>> & likewise, service restored at the moment.<br>
>><br>
>> Cheers,<br>
>><br>
>> Luke Thompson, CTO<br>
>> The Network Crew P/L<br>
>><br>
>> luke.t@tnc.works<br>
>> <a href="https://tnc.works/" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank">
https://tnc.works</a><br>
>><br>
>><br>
>> On 8/7/2026 8:40 am, <a href="mailto:lauricat@fastmail.fm" rel="noreferrer noreferrer noreferrer" target="_blank">
lauricat@fastmail.fm</a> wrote:<br>
>>><br>
>>> Good morning<br>
>>><br>
>>> Back here. Regional Vic Boost and T prepaid mobile data.<br>
>>><br>
>>> ----- Original message -----<br>
>>> From: <a href="mailto:lauricat@fastmail.fm" rel="noreferrer noreferrer noreferrer" target="_blank">
lauricat@fastmail.fm</a><br>
>>> To: <a href="mailto:ausnog@lists.ausnog.net" rel="noreferrer noreferrer noreferrer" target="_blank">
ausnog@lists.ausnog.net</a><br>
>>> Subject: [AusNOG] Telstra outage?<br>
>>> Date: Wednesday, 8 July 2026 7:44 AM<br>
>>><br>
>>> G'day.<br>
>>><br>
>>> How is T where you are?<br>
>>><br>
>>> Regional Vic Boost network no calls other than emergency can be made.<br>
>>><br>
>>> Cheers<br>
>>> Laurie<br>
>>><br>
>> _______________________________________________<br>
>> AusNOG mailing list<br>
>> <a href="mailto:AusNOG@lists.ausnog.net" rel="noreferrer noreferrer noreferrer" target="_blank">
AusNOG@lists.ausnog.net</a><br>
>> <a href="https://lists.ausnog.net/mailman/listinfo/ausnog" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank">
https://lists.ausnog.net/mailman/listinfo/ausnog</a><br>
><br>
><br>
> _______________________________________________<br>
> AusNOG mailing list<br>
> <a href="mailto:AusNOG@lists.ausnog.net" rel="noreferrer noreferrer noreferrer" target="_blank">
AusNOG@lists.ausnog.net</a><br>
> <a href="https://lists.ausnog.net/mailman/listinfo/ausnog" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank">
https://lists.ausnog.net/mailman/listinfo/ausnog</a><br>
_______________________________________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net" rel="noreferrer noreferrer noreferrer" target="_blank">AusNOG@lists.ausnog.net</a><br>
<a href="https://lists.ausnog.net/mailman/listinfo/ausnog" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank">https://lists.ausnog.net/mailman/listinfo/ausnog</a><br>
</blockquote>
</div>
_______________________________________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net" rel="noreferrer noreferrer" target="_blank">AusNOG@lists.ausnog.net</a><br>
<a href="https://lists.ausnog.net/mailman/listinfo/ausnog" rel="noreferrer noreferrer" target="_blank">https://lists.ausnog.net/mailman/listinfo/ausnog</a><br>
</div>
</blockquote>
</div>
<br>
</blockquote>
</div>
</div>
</div>
<div dir="auto"><br>
</div>
</div>
</div>
_______________________________________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net" rel="noreferrer noreferrer" target="_blank">AusNOG@lists.ausnog.net</a><br>
<a href="https://lists.ausnog.net/mailman/listinfo/ausnog" rel="noreferrer noreferrer noreferrer" target="_blank">https://lists.ausnog.net/mailman/listinfo/ausnog</a><br>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</div></blockquote></div></div>