[AusNOG] [patchmanagement] ntp server issues today... strange one... clutching at straws but just in case

Fri Feb 1 19:13:52 EST 2019

Yup, I've been running a stratum 2 server for the AU region and I was going
to say exactly this. If there was a pool member behaving so badly for this
long then the NTP Pool project would have pulled that server.

For a server to be included in the pool it has to maintain a score above
10, maximum score is 20. As soon as a problem is detected it's score starts
dropping very quickly. The worst score is -100 and it's a long slow
recovery to go from -100 back up to +10 to be included in the pool again.

On Fri., 1 Feb. 2019, 1:32 pm Rob Thomas, <xrobau at gmail.com> wrote:

> As someone who runs a few Stratum 1 and 2 servers, pool.ntp.org checks my
> servers every 15 minutes or so, and if they're more than 50ms out, they're
> de-prioritized, until they're removed. It's all completely automatic.
>
> Example: https://www.ntppool.org/scores/203.4.240.10
>
> I think this is more an issue on your end, honestly.
>
> --Rob
>
>
> On Fri, 1 Feb 2019 at 12:17, Roy Adams <roy at racs.com.au> wrote:
>
>> Single PDC site... Hit the PDC every 30 mins or so - forward, back,
>> forward, back etc
>> then of course started changing time on all domain members shortly after
>> - each time.
>> Problem is it was not instant for all members and the AD-integrated
>> Synology NAS....
>> Backups broke, complaints from 20 users randomly every 30 mins until
>> isolated.
>>
>> Cluster 3.au.pool.ntp.org has been fine since 3.39pm Brisbane time
>> yesterday.
>> I'll just ignore the 0. for now and wait for someone @ ntp.org to spot
>> it I think
>>
>> Could just be specific to win2008r2Sp1 - who knows.
>> AU Admins, you have been warned :)
>> Enuf of my time wasted on it
>>
>> Thanks for all the comments and PM's
>> I have actually picked up a lot of tips from you all - many thanks
>>
>>
>> Kindly,
>>
>> ROY ADAMS* | *P 07 3040 5010  | Web: http://www.racs.com.au/ | Wiki:
>> https://ex.racs.com.au:444/ | eMail: mailto:roy at racs.com.au
>> <roy at racs.com.au>
>> Please never upgrade to the latest Windows 10 - You don’t need the
>> hassle, and I don’t need the work.
>> If you think it's expensive to hire a professional to do the job, wait
>> until you hire an amateur - Red Adair.
>> Life is a journey through a series of adventures.. Live them, love them,
>> hate them, but never give up on your dreams, desires, and goals.
>>
>>
>>
>> On Fri, 1 Feb 2019 at 00:54, Joseph Daly <JDaly at arrowstreetcapital.com>
>> wrote:
>>
>>> One small thing and this is probably just the wording of the email.
>>>
>>>
>>>
>>>
>>>
>>> *I always use the below config for domain controllers:*
>>>
>>> *sc config W32Time start= auto & net start W32Time*
>>>
>>> *w32tm /config /manualpeerlist:"0.au.pool.ntp.org
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.au.pool.ntp.org&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=U499vWTuQrHOdlAPlmDHrA-rgZbLYU7PaXpE2Kd48eM&e=>
>>> 2.au.pool.ntp.org
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__2.au.pool.ntp.org&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=PRGGsc1Vf_jVqorVPZnTpw7JvXoa49lzKAVZTXF0gUs&e=>
>>> 3.au.pool.ntp.org
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__3.au.pool.ntp.org&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=B3Zd-DIP0O9MfiOMufSpJ84RI0He4iXoMtlIv_CdbB0&e=>"
>>> /syncfromflags:manual /reliable:yes /update & w32tm /resync /nowait*
>>>
>>>
>>>
>>> All DCs or just your PDC emulator? Usually the other DCs sync from the
>>> PDC emulator.
>>>
>>>
>>>
>>>
>>>
>>> *From:* Roy Adams <roy at racs.com.au>
>>> *Sent:* Thursday, January 31, 2019 1:33 AM
>>> *To:* Patch Management Mailing List <
>>> patchmanagement at listserv.patchmanagement.org>
>>> *Subject:* Re:[patchmanagement] [AusNOG] ntp server issues today...
>>> strange one... clutching at straws but just in case
>>>
>>>
>>>
>>> Thanks for the PM's offering ideas
>>>
>>> I am tempted to set it back to 0. to debug the offending ntp pool IP,
>>> but it was breaking all the backups among other things due to AD sync being
>>> more than 5 mins out.
>>>
>>>
>>>
>>> I always use the below config for domain controllers:
>>>
>>> sc config W32Time start= auto & net start W32Time
>>>
>>> w32tm /config /manualpeerlist:"0.au.pool.ntp.org
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.au.pool.ntp.org&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=U499vWTuQrHOdlAPlmDHrA-rgZbLYU7PaXpE2Kd48eM&e=>
>>> 2.au.pool.ntp.org
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__2.au.pool.ntp.org&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=PRGGsc1Vf_jVqorVPZnTpw7JvXoa49lzKAVZTXF0gUs&e=>
>>> 3.au.pool.ntp.org
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__3.au.pool.ntp.org&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=B3Zd-DIP0O9MfiOMufSpJ84RI0He4iXoMtlIv_CdbB0&e=>"
>>> /syncfromflags:manual /reliable:yes /update & w32tm /resync /nowait
>>>
>>>
>>>
>>> One of the replies noted that linux sanity checks by getting ntp time
>>> from 4 servers - I wish MS were that smart.
>>>
>>> Clearly MS are not using all the configured servers, I suspect they are
>>> purely for failover like a DNS client.
>>>
>>>
>>>
>>> I have just changed this site to:
>>>
>>> w32tm /config /manualpeerlist:"3.au.pool.ntp.org
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__3.au.pool.ntp.org&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=B3Zd-DIP0O9MfiOMufSpJ84RI0He4iXoMtlIv_CdbB0&e=>"
>>> /syncfromflags:manual /reliable:yes /update & w32tm /resync /nowait
>>>
>>> So far so good.. still stable
>>>
>>>
>>>
>>> All Domain members of course sync to the DC
>>>
>>> I am not seeing this on any other sites.. all sites are cookie cutter
>>> for me
>>>
>>>
>>>
>>>
>>>
>>> event logs confirm ONLY the change... not the server IP :(
>>>
>>> The system time has changed to ‎2019‎-‎01‎-‎31T01:47:11.254922100Z from
>>> ‎2019‎-‎01‎-‎31T02:18:29.514800000Z.
>>>
>>> The system time has changed to ‎2019‎-‎01‎-‎31T01:47:11.254000000Z from
>>> ‎2019‎-‎01‎-‎31T01:47:11.254922100Z.
>>>
>>> The system time has changed to ‎2019‎-‎01‎-‎31T03:43:51.747000000Z from
>>> ‎2019‎-‎01‎-‎31T03:12:32.312621000Z.
>>>
>>> The system time has changed to ‎2019‎-‎01‎-‎31T03:36:17.703840400Z from
>>> ‎2019‎-‎01‎-‎31T04:07:36.105000000Z.
>>>
>>> The system time has changed to ‎2019‎-‎01‎-‎31T03:36:17.703000000Z from
>>> ‎2019‎-‎01‎-‎31T03:36:17.703840400Z.
>>>
>>> The system time has changed to ‎2019‎-‎01‎-‎31T05:41:23.075000000Z from
>>> ‎2019‎-‎01‎-‎31T05:10:04.617935900Z.
>>>
>>> The system time has changed to ‎2019‎-‎01‎-‎31T06:01:12.107000000Z from
>>> ‎2019‎-‎01‎-‎31T06:01:12.107000000Z.
>>>
>>> The system time has changed to ‎2019‎-‎01‎-‎31T05:30:09.707385800Z from
>>> ‎2019‎-‎01‎-‎31T06:01:28.112628100Z.
>>>
>>> The system time has changed to ‎2019‎-‎01‎-‎31T05:30:09.707000000Z from
>>> ‎2019‎-‎01‎-‎31T05:30:09.707385800Z.
>>>
>>> The system time has changed to ‎2019‎-‎01‎-‎31T05:39:51.770000000Z from
>>> ‎2019‎-‎01‎-‎31T05:39:51.770276000Z.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Kindly,
>>>
>>>
>>>
>>> ROY ADAMS* | *P 07 3040 5010  | Web: http://www.racs.com.au/
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.racs.com.au_&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=ZRKXJDH4ciRXiwcDhbLIwFHIvgqzrytOtvGja-WyEso&e=>
>>>  | Wiki: https://ex.racs.com.au:444/
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__ex.racs.com.au-3A444_&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=DsV_izPAHTAwk-h02V5W_v5P98BrMy1Ul7Kol0HTTmc&e=> |
>>> eMail: mailto:roy at racs.com.au <roy at racs.com.au>
>>>
>>> Please never upgrade to the latest Windows 10 - You don’t need the
>>> hassle, and I don’t need the work.
>>> If you think it's expensive to hire a professional to do the job, wait
>>> until you hire an amateur - Red Adair.
>>>
>>> Life is a journey through a series of adventures.. Live them, love them,
>>> hate them, but never give up on your dreams, desires, and goals.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, 31 Jan 2019 at 16:13, Nick Stallman <nick at agentpoint.com> wrote:
>>>
>>> Do you know which server specifically? The ntp pools hand out random NTP
>>> server IPs, it's not a fixed server.
>>>
>>> I'm not a Windows server admin, but this would likely be why Linux
>>> connects to ~4 NTP servers so it can disregard dodgy servers.
>>>
>>> On 31/1/19 5:09 pm, Roy Adams wrote:
>>>
>>> Hi All, I have a domain controller *seemingly* receiving bad time info
>>> today from 0.au.pool.ntp.org
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.au.pool.ntp.org&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=U499vWTuQrHOdlAPlmDHrA-rgZbLYU7PaXpE2Kd48eM&e=>
>>>
>>> Issuing this confirmed the time was flapping forward 30 mins, then 30
>>> mins later back to normal:
>>>
>>> w32tm /query /status
>>>
>>> It confirmed the above ntp server as the server that supplied the bad
>>> (then good, then bad, then good etc) time
>>>
>>> I have now changed the DC to pull instead from 3.au.pool.ntp.org
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__au.pool.ntp.org&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=vF1MfzyyrPVr436Gt8h40rxV1qMJ68GEs4Gu9MqWD-k&e=>
>>> .
>>>
>>> 1 hour has passed and so far so good.
>>>
>>>
>>>
>>> Cannot say I have ever seen anything like this...
>>>
>>> It's only occurring on one site on a windows2008r2sp1 domain controller.
>>>
>>> The DC in turn relays this updated time to all domain members of course.
>>>
>>> Anyone else had time issues on any sites today in Aus?
>>>
>>>
>>>
>>>
>>>
>>> Kindly,
>>>
>>>
>>>
>>> ROY ADAMS* | *P 07 3040 5010  | Web: http://www.racs.com.au/
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.racs.com.au_&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=ZRKXJDH4ciRXiwcDhbLIwFHIvgqzrytOtvGja-WyEso&e=>
>>>  | Wiki: https://ex.racs.com.au:444/
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__ex.racs.com.au-3A444_&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=DsV_izPAHTAwk-h02V5W_v5P98BrMy1Ul7Kol0HTTmc&e=> |
>>> eMail: mailto:roy at racs.com.au <roy at racs.com.au>
>>>
>>> Please never upgrade to the latest Windows 10 - You don’t need the
>>> hassle, and I don’t need the work.
>>> If you think it's expensive to hire a professional to do the job, wait
>>> until you hire an amateur - Red Adair.
>>>
>>> Life is a journey through a series of adventures.. Live them, love them,
>>> hate them, but never give up on your dreams, desires, and goals.
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>>
>>> AusNOG mailing list
>>>
>>> AusNOG at lists.ausnog.net
>>>
>>> http://lists.ausnog.net/mailman/listinfo/ausnog <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.ausnog.net_mailman_listinfo_ausnog&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=FeOndl6vwYUzDu74O11bqvYM6U3YN9aOiq9rAI3KKvw&e=>
>>>
>>> --
>>>
>>> *Nick Stallman*
>>>
>>> *Technical Director*
>>>
>>> nick at agentpoint.com
>>>
>>> 02 8039 6820 <0280396820>
>>>
>>> www.agentpoint.com.au
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.agentpoint.com.au_&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=PN87Mw56MVmSkWBGo0FRbjQ1ii1ML1UB8SR17fFvyIQ&e=>
>>>
>>>
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.agentpoint.com.au_&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=PN87Mw56MVmSkWBGo0FRbjQ1ii1ML1UB8SR17fFvyIQ&e=>
>>>
>>>
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.agentpoint.com.au_&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=PN87Mw56MVmSkWBGo0FRbjQ1ii1ML1UB8SR17fFvyIQ&e=>
>>>
>>> Level 3, 100 Harris Street, Pyrmont NSW 2009
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.agentpoint.com.au_&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=PN87Mw56MVmSkWBGo0FRbjQ1ii1ML1UB8SR17fFvyIQ&e=>
>>>
>>>
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.agentpoint.com.au_&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=PN87Mw56MVmSkWBGo0FRbjQ1ii1ML1UB8SR17fFvyIQ&e=>
>>>
>>>
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.agentpoint.com.au_&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=PN87Mw56MVmSkWBGo0FRbjQ1ii1ML1UB8SR17fFvyIQ&e=>
>>>
>>> Arrowstreet Capital,LP-DISCLAIMER:
>>> ==============================
>>> This email message and its attachments are being sent by Arrowstreet
>>> Capital, Limited Partnership and are confidential and proprietary. If you
>>> are not the intended recipient, please notify us immediately by replying to
>>> this message and destroy all copies of this message and any attachments.
>>> Thank you.
>>>
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> http://lists.ausnog.net/mailman/listinfo/ausnog
>>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20190201/bd72d4b6/attachment-0001.html>