[AusNOG] [patchmanagement] ntp server issues today... strange one... clutching at straws but just in case

Roy Adams roy at racs.com.au
Fri Feb 1 20:21:24 EST 2019 

I hear you and completely understand how reliable that setup is, but
evaluating the facts:
Host to VM time sync was definitely disabled - always - standard practice
Host BIOS clock is set to UTC - again standard practice
Host OS (VMWARE v5.5U2 - Dell cut) also syncs to the same internet ntp
servers as PDC - purely for it's own purposes
BDC and all members always sync from PDC, including AD-integrated Synology
NAS

Someone raised a point however regarding VMTools maybe screwing up
We have recently upgraded all sites/vm's to stable vmtools v10.3.5
I have not seen any other sites with time issues since the upgrade.
I'll investigate this path, had not considered this, keeping my mind open

However, the fact still remains:
*I still find it strange that since changing the PDC to sync to sync ONLY
from 3.au.pool.ntp.org <http://3.au.pool.ntp.org/>, that the problem went
away*
*This one point still bothers me*

Like most of you, I am an old war dog in this industry - 41 years now..
Not competing, just saying.. I have learned not to jump to conclusions too
often
I am open to any plausible explanations

Kindly,

ROY ADAMS* | *P 07 3040 5010  | Web: http://www.racs.com.au/ | Wiki:
https://ex.racs.com.au:444/ | eMail: mailto:roy at racs.com.au
<roy at racs.com.au>
Please never upgrade to the latest Windows 10 - You don’t need the hassle,
and I don’t need the work.
If you think it's expensive to hire a professional to do the job, wait
until you hire an amateur - Red Adair.
Life is a journey through a series of adventures.. Live them, love them,
hate them, but never give up on your dreams, desires, and goals.



On Fri, 1 Feb 2019 at 18:14, Chris Barnes <chris.p.barnes at gmail.com> wrote:

> Yup, I've been running a stratum 2 server for the AU region and I was
> going to say exactly this. If there was a pool member behaving so badly for
> this long then the NTP Pool project would have pulled that server.
>
> For a server to be included in the pool it has to maintain a score above
> 10, maximum score is 20. As soon as a problem is detected it's score starts
> dropping very quickly. The worst score is -100 and it's a long slow
> recovery to go from -100 back up to +10 to be included in the pool again.
>
>
> On Fri., 1 Feb. 2019, 1:32 pm Rob Thomas, <xrobau at gmail.com> wrote:
>
>> As someone who runs a few Stratum 1 and 2 servers, pool.ntp.org checks
>> my servers every 15 minutes or so, and if they're more than 50ms out,
>> they're de-prioritized, until they're removed. It's all completely
>> automatic.
>>
>> Example: https://www.ntppool.org/scores/203.4.240.10
>>
>> I think this is more an issue on your end, honestly.
>>
>> --Rob
>>
>>
>> On Fri, 1 Feb 2019 at 12:17, Roy Adams <roy at racs.com.au> wrote:
>>
>>> Single PDC site... Hit the PDC every 30 mins or so - forward, back,
>>> forward, back etc
>>> then of course started changing time on all domain members shortly after
>>> - each time.
>>> Problem is it was not instant for all members and the AD-integrated
>>> Synology NAS....
>>> Backups broke, complaints from 20 users randomly every 30 mins until
>>> isolated.
>>>
>>> Cluster 3.au.pool.ntp.org has been fine since 3.39pm Brisbane time
>>> yesterday.
>>> I'll just ignore the 0. for now and wait for someone @ ntp.org to spot
>>> it I think
>>>
>>> Could just be specific to win2008r2Sp1 - who knows.
>>> AU Admins, you have been warned :)
>>> Enuf of my time wasted on it
>>>
>>> Thanks for all the comments and PM's
>>> I have actually picked up a lot of tips from you all - many thanks
>>>
>>>
>>> Kindly,
>>>
>>> ROY ADAMS* | *P 07 3040 5010  | Web: http://www.racs.com.au/ | Wiki:
>>> https://ex.racs.com.au:444/ | eMail: mailto:roy at racs.com.au
>>> <roy at racs.com.au>
>>> Please never upgrade to the latest Windows 10 - You don’t need the
>>> hassle, and I don’t need the work.
>>> If you think it's expensive to hire a professional to do the job, wait
>>> until you hire an amateur - Red Adair.
>>> Life is a journey through a series of adventures.. Live them, love them,
>>> hate them, but never give up on your dreams, desires, and goals.
>>>
>>>
>>>
>>> On Fri, 1 Feb 2019 at 00:54, Joseph Daly <JDaly at arrowstreetcapital.com>
>>> wrote:
>>>
>>>> One small thing and this is probably just the wording of the email.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *I always use the below config for domain controllers:*
>>>>
>>>> *sc config W32Time start= auto & net start W32Time*
>>>>
>>>> *w32tm /config /manualpeerlist:"0.au.pool.ntp.org
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.au.pool.ntp.org&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=U499vWTuQrHOdlAPlmDHrA-rgZbLYU7PaXpE2Kd48eM&e=>
>>>> 2.au.pool.ntp.org
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__2.au.pool.ntp.org&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=PRGGsc1Vf_jVqorVPZnTpw7JvXoa49lzKAVZTXF0gUs&e=>
>>>> 3.au.pool.ntp.org
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__3.au.pool.ntp.org&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=B3Zd-DIP0O9MfiOMufSpJ84RI0He4iXoMtlIv_CdbB0&e=>"
>>>> /syncfromflags:manual /reliable:yes /update & w32tm /resync /nowait*
>>>>
>>>>
>>>>
>>>> All DCs or just your PDC emulator? Usually the other DCs sync from the
>>>> PDC emulator.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *From:* Roy Adams <roy at racs.com.au>
>>>> *Sent:* Thursday, January 31, 2019 1:33 AM
>>>> *To:* Patch Management Mailing List <
>>>> patchmanagement at listserv.patchmanagement.org>
>>>> *Subject:* Re:[patchmanagement] [AusNOG] ntp server issues today...
>>>> strange one... clutching at straws but just in case
>>>>
>>>>
>>>>
>>>> Thanks for the PM's offering ideas
>>>>
>>>> I am tempted to set it back to 0. to debug the offending ntp pool IP,
>>>> but it was breaking all the backups among other things due to AD sync being
>>>> more than 5 mins out.
>>>>
>>>>
>>>>
>>>> I always use the below config for domain controllers:
>>>>
>>>> sc config W32Time start= auto & net start W32Time
>>>>
>>>> w32tm /config /manualpeerlist:"0.au.pool.ntp.org
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.au.pool.ntp.org&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=U499vWTuQrHOdlAPlmDHrA-rgZbLYU7PaXpE2Kd48eM&e=>
>>>> 2.au.pool.ntp.org
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__2.au.pool.ntp.org&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=PRGGsc1Vf_jVqorVPZnTpw7JvXoa49lzKAVZTXF0gUs&e=>
>>>> 3.au.pool.ntp.org
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__3.au.pool.ntp.org&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=B3Zd-DIP0O9MfiOMufSpJ84RI0He4iXoMtlIv_CdbB0&e=>"
>>>> /syncfromflags:manual /reliable:yes /update & w32tm /resync /nowait
>>>>
>>>>
>>>>
>>>> One of the replies noted that linux sanity checks by getting ntp time
>>>> from 4 servers - I wish MS were that smart.
>>>>
>>>> Clearly MS are not using all the configured servers, I suspect they are
>>>> purely for failover like a DNS client.
>>>>
>>>>
>>>>
>>>> I have just changed this site to:
>>>>
>>>> w32tm /config /manualpeerlist:"3.au.pool.ntp.org
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__3.au.pool.ntp.org&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=B3Zd-DIP0O9MfiOMufSpJ84RI0He4iXoMtlIv_CdbB0&e=>"
>>>> /syncfromflags:manual /reliable:yes /update & w32tm /resync /nowait
>>>>
>>>> So far so good.. still stable
>>>>
>>>>
>>>>
>>>> All Domain members of course sync to the DC
>>>>
>>>> I am not seeing this on any other sites.. all sites are cookie cutter
>>>> for me
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> event logs confirm ONLY the change... not the server IP :(
>>>>
>>>> The system time has changed to ‎2019‎-‎01‎-‎31T01:47:11.254922100Z from
>>>> ‎2019‎-‎01‎-‎31T02:18:29.514800000Z.
>>>>
>>>> The system time has changed to ‎2019‎-‎01‎-‎31T01:47:11.254000000Z from
>>>> ‎2019‎-‎01‎-‎31T01:47:11.254922100Z.
>>>>
>>>> The system time has changed to ‎2019‎-‎01‎-‎31T03:43:51.747000000Z from
>>>> ‎2019‎-‎01‎-‎31T03:12:32.312621000Z.
>>>>
>>>> The system time has changed to ‎2019‎-‎01‎-‎31T03:36:17.703840400Z from
>>>> ‎2019‎-‎01‎-‎31T04:07:36.105000000Z.
>>>>
>>>> The system time has changed to ‎2019‎-‎01‎-‎31T03:36:17.703000000Z from
>>>> ‎2019‎-‎01‎-‎31T03:36:17.703840400Z.
>>>>
>>>> The system time has changed to ‎2019‎-‎01‎-‎31T05:41:23.075000000Z from
>>>> ‎2019‎-‎01‎-‎31T05:10:04.617935900Z.
>>>>
>>>> The system time has changed to ‎2019‎-‎01‎-‎31T06:01:12.107000000Z from
>>>> ‎2019‎-‎01‎-‎31T06:01:12.107000000Z.
>>>>
>>>> The system time has changed to ‎2019‎-‎01‎-‎31T05:30:09.707385800Z from
>>>> ‎2019‎-‎01‎-‎31T06:01:28.112628100Z.
>>>>
>>>> The system time has changed to ‎2019‎-‎01‎-‎31T05:30:09.707000000Z from
>>>> ‎2019‎-‎01‎-‎31T05:30:09.707385800Z.
>>>>
>>>> The system time has changed to ‎2019‎-‎01‎-‎31T05:39:51.770000000Z from
>>>> ‎2019‎-‎01‎-‎31T05:39:51.770276000Z.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Kindly,
>>>>
>>>>
>>>>
>>>> ROY ADAMS* | *P 07 3040 5010  | Web: http://www.racs.com.au/
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.racs.com.au_&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=ZRKXJDH4ciRXiwcDhbLIwFHIvgqzrytOtvGja-WyEso&e=>
>>>>  | Wiki: https://ex.racs.com.au:444/
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__ex.racs.com.au-3A444_&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=DsV_izPAHTAwk-h02V5W_v5P98BrMy1Ul7Kol0HTTmc&e=> |
>>>> eMail: mailto:roy at racs.com.au <roy at racs.com.au>
>>>>
>>>> Please never upgrade to the latest Windows 10 - You don’t need the
>>>> hassle, and I don’t need the work.
>>>> If you think it's expensive to hire a professional to do the job, wait
>>>> until you hire an amateur - Red Adair.
>>>>
>>>> Life is a journey through a series of adventures.. Live them, love
>>>> them, hate them, but never give up on your dreams, desires, and goals.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, 31 Jan 2019 at 16:13, Nick Stallman <nick at agentpoint.com>
>>>> wrote:
>>>>
>>>> Do you know which server specifically? The ntp pools hand out random
>>>> NTP server IPs, it's not a fixed server.
>>>>
>>>> I'm not a Windows server admin, but this would likely be why Linux
>>>> connects to ~4 NTP servers so it can disregard dodgy servers.
>>>>
>>>> On 31/1/19 5:09 pm, Roy Adams wrote:
>>>>
>>>> Hi All, I have a domain controller *seemingly* receiving bad time info
>>>> today from 0.au.pool.ntp.org
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.au.pool.ntp.org&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=U499vWTuQrHOdlAPlmDHrA-rgZbLYU7PaXpE2Kd48eM&e=>
>>>>
>>>> Issuing this confirmed the time was flapping forward 30 mins, then 30
>>>> mins later back to normal:
>>>>
>>>> w32tm /query /status
>>>>
>>>> It confirmed the above ntp server as the server that supplied the bad
>>>> (then good, then bad, then good etc) time
>>>>
>>>> I have now changed the DC to pull instead from 3.au.pool.ntp.org
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__au.pool.ntp.org&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=vF1MfzyyrPVr436Gt8h40rxV1qMJ68GEs4Gu9MqWD-k&e=>
>>>> .
>>>>
>>>> 1 hour has passed and so far so good.
>>>>
>>>>
>>>>
>>>> Cannot say I have ever seen anything like this...
>>>>
>>>> It's only occurring on one site on a windows2008r2sp1 domain controller.
>>>>
>>>> The DC in turn relays this updated time to all domain members of course.
>>>>
>>>> Anyone else had time issues on any sites today in Aus?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Kindly,
>>>>
>>>>
>>>>
>>>> ROY ADAMS* | *P 07 3040 5010  | Web: http://www.racs.com.au/
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.racs.com.au_&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=ZRKXJDH4ciRXiwcDhbLIwFHIvgqzrytOtvGja-WyEso&e=>
>>>>  | Wiki: https://ex.racs.com.au:444/
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__ex.racs.com.au-3A444_&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=DsV_izPAHTAwk-h02V5W_v5P98BrMy1Ul7Kol0HTTmc&e=> |
>>>> eMail: mailto:roy at racs.com.au <roy at racs.com.au>
>>>>
>>>> Please never upgrade to the latest Windows 10 - You don’t need the
>>>> hassle, and I don’t need the work.
>>>> If you think it's expensive to hire a professional to do the job, wait
>>>> until you hire an amateur - Red Adair.
>>>>
>>>> Life is a journey through a series of adventures.. Live them, love
>>>> them, hate them, but never give up on your dreams, desires, and goals.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>>
>>>> AusNOG mailing list
>>>>
>>>> AusNOG at lists.ausnog.net
>>>>
>>>> http://lists.ausnog.net/mailman/listinfo/ausnog <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.ausnog.net_mailman_listinfo_ausnog&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=FeOndl6vwYUzDu74O11bqvYM6U3YN9aOiq9rAI3KKvw&e=>
>>>>
>>>> --
>>>>
>>>> *Nick Stallman*
>>>>
>>>> *Technical Director*
>>>>
>>>> nick at agentpoint.com
>>>>
>>>> 02 8039 6820 <0280396820>
>>>>
>>>> www.agentpoint.com.au
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.agentpoint.com.au_&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=PN87Mw56MVmSkWBGo0FRbjQ1ii1ML1UB8SR17fFvyIQ&e=>
>>>>
>>>>
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.agentpoint.com.au_&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=PN87Mw56MVmSkWBGo0FRbjQ1ii1ML1UB8SR17fFvyIQ&e=>
>>>>
>>>>
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.agentpoint.com.au_&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=PN87Mw56MVmSkWBGo0FRbjQ1ii1ML1UB8SR17fFvyIQ&e=>
>>>>
>>>> Level 3, 100 Harris Street, Pyrmont NSW 2009
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.agentpoint.com.au_&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=PN87Mw56MVmSkWBGo0FRbjQ1ii1ML1UB8SR17fFvyIQ&e=>
>>>>
>>>>
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.agentpoint.com.au_&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=PN87Mw56MVmSkWBGo0FRbjQ1ii1ML1UB8SR17fFvyIQ&e=>
>>>>
>>>>
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.agentpoint.com.au_&d=DwMFaQ&c=CI4-aYV4fOpCXKvKTF2ntOZntzAcNsg3IKFmdux4fpc&r=ZO0rs3RFhaNJvARM24Iy1y9-0qhOC5NjmySIqVusRM4&m=P1VICLtys7ExKCosSmwoCaMSdezGSjskIV3o0GAwJZs&s=PN87Mw56MVmSkWBGo0FRbjQ1ii1ML1UB8SR17fFvyIQ&e=>
>>>>
>>>> Arrowstreet Capital,LP-DISCLAIMER:
>>>> ==============================
>>>> This email message and its attachments are being sent by Arrowstreet
>>>> Capital, Limited Partnership and are confidential and proprietary. If you
>>>> are not the intended recipient, please notify us immediately by replying to
>>>> this message and destroy all copies of this message and any attachments.
>>>> Thank you.
>>>>
>>> _______________________________________________
>>> AusNOG mailing list
>>> AusNOG at lists.ausnog.net
>>> http://lists.ausnog.net/mailman/listinfo/ausnog
>>>
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> http://lists.ausnog.net/mailman/listinfo/ausnog
>>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20190201/6321094d/attachment.html>

More information about the AusNOG mailing list