[AusNOG] Global DNS yuck?

Wed Oct 6 14:49:26 EST 2021

This isn't just old devices either, my Android phone is running near enough to the latest release and this broke my radius auth against a Lets encrypt cert.

I had to manually edit the CA to remove the final root cert, which Lets Encrypt are still including via the renewal tool I'm using.

I knew it was coming, but didn't expect it to cause this problem....

________________________________
From: AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Luke Thompson <luke.t at tncrew.com.au>
Sent: Friday, 1 October 2021 12:44:44 PM
To: Lachlan Gilmour; Mark Andrews
Cc: ausnog at lists.ausnog.net
Subject: Re: [AusNOG] Global DNS yuck?

cPanel also failed to plan for the expiry, so we're seeing workarounds then revocations (oops, that didn't work - etc). Still no real headway after 12~ hours.

The root cert expiry was a long time coming, though if you check Twitter it seems like it's caught many out.

Cheers,

Luke Thompson
Operations Manager

On 1/10/21 12:40 pm, Lachlan Gilmour wrote:
I believe it is related to the Lets Encrypt root cert that expired overnight.

I've seen quite a few older devices today having issues accessing sites using lets encrypt certs.

More info on the issue can be found here: https://docs.certifytheweb.com/docs/kb/kb-202109-letsencrypt/<http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZhMDU5Mjk2ZGMwY2FkNGNjYz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiYyZjk4NjkwOGJlM2E1YTI9MTMzMyYmdXJsPWh0dHBzJTNBJTJGJTJGZG9jcyUyRWNlcnRpZnl0aGV3ZWIlMkVjb20lMkZkb2NzJTJGa2IlMkZrYi0yMDIxMDktbGV0c2VuY3J5cHQlMkY=>

On Fri, Oct 1, 2021 at 12:36 PM Mark Andrews <marka at isc.org<mailto:marka at isc.org>> wrote:
More correctly they had working DNSSEC deployed (https://dnsviz.net/d/slack.com/YVXX_g/dnssec/<http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiMTQ0NjA3MDliZDFkZGRhYz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiY0ZjA4NjkwOGJlM2E1OWY9MTMzMyYmdXJsPWh0dHBzJTNBJTJGJTJGZG5zdml6JTJFbmV0JTJGZCUyRnNsYWNrJTJFY29tJTJGWVZYWCU1RmclMkZkbnNzZWMlMkY=>) and then pulled both the DS records for slack.com<http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYzQ0M2I2NDhlOGQ5ZWM4Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiZiZmFjYjk4Y2FlZGU1YWI9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZzbGFjayUyRWNvbQ==> and the DNSSEC records in slack.com<http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYzQ0M2I2NDhlOGQ5ZWM4Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiZiZmFjYjk4Y2FlZGU1YWI9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZzbGFjayUyRWNvbQ==> AT THE SAME TIME resulting in DNSSEC validation failures. Cached DS records said slack.com<http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYzQ0M2I2NDhlOGQ5ZWM4Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiZiZmFjYjk4Y2FlZGU1YWI9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZzbGFjayUyRWNvbQ==> is signed but the answers from the slack.com<http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYzQ0M2I2NDhlOGQ5ZWM4Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiZiZmFjYjk4Y2FlZGU1YWI9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZzbGFjayUyRWNvbQ==> servers where missing the DNSSEC records. They failed to wait for the DS records to expire from DNS caches before removing the DNSSEC records in slack.com<http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYzQ0M2I2NDhlOGQ5ZWM4Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiZiZmFjYjk4Y2FlZGU1YWI9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZzbGFjayUyRWNvbQ==>.  Failure to wait for unsigned responses to clear caches before publishing DS records can also cause issues with multiple levels of caching.

> On 1 Oct 2021, at 08:23, Scott Howard <scott at doc.net.au<mailto:scott at doc.net.au>> wrote:
>
> They broke (and subsequently fixed) their DNSSEC configuration many hours ago, but it was broken long enough to get cached by some servers for up to 24 hours so some users are still having issues connecting.
>
> Short of the classic "have your ISP clear their DNS cache" not much anyone can do except wait it out...
>
> https://status.slack.com/2021-09/06c1e17de93e7dc2<http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiODUxMmM3ZjlhYzFkZWQ2OD02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiY4YTk5OGMxZDVhM2JhZmY9MTMzMyYmdXJsPWh0dHBzJTNBJTJGJTJGc3RhdHVzJTJFc2xhY2slMkVjb20lMkYyMDIxLTA5JTJGMDZjMWUxN2RlOTNlN2RjMg==>
>
>   Scott
>
>
> On Thu, Sep 30, 2021 at 3:19 PM Andrew Yager <andrew at rwts.com.au<mailto:andrew at rwts.com.au>> wrote:
> Hi,
>
> Slack is down and finding a few other (non slack) services etc being broken seemingly with DNS things. Anyone know what’s going on?
>
> A
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
> http://lists.ausnog.net/mailman/listinfo/ausnog<http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYTVmMjgzYWRhYzdjNTk0Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiZhZmFjMTlmODllZmU0ZTk9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZsaXN0cyUyRWF1c25vZyUyRW5ldCUyRm1haWxtYW4lMkZsaXN0aW5mbyUyRmF1c25vZw==>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
> http://lists.ausnog.net/mailman/listinfo/ausnog<http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYTVmMjgzYWRhYzdjNTk0Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiZhZmFjMTlmODllZmU0ZTk9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZsaXN0cyUyRWF1c25vZyUyRW5ldCUyRm1haWxtYW4lMkZsaXN0aW5mbyUyRmF1c25vZw==>

--
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742              INTERNET: marka at isc.org<mailto:marka at isc.org>

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
http://lists.ausnog.net/mailman/listinfo/ausnog<http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYTVmMjgzYWRhYzdjNTk0Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiZhZmFjMTlmODllZmU0ZTk9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZsaXN0cyUyRWF1c25vZyUyRW5ldCUyRm1haWxtYW4lMkZsaXN0aW5mbyUyRmF1c25vZw==>

--

Lachlan Gilmour

[https://www.google.com/a/surfpacific.com.au/images/logo.gif]

w       : surfpacific.com.au<http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYzQ0M2I2NGM3OTg5ZTk0ZD02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiY0ZWVkYTk1OTRlZmU5YWY9MTMzMyYmdXJsPWh0dHBzJTNBJTJGJTJGc3VyZnBhY2lmaWMlMkVjb20lMkVhdSUyRg==>
p       : +61 7 5571 1161<tel:+61755711161>
f       : +61 7 5676 6652
e       : lachlan.gilmour at surfpacific.com.au<mailto:lachlan.gilmour at surfpacific.com.au>
a

: Suite 30307, Level 3, Tower 3 Southport Central Commercial,
  9 Lawson Street, Southport, Queensland 4215, Australia.

[https://i.xink.io/Images/Get/N4269/s41.png]<http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYzQ0M2I2NDhlOGQ5ZWM5Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiYyZjZjNzg3ODFhMGY5YjM9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZyZW1vdGUlMkVzdXJmcGFjaWZpYyUyRWNvbSUyRg==>
________________________________

Legal Notice: If this email message is received by other than the named addressee(s), then the recipient is requested immediately to notify us and delete the email from the recipient’s computer memory and to destroy all hard and other copies of it. Privilege is not waived or lost by reason of a mistaken delivery or transmission to other than the addressee. Please

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
http://lists.ausnog.net/mailman/listinfo/ausnog<http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYTVmMjgzYWRhYzdjNTk0Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiZhZmFjMTlmODllZmU0ZTk9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZsaXN0cyUyRWF1c25vZyUyRW5ldCUyRm1haWxtYW4lMkZsaXN0aW5mbyUyRmF1c25vZw==>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20211006/a8aef50e/attachment.html>