<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body>
<p>Interesting you mention that - it's what cPanel flagged as a
contributing factor towards how long their restoration took.</p>
<p><br>
</p>
<p>"While we were aware of Lets Encrypt's upcoming root expiration,
we were not aware that they would and still are providing us a
chain to that expired root as it is the default chain when using
their API. Identifying and understanding this certificate
behaviour ahead of time would have helped us pre-empt these issues
and I can assure you that the importance of this early
confirmation in the future is being emphasised."</p>
<p><br>
</p>
<p>The main forum threads on the LE site are rather vocal shall we
say! Understandably so.<br>
</p>
<p><br>
</p>
<p>Hopefully it's a learning exercise that helps with future
transitions.<br>
</p>
<p><br>
</p>
<p>Cheers,</p>
<p><br>
</p>
<div class="moz-signature">Luke Thompson<br>
Operations Manager<br>
<br>
<br>
</div>
<div class="moz-cite-prefix">On 6/10/21 2:49 pm, Beeson, Ayden
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:1611811c386f43f3ab22cfd83d522368@csu.edu.au">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<style type="text/css" style="display:none;">P {margin-top:0;margin-bottom:0;}</style>
<div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;"
dir="ltr">
<p>This isn't just old devices either, my Android phone is
running near enough to the latest release and this broke my
radius auth against a Lets encrypt cert.</p>
<p><br>
</p>
<p>I had to manually edit the CA to remove the final root cert,
which Lets Encrypt are still including via the renewal tool
I'm using.</p>
<p><br>
</p>
<p>I knew it was coming, but didn't expect it to cause this
problem....<br>
</p>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt"
face="Calibri, sans-serif" color="#000000"><b>From:</b> AusNOG
<a class="moz-txt-link-rfc2396E" href="mailto:ausnog-bounces@lists.ausnog.net"><ausnog-bounces@lists.ausnog.net></a> on behalf of Luke
Thompson <a class="moz-txt-link-rfc2396E" href="mailto:luke.t@tncrew.com.au"><luke.t@tncrew.com.au></a><br>
<b>Sent:</b> Friday, 1 October 2021 12:44:44 PM<br>
<b>To:</b> Lachlan Gilmour; Mark Andrews<br>
<b>Cc:</b> <a class="moz-txt-link-abbreviated" href="mailto:ausnog@lists.ausnog.net">ausnog@lists.ausnog.net</a><br>
<b>Subject:</b> Re: [AusNOG] Global DNS yuck?</font>
<div> </div>
</div>
<div>
<p>cPanel also failed to plan for the expiry, so we're seeing
workarounds then revocations (oops, that didn't work - etc).
Still no real headway after 12~ hours.</p>
<p>The root cert expiry was a long time coming, though if you
check Twitter it seems like it's caught many out.</p>
<p>Cheers,<br>
</p>
<div class="moz-signature">Luke Thompson<br>
Operations Manager<br>
<br>
<br>
</div>
<div class="moz-cite-prefix">On 1/10/21 12:40 pm, Lachlan
Gilmour wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAF72zDBcNpL2ri3YaRi1rdN9qj01cu12ohejjDgsNWBmE-47Yw@mail.gmail.com">
<div dir="ltr">I believe it is related to the Lets Encrypt
root cert that expired overnight.
<div><br>
</div>
<div>I've seen quite a few older devices today having issues
accessing sites using lets encrypt certs. <br>
<div><br>
</div>
<div>More info on the issue can be found here: <a
href="http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZhMDU5Mjk2ZGMwY2FkNGNjYz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiYyZjk4NjkwOGJlM2E1YTI9MTMzMyYmdXJsPWh0dHBzJTNBJTJGJTJGZG9jcyUyRWNlcnRpZnl0aGV3ZWIlMkVjb20lMkZkb2NzJTJGa2IlMkZrYi0yMDIxMDktbGV0c2VuY3J5cHQlMkY="
moz-do-not-send="true">
https://docs.certifytheweb.com/docs/kb/kb-202109-letsencrypt/</a></div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Oct 1, 2021 at
12:36 PM Mark Andrews <<a href="mailto:marka@isc.org"
moz-do-not-send="true">marka@isc.org</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
More correctly they had working DNSSEC deployed (<a
href="http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiMTQ0NjA3MDliZDFkZGRhYz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiY0ZjA4NjkwOGJlM2E1OWY9MTMzMyYmdXJsPWh0dHBzJTNBJTJGJTJGZG5zdml6JTJFbmV0JTJGZCUyRnNsYWNrJTJFY29tJTJGWVZYWCU1RmclMkZkbnNzZWMlMkY="
rel="noreferrer" target="_blank" moz-do-not-send="true">https://dnsviz.net/d/slack.com/YVXX_g/dnssec/</a>)
and then pulled both the DS records for <a
href="http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYzQ0M2I2NDhlOGQ5ZWM4Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiZiZmFjYjk4Y2FlZGU1YWI9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZzbGFjayUyRWNvbQ=="
rel="noreferrer" target="_blank" moz-do-not-send="true">
slack.com</a> and the DNSSEC records in <a
href="http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYzQ0M2I2NDhlOGQ5ZWM4Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiZiZmFjYjk4Y2FlZGU1YWI9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZzbGFjayUyRWNvbQ=="
rel="noreferrer" target="_blank" moz-do-not-send="true">
slack.com</a> AT THE SAME TIME resulting in DNSSEC
validation failures. Cached DS records said
<a
href="http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYzQ0M2I2NDhlOGQ5ZWM4Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiZiZmFjYjk4Y2FlZGU1YWI9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZzbGFjayUyRWNvbQ=="
rel="noreferrer" target="_blank" moz-do-not-send="true">
slack.com</a> is signed but the answers from the <a
href="http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYzQ0M2I2NDhlOGQ5ZWM4Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiZiZmFjYjk4Y2FlZGU1YWI9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZzbGFjayUyRWNvbQ=="
rel="noreferrer" target="_blank" moz-do-not-send="true">
slack.com</a> servers where missing the DNSSEC records.
They failed to wait for the DS records to expire from DNS
caches before removing the DNSSEC records in
<a
href="http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYzQ0M2I2NDhlOGQ5ZWM4Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiZiZmFjYjk4Y2FlZGU1YWI9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZzbGFjayUyRWNvbQ=="
rel="noreferrer" target="_blank" moz-do-not-send="true">
slack.com</a>. Failure to wait for unsigned responses
to clear caches before publishing DS records can also
cause issues with multiple levels of caching.<br>
<br>
> On 1 Oct 2021, at 08:23, Scott Howard <<a
href="mailto:scott@doc.net.au" target="_blank"
moz-do-not-send="true">scott@doc.net.au</a>> wrote:<br>
> <br>
> They broke (and subsequently fixed) their DNSSEC
configuration many hours ago, but it was broken long
enough to get cached by some servers for up to 24 hours so
some users are still having issues connecting.<br>
> <br>
> Short of the classic "have your ISP clear their DNS
cache" not much anyone can do except wait it out...<br>
> <br>
> <a
href="http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiODUxMmM3ZjlhYzFkZWQ2OD02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiY4YTk5OGMxZDVhM2JhZmY9MTMzMyYmdXJsPWh0dHBzJTNBJTJGJTJGc3RhdHVzJTJFc2xhY2slMkVjb20lMkYyMDIxLTA5JTJGMDZjMWUxN2RlOTNlN2RjMg=="
rel="noreferrer" target="_blank" moz-do-not-send="true">
https://status.slack.com/2021-09/06c1e17de93e7dc2</a><br>
> <br>
> Scott<br>
> <br>
> <br>
> On Thu, Sep 30, 2021 at 3:19 PM Andrew Yager <<a
href="mailto:andrew@rwts.com.au" target="_blank"
moz-do-not-send="true">andrew@rwts.com.au</a>> wrote:<br>
> Hi,<br>
> <br>
> Slack is down and finding a few other (non slack)
services etc being broken seemingly with DNS things.
Anyone know what’s going on?<br>
> <br>
> A<br>
> _______________________________________________<br>
> AusNOG mailing list<br>
> <a href="mailto:AusNOG@lists.ausnog.net"
target="_blank" moz-do-not-send="true">
AusNOG@lists.ausnog.net</a><br>
> <a
href="http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYTVmMjgzYWRhYzdjNTk0Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiZhZmFjMTlmODllZmU0ZTk9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZsaXN0cyUyRWF1c25vZyUyRW5ldCUyRm1haWxtYW4lMkZsaXN0aW5mbyUyRmF1c25vZw=="
rel="noreferrer" target="_blank" moz-do-not-send="true">
http://lists.ausnog.net/mailman/listinfo/ausnog</a><br>
> _______________________________________________<br>
> AusNOG mailing list<br>
> <a href="mailto:AusNOG@lists.ausnog.net"
target="_blank" moz-do-not-send="true">
AusNOG@lists.ausnog.net</a><br>
> <a
href="http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYTVmMjgzYWRhYzdjNTk0Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiZhZmFjMTlmODllZmU0ZTk9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZsaXN0cyUyRWF1c25vZyUyRW5ldCUyRm1haWxtYW4lMkZsaXN0aW5mbyUyRmF1c25vZw=="
rel="noreferrer" target="_blank" moz-do-not-send="true">
http://lists.ausnog.net/mailman/listinfo/ausnog</a><br>
<br>
-- <br>
Mark Andrews, ISC<br>
1 Seymour St., Dundas Valley, NSW 2117, Australia<br>
PHONE: +61 2 9871 4742 INTERNET: <a
href="mailto:marka@isc.org" target="_blank"
moz-do-not-send="true">
marka@isc.org</a><br>
<br>
_______________________________________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net" target="_blank"
moz-do-not-send="true">AusNOG@lists.ausnog.net</a><br>
<a
href="http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYTVmMjgzYWRhYzdjNTk0Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiZhZmFjMTlmODllZmU0ZTk9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZsaXN0cyUyRWF1c25vZyUyRW5ldCUyRm1haWxtYW4lMkZsaXN0aW5mbyUyRmF1c25vZw=="
rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.ausnog.net/mailman/listinfo/ausnog</a><br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<table
style="max-width:600px;color:rgb(0,0,0);font-family:"times
new roman";font-size:medium" width="100%">
<tbody>
<tr width="100%">
<td style="font-family:arial,sans-serif"
width="100%">
<p style="font-family:arial,helvetica,"sans
sefif";line-height:20px">
<span style="font-size:18px">Lachlan Gilmour</span><br>
<br>
</p>
<img style="width: 237.594px;"
moz-do-not-send="true"
src="https://www.google.com/a/surfpacific.com.au/images/logo.gif"
border="0">
<p> </p>
<table>
<tbody>
<tr>
<td style="text-align:center;width:10px"><span
style="font-size:12px">w</span></td>
<td><span style="font-size:12px">: <a
href="http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYzQ0M2I2NGM3OTg5ZTk0ZD02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiY0ZWVkYTk1OTRlZmU5YWY9MTMzMyYmdXJsPWh0dHBzJTNBJTJGJTJGc3VyZnBhY2lmaWMlMkVjb20lMkVhdSUyRg=="
style="color:rgb(17,85,204)"
target="_blank" moz-do-not-send="true">surfpacific.com.au</a></span></td>
</tr>
<tr>
<td style="text-align:center;width:10px"><span
style="font-size:12px">p</span></td>
<td><span style="font-size:12px">: <a
href="tel:+61755711161"
style="color:rgb(17,85,204)"
target="_blank" moz-do-not-send="true">+61
7 5571 1161</a></span></td>
</tr>
<tr>
<td style="text-align:center;width:10px"><span
style="font-size:12px">f</span></td>
<td><span style="font-size:12px">: +61 7
5676 6652</span></td>
</tr>
<tr>
<td style="text-align:center;width:10px"><span
style="font-size:12px">e</span></td>
<td><span style="font-size:12px">: <a
href="mailto:lachlan.gilmour@surfpacific.com.au"
style="color:rgb(17,85,204)"
target="_blank" moz-do-not-send="true">lachlan.gilmour@surfpacific.com.au</a></span></td>
</tr>
<tr>
<td style="text-align:center;width:10px"
valign="top"><span style="font-size:12px">a</span></td>
<td style="font-family:arial,sans-serif"
valign="top">
<p
style="font-family:arial,helvetica,"sans
sefif";font-size:12px;line-height:20px;margin:0px">
: Suite 30307, Level 3, Tower 3
Southport Central Commercial,<br>
9 Lawson Street, Southport, Queensland
4215, Australia.<br>
</p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<table
style="max-width:600px;color:rgb(0,0,0);font-family:"times
new roman";font-size:medium" width="100%">
<tbody>
<tr>
</tr>
</tbody>
</table>
<table
style="max-width:600px;color:rgb(0,0,0);font-family:"times
new roman";font-size:medium">
<tbody>
<tr>
<td><a
href="http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYzQ0M2I2NDhlOGQ5ZWM5Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiYyZjZjNzg3ODFhMGY5YjM9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZyZW1vdGUlMkVzdXJmcGFjaWZpYyUyRWNvbSUyRg=="
style="color:rgb(17,85,204)" target="_blank"
moz-do-not-send="true"><img alt=""
moz-do-not-send="true"
src="https://i.xink.io/Images/Get/N4269/s41.png"
width="61" height="61" border="0"></a></td>
</tr>
<tr>
<td style="font-family:arial,sans-serif"
width="100%">
<hr
style="min-height:1px;color:rgb(36,66,137);background-color:rgb(36,66,137)"></td>
</tr>
<tr>
<td style="font-family:arial,sans-serif"
width="100%"><small
style="font-size:11px;font-family:arial,helvetica,sans-serif;margin-top:10px;display:block"><b>Legal
Notice:</b> If this email message is received
by other than the named addressee(s), then the
recipient is requested immediately to notify us
and delete the email from the recipient’s
computer memory and to destroy all hard and
other copies of it. Privilege is not waived or
lost by reason of a mistaken delivery or
transmission to other than the addressee.
Please </small></td>
</tr>
</tbody>
</table>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
AusNOG mailing list
<a class="moz-txt-link-abbreviated" href="mailto:AusNOG@lists.ausnog.net" moz-do-not-send="true">AusNOG@lists.ausnog.net</a>
<a class="moz-txt-link-freetext" href="http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYTVmMjgzYWRhYzdjNTk0Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiZhZmFjMTlmODllZmU0ZTk9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZsaXN0cyUyRWF1c25vZyUyRW5ldCUyRm1haWxtYW4lMkZsaXN0aW5mbyUyRmF1c25vZw==" moz-do-not-send="true">http://lists.ausnog.net/mailman/listinfo/ausnog</a>
</pre>
</blockquote>
</div>
</blockquote>
</body>
</html>