<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>cPanel also failed to plan for the expiry, so we're seeing
workarounds then revocations (oops, that didn't work - etc). Still
no real headway after 12~ hours.</p>
<p>The root cert expiry was a long time coming, though if you check
Twitter it seems like it's caught many out.</p>
<p>Cheers,<br>
</p>
<div class="moz-signature">Luke Thompson<br>
Operations Manager<br>
<br>
<br>
</div>
<div class="moz-cite-prefix">On 1/10/21 12:40 pm, Lachlan Gilmour
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAF72zDBcNpL2ri3YaRi1rdN9qj01cu12ohejjDgsNWBmE-47Yw@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">I believe it is related to the Lets Encrypt root
cert that expired overnight.
<div><br>
</div>
<div>I've seen quite a few older devices today having issues
accessing sites using lets encrypt certs. <br>
<div><br>
</div>
<div>More info on the issue can be found here: <a
href="https://docs.certifytheweb.com/docs/kb/kb-202109-letsencrypt/"
moz-do-not-send="true">https://docs.certifytheweb.com/docs/kb/kb-202109-letsencrypt/</a></div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Oct 1, 2021 at 12:36
PM Mark Andrews <<a href="mailto:marka@isc.org"
moz-do-not-send="true">marka@isc.org</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">More
correctly they had working DNSSEC deployed (<a
href="https://dnsviz.net/d/slack.com/YVXX_g/dnssec/"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://dnsviz.net/d/slack.com/YVXX_g/dnssec/</a>)
and then pulled both the DS records for <a
href="http://slack.com" rel="noreferrer" target="_blank"
moz-do-not-send="true">slack.com</a> and the DNSSEC records
in <a href="http://slack.com" rel="noreferrer"
target="_blank" moz-do-not-send="true">slack.com</a> AT THE
SAME TIME resulting in DNSSEC validation failures. Cached DS
records said <a href="http://slack.com" rel="noreferrer"
target="_blank" moz-do-not-send="true">slack.com</a> is
signed but the answers from the <a href="http://slack.com"
rel="noreferrer" target="_blank" moz-do-not-send="true">slack.com</a>
servers where missing the DNSSEC records. They failed to wait
for the DS records to expire from DNS caches before removing
the DNSSEC records in <a href="http://slack.com"
rel="noreferrer" target="_blank" moz-do-not-send="true">slack.com</a>.
Failure to wait for unsigned responses to clear caches before
publishing DS records can also cause issues with multiple
levels of caching.<br>
<br>
> On 1 Oct 2021, at 08:23, Scott Howard <<a
href="mailto:scott@doc.net.au" target="_blank"
moz-do-not-send="true">scott@doc.net.au</a>> wrote:<br>
> <br>
> They broke (and subsequently fixed) their DNSSEC
configuration many hours ago, but it was broken long enough to
get cached by some servers for up to 24 hours so some users
are still having issues connecting.<br>
> <br>
> Short of the classic "have your ISP clear their DNS
cache" not much anyone can do except wait it out...<br>
> <br>
> <a
href="https://status.slack.com/2021-09/06c1e17de93e7dc2"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://status.slack.com/2021-09/06c1e17de93e7dc2</a><br>
> <br>
> Scott<br>
> <br>
> <br>
> On Thu, Sep 30, 2021 at 3:19 PM Andrew Yager <<a
href="mailto:andrew@rwts.com.au" target="_blank"
moz-do-not-send="true">andrew@rwts.com.au</a>> wrote:<br>
> Hi,<br>
> <br>
> Slack is down and finding a few other (non slack)
services etc being broken seemingly with DNS things. Anyone
know what’s going on?<br>
> <br>
> A<br>
> _______________________________________________<br>
> AusNOG mailing list<br>
> <a href="mailto:AusNOG@lists.ausnog.net" target="_blank"
moz-do-not-send="true">AusNOG@lists.ausnog.net</a><br>
> <a
href="http://lists.ausnog.net/mailman/listinfo/ausnog"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.ausnog.net/mailman/listinfo/ausnog</a><br>
> _______________________________________________<br>
> AusNOG mailing list<br>
> <a href="mailto:AusNOG@lists.ausnog.net" target="_blank"
moz-do-not-send="true">AusNOG@lists.ausnog.net</a><br>
> <a
href="http://lists.ausnog.net/mailman/listinfo/ausnog"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.ausnog.net/mailman/listinfo/ausnog</a><br>
<br>
-- <br>
Mark Andrews, ISC<br>
1 Seymour St., Dundas Valley, NSW 2117, Australia<br>
PHONE: +61 2 9871 4742 INTERNET: <a
href="mailto:marka@isc.org" target="_blank"
moz-do-not-send="true">marka@isc.org</a><br>
<br>
_______________________________________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net" target="_blank"
moz-do-not-send="true">AusNOG@lists.ausnog.net</a><br>
<a href="http://lists.ausnog.net/mailman/listinfo/ausnog"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.ausnog.net/mailman/listinfo/ausnog</a><br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<table
style="max-width:600px;color:rgb(0,0,0);font-family:"times
new roman";font-size:medium" width="100%">
<tbody>
<tr width="100%">
<td style="font-family:arial,sans-serif" width="100%">
<p style="font-family:arial,helvetica,"sans
sefif";line-height:20px"><span
style="font-size:18px">Lachlan Gilmour</span><br>
<br>
</p>
<img
src="https://www.google.com/a/surfpacific.com.au/images/logo.gif"
style="width: 237.594px;" moz-do-not-send="true"
border="0">
<p> </p>
<table>
<tbody>
<tr>
<td style="text-align:center;width:10px"><span
style="font-size:12px">w</span></td>
<td><span style="font-size:12px">: <a
href="https://surfpacific.com.au/"
style="color:rgb(17,85,204)"
target="_blank" moz-do-not-send="true">surfpacific.com.au</a></span></td>
</tr>
<tr>
<td style="text-align:center;width:10px"><span
style="font-size:12px">p</span></td>
<td><span style="font-size:12px">: <a
href="tel:+61755711161"
style="color:rgb(17,85,204)"
target="_blank" moz-do-not-send="true">+61
7 5571 1161</a></span></td>
</tr>
<tr>
<td style="text-align:center;width:10px"><span
style="font-size:12px">f</span></td>
<td><span style="font-size:12px">: +61 7 5676
6652</span></td>
</tr>
<tr>
<td style="text-align:center;width:10px"><span
style="font-size:12px">e</span></td>
<td><span style="font-size:12px">: <a
href="mailto:lachlan.gilmour@surfpacific.com.au"
style="color:rgb(17,85,204)"
target="_blank" moz-do-not-send="true">lachlan.gilmour@surfpacific.com.au</a></span></td>
</tr>
<tr>
<td style="text-align:center;width:10px"
valign="top"><span style="font-size:12px">a</span></td>
<td style="font-family:arial,sans-serif"
valign="top">
<p
style="font-family:arial,helvetica,"sans
sefif";font-size:12px;line-height:20px;margin:0px">: Suite 30307,
Level 3, Tower 3 Southport Central
Commercial,<br>
9 Lawson Street, Southport, Queensland
4215, Australia.<br>
</p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<table
style="max-width:600px;color:rgb(0,0,0);font-family:"times
new roman";font-size:medium" width="100%">
<tbody>
<tr>
</tr>
</tbody>
</table>
<table
style="max-width:600px;color:rgb(0,0,0);font-family:"times
new roman";font-size:medium">
<tbody>
<tr>
<td><a href="http://remote.surfpacific.com/"
style="color:rgb(17,85,204)" target="_blank"
moz-do-not-send="true"><img alt=""
src="https://i.xink.io/Images/Get/N4269/s41.png"
moz-do-not-send="true" width="61" height="61"
border="0"></a></td>
</tr>
<tr>
<td style="font-family:arial,sans-serif" width="100%">
<hr
style="min-height:1px;color:rgb(36,66,137);background-color:rgb(36,66,137)"></td>
</tr>
<tr>
<td style="font-family:arial,sans-serif" width="100%"><small
style="font-size:11px;font-family:arial,helvetica,sans-serif;margin-top:10px;display:block"><b>Legal
Notice:</b> If this email message is received by
other than the named addressee(s), then the
recipient is requested immediately to notify us and
delete the email from the recipient’s computer
memory and to destroy all hard and other copies of
it. Privilege is not waived or lost by reason of a
mistaken delivery or transmission to other than the
addressee. Please </small></td>
</tr>
</tbody>
</table>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
AusNOG mailing list
<a class="moz-txt-link-abbreviated" href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a>
<a class="moz-txt-link-freetext" href="http://lists.ausnog.net/mailman/listinfo/ausnog">http://lists.ausnog.net/mailman/listinfo/ausnog</a>
</pre>
</blockquote>
</body>
</html>