We have just discovered today that many sites may be being blocked due to a widely used security certificate group (the https/ssl portion of a website) expiring today. I have copied an article detailing the issue below. Please be patient as this unfortunately, is not an issue we can resolve but is external to Crowder. A screenshot below that shows what you will likely see if you reach a site using this certificate group.
Let's Encrypt R3 Intermediate Certificate Expiration (30 September 2021)
Simone Carletti — 28 September 2021
It has come to our attention that we're currently including, by default, in our HTTPS redirector and DNSimple app certificate installer, an intermediate SSL certificate for Let's Encrypt that will expire on September 30, 2021.
In this post, we'll explain the actions we're taking to address this, and caveats to be aware of regarding the intermediate certificate provided by Let's Encrypt as a replacement.
The Let's Encrypt R3 intermediate
Let's Encrypt adopts a common architecture for a certification authority based on root and intermediate certificates. The certificate authority owns one (or more) root certificates that are embedded root stores in the various browsers and operating systems. These certificates are not used directly for signining server certificates, to minimize security risks. Instead, these certificates are used to sign intermediate certificates that are used to sign server certificates. This approach helps segregate root certificates in highly secure offline facilities, removing the need from using them for live signing – a process that inheritely increases the risk of the certificate being compromised. Root certificates are highly valuable and very complicated to revoke/replace compared to intermediate certificates.
Since their inception, Let's Encrypt has adopted cross-signed root and intermediate certificates. This approach allowed Let's Encrypt to issue trusted certificates from the beginning using another certificate authority root (IdenTrust), while working to get their root certificates accepted by the various root stores.
On November 6, 2020, Let's Encrypt announced their intention to switch away from cross-signing to rely solely on their own root certificates. They altered the plan soon after when they realized some incompatibilities with certain older devices – in particular Android devices.
In a follow-up announcement on 21 December 2020, they announced their revised plan for Android Compatibility. This plan included the use of additional cross-signing on their root certificates, and they postponed the transition from the cross-signed R3 intermediate to their own R3 intermediate.
The transition was completed on May 2021. At this time, Let's Encrypt switched their default intermediate chain from using the certificate R3 (Cross-signed by IdenTrust) to the certificate R3 (Signed by ISRG Root X1).
This is the current Let's Encrypt Hierarchy as of August 2021.
The former certificate R3 (Cross-signed by IdenTrust) has since officially been flagged as retired by Let's Encrypt. At DNSimple, we continued to include this certificate in the default intermediate chain we serve from the HTTPS redirector and our DNSimple app certificate installer.
We planned to verify and switch the chain in the upcoming weeks, but it has taken longer than expected. Unfortunately, the retired certificate will expire in four days, on 30 September 2021. For this reason, we're now taking immediate action as described below.
What we did
Updated the HTTPS redirector chain
As of 28 September 2021 10:00 UTC, the HTTPS redirector has been updated to use the new R3 certificate.
Updated the DNSimple-app installer chain
As of 28 September 2021 10:00 UTC, the certificate installation wizard you can access from the dowload certificate page in the DNSimple app has updated to use the new R3 certificate.
What you need to do
If you're using Let's Encrypt certificates generated via DNSimple in our HTTPS redirector, you don't need to make any changes. Effective immediately, new HTTPS handshakes will use the non-expiring certificate.
If you're using Let's Encrypt certificates generated via DNSimple in your site, make sure to promptly replace the intermediate chain to use the non-expired intermediate certificate, as outlined below.
Replacing the retired R3 intermediate certificate
1 Use this tool to verify if your site is using the retired R3 intermediate certificate.
2 If confirmed, follow the instructions below to replace the certificate.
3 You don't need to request a new certificate, or a reissue – just replace the intermediate certificate bundle.
4 If you installed the certificate from the site, follow the same process to install it again. Download the certificate along with the updated bundle, and install it on your server or in the service like you would normally do on renewal every 90 days.
5 If you installed the certificate via API, you likely have a process to download and replace a certificate on every renewal. Run your integration again to pull the updated version of the bundle, and replace it on your servers.
6 Ship the changes, and confirm the fix by running the tool again.
What you need to know
As explained in the DST Root CA X3 Expiration (September 2021) page, there are some caveats when switching to this new chain.
On September 30 2021, there will be a small change in how older browsers and devices trust Let's Encrypt certificates. If you run a typical website, you won't notice a difference - the vast majority of your visitors will still accept your Let’s Encrypt certificate. If you provide an API or have to support IoT devices, you might have to pay a little more attention to the change.
If you actively use a Let's Encrypt certificate for one of those two use cases (API provider and/or IoT devices), we strongly encourage reading the article above and following the Let's Encrypt recommendations to be prepared for the switch.
What we learned and future improvements
At DNSimple, we support two different certificate authorities: Let's Encrypt and Sectigo.
The last intermediate/root certificate change involving Sectigo resulted in more issues than expected. For this reason, we've taken additional steps to proactively notify all customers with an active Let's Encrypt certificate.
• There is a status page to track the progress of the update at DNSimple.
• This blog post will go live to document the issue and provide instructions.
• We'll notify all customers with an active Let's Encrypt certificate via email.
We're also rolling out some changes to automate the process of switching Let's Encrypt intermediate chain using the default bundle provided by Let's Encrypt ACME protocol. We've been hesitant to implement this automation to avoid altering the chain without proper vetting and notification. However, this approach will better mimic the ACME protocol experience and result in a better user experience for our customers.
If you have any questions, please contact support – we're always here to help.