Technical -

The Hidden Consequences of Let’s Encrypt’s Expired Root Certificate

By Sam Webster Published December 7, 2021

Our deployment system here at Gravity Forms relies on a number of tools and some third-party services to build, package, and distribute our plugin. We recently ran into an interesting problem where our deployments started failing due to an unexpected conflict between these systems that revealed how dependent we often are on underlying technologies to work as we expect – and how this can introduce fragility into the systems we rely on.

The issue for us started with the expiration of Let’s Encrypt’s DST Root CA X3 certificate on September 30, 2021. This had been coming for a long time, and for the most part, should have been handled by modern browsers and devices without incident. But due to some quirks in older software versions, this caused us some problems.

It’s always good for engineers and developers to be able to troubleshoot these sorts of problems – and hopefully you can learn from our experience too. Let’s jump in…

The Issue

We initially encountered the issue in our deployment process where a step of the deployment was failing due to an apparent error with an expired SSL certificate during a file upload. Curl was returning this message:

We checked the URL we were trying to upload to and its certificates were valid, so that was kind of strange.

The deployment tool we were using could be run in a Docker container or locally. So while we were troubleshooting, I tried to run the tool locally and got the same result. Eventually we fixed the automated deployment by bumping the version of PHP used in the docker container from 7.2 to 7.4, but I could still replicate the issue on my local machine regardless of the version of PHP I attempted to use.

The problem wasn’t limited to curl in my case either. If I ran `openssl s_client -showcerts -connect targeturl:443` I would be shown expired certificates.

So why would curl and openssl s_client commands return a different certificate than a web browser? And why would upgrading to PHP 7.4 fix this issue for our automated tools?

The Expiration of Let’s Encrypt’s DST Root CA X3 Certificate

This brought us back to the expiration of Let’s Encrypt’s DST Root CA X3 certificate. Browsers and devices trust SSL certificates, including Let’s Encrypt’s certificates, because the browsers and devices have copies of root certificates used in the certificate chain. If a browser doesn’t have a copy of the root certificate used by another certificate, it won’t trust that certificate.

Let’s Encrypt now has a root certificate called ISRG Root X1 that most browsers and devices should have. So for the most part, the expiration of the old DST Root CA X3 certificate shouldn’t cause any problems outside of some very particular situations. Web browsers were able to visit the site in question without any problems, because they were correctly using the new Let’s Encrypt root certificate, as we expected they would. It turned out that we had run into an edge case where this expiration could cause issues!

In order to maintain compliance for some older devices that don’t get regular updates, Let’s Encrypt includes a cross-signed certificate in their new chain for the expired DST Root CA X3. This allows older Android devices to still trust Let’s Encrypt certificates.

There is, however, an issue with certain versions of OpenSSL where untrusted certificate chains are always preferred. So when DST Root CA X3 expired, one path of the certificate chain was no longer trusted, and some versions of OpenSSL failed to validate.

So that was the issue we experienced. To avoid this validation issue, you have to be using OpenSSL at least 1.1.0 or later. Our version of OpenSSL was too old, and bumping the docker container to PHP 7.4 fixed that and allowed our builds to complete.

Getting it to Work Locally

My build was still failing locally, so I naturally assumed my version of OpenSSL was also out of date. I ran `openssl version` and sure enough, I was using an older version of OpenSSL. So I ran `brew update && brew upgrade` then `brew install OpenSSL`. I checked the OpenSSL version again, and:

So that’s good enough. I ran `openssl s_client -showcerts -connect` and it now showed valid certs. So I tried to run the build and it still failed. Curl was still seeing the certificates as expired. So I checked the curl version with `curl –version`.

That revealed the problem. My local version of curl wasn’t using the system version of OpenSSL, it had been compiled against LibreSSL 2.6.5 (a fork of OpenSSL) which still had the validation issue. I needed to update curl as well.

Updating Curl

The solution here depends on your system. On a mac, you can use homebrew and simply run `brew update` and `brew install curl`. This will download the latest version of curl, and you should be able to run `curl –version` again and see the correct version.

If not, you may need to add the correct version of curl to your path. You can see the line you need to add to your path by running `brew info curl`. You should see an output like this.

Look for the line that says “if you need to have curl in your PATH, run:” and run the following command in your terminal. Or manually add it to your .zshrc, .bash_profile, or .bashrc as appropriate.

If you’re not on a mac, you may need to compile curl yourself to get an appropriate version.

Once that’s done, your curl should no longer throw unexpected errors.

I tried the build again after updating curl and it worked like a charm!

Final Thoughts

So that’s the sneaky way that the expiry of a root SSL certificate caused some hiccups in our build process and forced me to reconsider the ways these events can affect the software we depend on. Sure, this all could be avoided by keeping your software up to date for the most part, but I’d bet that we don’t all regularly think about whether or not we need to recompile curl on our systems.

It is an important reminder though that the problems with a given technology stack may not always be where you expect, and understanding the fundamentals of how parts of your workflow fit together can save you a lot of headaches.