Letsencrypt DST Root CA X3 expiration (September 30, 2021) workaround on CentOS 7.x OpenSSL 1.0.2

Letsencrypt's DST Root CA X3 certificate is expiring on September 30, 2021 which means those older devices that don’t trust ISRG Root X1 will start getting certificate warnings for Letsencrypt SSL certificate based web sites - with the exception of older Android devices (see below).

Centmin Mod 123.09beta01 and newer has been updated to apply a workaround fix on CentOS 7 which system wide applications use older OpenSSL 1.0.2k and can not easily be updated to OpenSSL 1.1.1. Centmin Mod 123.09beta01's Nginx build though uses newer OpenSSL 1.1.1 by default on CentOS 7.

Workaround fix for CentOS 7 for DST Root CA X3 certificate expiration

The work around is to remove via CA Trust blacklisting the soon to expire Letsencrypt DST Root CA X3 certificate (September 30, 2021) from system CA Trust store on CentOS 7 leaving system OpenSSL 1.0.2k to verify Letsencrypt SSL certificates using already included ISRG Root X1 in system CA Trust store on CentOS 7.

Before removal of DST Root CA X3, the trust field = anchor when you run trust list grep command:

Code (Text):

trust list | grep -C3 'DST Root CA X3'

pkcs11:id=%c4%a7%b1%a4%7b%2c%71%fa%db%e1%4b%90%75%ff%c4%15%60%85%89%10;type=cert
   type: certificate
   label: DST Root CA X3
   trust: anchor
   category: authority

Fix via centmin.sh

Centmin Mod 123.09beta01 or higher folks on CentOS 7 can run SSH command:

Code (Text):

cmupdate

Example cmupdate output

Code (Text):

cmupdate

From https://github.com/centminmod/centminmod
   d4fb17c..fe014e7  123.09beta01 -> origin/123.09beta01
Updating d4fb17c..fe014e7
Fast-forward
 centmin-cli.sh                   |  2 +-
 centmin.sh                       |  2 +-
 inc/cpcheck.inc                  | 28 ++++++++++++++++++++++++++++
 installer-dnf-gitlab.sh          | 28 ++++++++++++++++++++++++++++
 installer-dnf.sh                 | 28 ++++++++++++++++++++++++++++
 installer-gitlab.sh              | 28 ++++++++++++++++++++++++++++
 installer-latest-gitlab.sh       | 28 ++++++++++++++++++++++++++++
 installer-latest.sh              | 28 ++++++++++++++++++++++++++++
 installer-minimal-gitlab.sh      | 28 ++++++++++++++++++++++++++++
 installer-minimal.sh             | 28 ++++++++++++++++++++++++++++
 installer-ngxpagespeed-gitlab.sh | 28 ++++++++++++++++++++++++++++
 installer-ngxpagespeed.sh        | 28 ++++++++++++++++++++++++++++
 installer.sh                     | 28 ++++++++++++++++++++++++++++
 installer55-gitlab.sh            | 28 ++++++++++++++++++++++++++++
 installer55.sh                   | 28 ++++++++++++++++++++++++++++
 installer56-gitlab.sh            | 28 ++++++++++++++++++++++++++++
 installer56.sh                   | 28 ++++++++++++++++++++++++++++
 installer7-gitlab.sh             | 28 ++++++++++++++++++++++++++++
 installer7.sh                    | 28 ++++++++++++++++++++++++++++
 installer71-gitlab.sh            | 28 ++++++++++++++++++++++++++++
 installer71.sh                   | 28 ++++++++++++++++++++++++++++
 installer72-gitlab.sh            | 28 ++++++++++++++++++++++++++++
 installer72.sh                   | 28 ++++++++++++++++++++++++++++
 installer73-gitlab.sh            | 28 ++++++++++++++++++++++++++++
 installer73.sh                   | 28 ++++++++++++++++++++++++++++
 installer74-gitlab.sh            | 28 ++++++++++++++++++++++++++++
 installer74.sh                   | 28 ++++++++++++++++++++++++++++
 installer80-gitlab.sh            | 28 ++++++++++++++++++++++++++++
 installer80.sh                   | 28 ++++++++++++++++++++++++++++
 tools/dst-root-ca-fix.sh         | 41 +++++++++++++++++++++++++++++++++++++++++
 30 files changed, 799 insertions(+), 2 deletions(-)
 create mode 100755 tools/dst-root-ca-fix.sh

then run once the centmin.sh menu command and then exist the menu

Code (Text):

centmin

running centmin.sh on updated Centmin Mod 123.09beta01 will automatically apply the workaround fix and remove via CA Trust blacklisting the soon to expire Letsencrypt DST Root CA X3 certificate.

Fix via dst-root-ca-fix.sh script

---

Alternate way is via standalone script, /usr/local/src/centminmod/tools/dst-root-ca-fix.sh which you can manually run or automate yourself to run. Centmin Mod 123.09beta01 or higher folks on CentOS 7 can run SSH command:

Code (Text):

cmupdate

Then run the standalone script

Code (Text):

/usr/local/src/centminmod/tools/dst-root-ca-fix.sh

Example script run output

Code (Text):

/usr/local/src/centminmod/tools/dst-root-ca-fix.sh

Update workaround to blacklist expiring Letsencrypt DST Root CA X3 certificate...
https://community.centminmod.com/threads/21965/

Diff check file at /root/tools/backup-ca-certs/diff-ca-bundle.crt.diff

Check to see if DST Root CA X3 is blacklisted
trust list | grep -C3 'DST Root CA X3' | grep -B1 'blacklisted'

    label: DST Root CA X3
    trust: blacklisted

After removal of DST Root CA X3, the trust field = blacklisted

Code (Text):

trust list | grep -C3 'DST Root CA X3' | grep 'blacklisted'
   trust: blacklisted

Code (Text):

trust list | grep -C3 'DST Root CA X3'
pkcs11:id=%c4%a7%b1%a4%7b%2c%71%fa%db%e1%4b%90%75%ff%c4%15%60%85%89%10;type=cert
   type: certificate
   label: DST Root CA X3
   trust: blacklisted
   category: authority

CentOS other versions:

• CentOS 6 OpenSSL 1.0.1 solution is still do be determined and CentOS 6 was end of life anyway, so folks should eventually have moved from CentOS 6 to CentOS 7 server anyway.
• CentOS 8 uses OpenSSL 1.1.1 so should be fine as OpenSSL 1.1.0+ has set a new flag which is enabled by default for X509_V_FLAG_TRUSTED_FIRST. Which means OpenSSL 1.1.0+ will stop verification checks once it encounters the valid ISRG Root X1 certificate in the CA Trust store on the system. While on CentOS 6/7 OpenSSL 1.0.1/1.0.2, it will continue checking to the last certificate in the new Letsencrypt default long chain which is the DST Root CA X3 certificate anchor which will expire on September 30, 2021. Which is why the above workaround fix in theory should work for OpenSSL 1.0.2 on CentOS 7 by blacklisting DST Root CA X3 certificate anchor, it won't be the last in default long chain to be verified.
• On OpenSSL 1.0.2 this flag is available but disabled by default. OpenSSL 1.0.1 AFAIK, doesn't support the flag.

From https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

From https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/

• https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/
• https://community.letsencrypt.org/t/production-chain-changes/150739
• https://scotthelme.co.uk/lets-encrypt-old-root-expiration/
• https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/

 

CentOS 7 ca-certificates YUM Update

Looks like CentOS 7 YUM update is available for up to date ca-certificates RPM which updates the system CA Trust store and removes the soon to expire DST Root CA X3 certificate.

Code (Text):

yum list updates -q
Updated                 Packages
ca-certificates.noarch  2021.2.50-72.el7_9  updates

Code (Text):

rpm -qa --changelog ca-certificates | head -n5
* Tue Sep 14 2021 Bob Relyea <rrelyea@redhat.com> - 2021.2.50-72
- Fix expired certificate.
-    Removing:
-     # Certificate "DST Root CA X3"

So doing a yum update should fix it too.

Code (Text):

yum -y update ca-certificates

checking and DST Root CA X3 is still listed as blacklisted from above work around

Code (Text):

trust list | grep -C3 'DST Root CA X3'
pkcs11:id=%c4%a7%b1%a4%7b%2c%71%fa%db%e1%4b%90%75%ff%c4%15%60%85%89%10;type=cert
    type: certificate
    label: DST Root CA X3
    trust: blacklisted
    category: authority

Though checking actual CA bundle, DST Root doesn't exist now via grep, returns empty

Code (Text):

grep -i 'DST Root' /etc/pki/tls/certs/ca-bundle.crt

 

Updated Centmin Mod 123.09beta01 with an updated routine to account for ca-certificates YUM update in the previous post which removes the expiring DST Root CA X3 certificate from
/etc/pki/tls/certs/ca-bundle.crt CA Trust system bundle. So previous workaround would give an error outlined at https://community.centminmod.com/posts/90262/. This update accounts for folks updating their ca-certificates YUM package.

Existing users, just run cmupdate command to update your local 123.09beta01 code

 

Letsencrypt posted a Youtube video of Boulder developer Aaron Gable explaining the DST Root CA X3 certificate expiry and its impact.

 

CentOS 6 OpenSSL 1.0.1, wget, curl client fixes

Updated Centmin Mod 123.09beta01 with specific fixes only if you created Centmin Mod Nginx HTTPS sites with Letsencrypt front facing SSL certificates and need for older CentOS 6 OpenSSL/wget/curl clients to be able to connect to those Centmin Mod Nginx HTTPS sites. If you don't have such use case, there is no need to do a reissue for the updated preferred chain outlined at https://community.centminmod.com/threads/update-acmetool-sh-1-0-76.22024/.

Centmin Mod Nginx HTTPS sites with front facing Letsencrypt SSL certificates may have the old longer DST Root CA X3 chain which has expired and on older client devices, the web browser like Chrome might report insecure connections to your sites:

Code (Text):

NET::ERR_CERT_AUTHORITY_INVALID
NET::ERR_CERT_DATE_INVALID

Solution here is to update Centmin Mod 123.09beta01 via cmupdate command and then reissue your Letsencrypt SSL certificate for your domains using the new default shorter chain ISRG Root X1 via SSH command line using acmeotool.sh addon wrapper script. Replace yourdomain.com with your Centmin Mod Nginx site’s domain name or subdomain name. The reissue-only option will only touch your existing Centmin Mod Nginx site’s SSL certificate configuration leaving the rest of your Nginx HTTPS vhost configuration intact.

Code (Text):

cmupdate
/usr/local/src/centminmod/addons/acmetool.sh reissue-only yourdomain.com live

Setting Preferred Chain System Wide with acme.sh

Update October 4, 2021: Neil has updated acme.sh client to v3.0.1 to support setting the preferred chain flag at a system wide level instead of requiring it to be passed as a --preferred-chain flag only on the command line for acme.sh issuances. So you don’t need to use acmetool.sh reissue-only one at a time for each domain. The below SSH commands, will first upgrade acme.sh client to a version that supports the new system wide flag i.e. v3.0.1+ and then set the default preferred chain and then reissue and renew all acme.sh site SSL certificates with the new preferred shorter chain for ISRG Root X1.

Code (Text):

acme.sh --upgrade
acme.sh --set-default-chain --preferred-chain "ISRG" --server letsencrypt
acme.sh --renewAll --force

 

ZeroSSL SSL Certificates Alternative To Letsencrypt SSL Certificates

The other option is to use a different free SSL certificate provider other than Letsencrypt temporarily like Buypass or ZeroSSL until Letsencrypt issues are sorted out.

For the last 5yrs, acmetool.sh addon wrapper script has been using Neil Pang’s acme.sh client as the underlying tool to issue and obtain free Letsencrypt certificates. Over time acme.sh client has added support for other free ACME compatible CA SSL providers like Buypass and ZeroSSL. I choose to use ZeroSSL which is provided and uses Sectigo (Comodo) CA Root certificate chain as it supports free wildcard SSL certificates and doesn’t have any rate limiting for SSL certificate issuance.

ZeroSSL optionally requires you to register an account with at ZeroSSL.com first to obtain the EAB credentials via https://app.zerossl.com/developer that you need to register so that acme.sh client and thus acmetool.sh addon for Centmin Mod’s automated Nginx HTTPS site creation to issue free ZeroSSL SSL certificates instead of Letsencrypt SSL certificates. Or you can just pass your email address on command line to register with ZeroSSL and automatically obtain and register your EAB credentials which end up being saved to the configuration file at /root/.acme.sh/ca/acme.zerossl.com/v2/DV90/ca.conf.

Code (Text):

ls -lah /root/.acme.sh/ca/
total 0
drwxr-xr-x 4 root root  66 Sep 26 00:39 .
drwx------ 9 root root 233 Sep 30 23:43 ..
drwxr-xr-x 3 root root  23 Sep 26 00:06 acme-v02.api.letsencrypt.org
drwxr-xr-x 3 root root  16 Sep 26 00:39 acme.zerossl.com

Steps to switch Centmin Mod 123.09beta01 from using free Letsencrypt SSL certificates to using free ZeroSSL SSL certificates:

Step 1. Register an account at ZeroSSL.com and go to https://app.zerossl.com/developer to obtain the EAB credentials. This signup is actually optional as you can instead just provide your email address on the command line then you can actually skip Step 2:

Code (Text):

acme.sh --register-account -m myemail@example.com --server zerossl

Step 2. SSH login to your Centmin Mod server and register your EAB credentials with acme.sh client via the command line:

Code (Text):

acme.sh --register-account --server zerossl --eab-kid xxxxxxxxxxxx --eab-hmac-key xxxxxxxxx

Step 3. Configure Centmin Mod acmetool.sh addon to use ZeroSSL instead of Letsencrypt as default CA SSL certificate provider via the persistent configuration file /etc/centminmod/custom_config.inc variable you add:

Code (Text):

ACME_DEFAULT_CA='zerossl'

Update Oct 4, 2021: slight typo correction, the variable above was missing a closing single quote. Make sure it's ACME_DEFAULT_CA='zerossl'

If you want to switch back from ZeroSSL to Letsencrypt defaults you can remove that variable from the persistent configuration file /etc/centminmod/custom_config.inc or you can specifically set it to the already default value

Code (Text):

ACME_DEFAULT_CA='letsencrypt'

The acmetool.sh addon will pickup whichever setting value you have for the variable set in the persistent configuration file /etc/centminmod/custom_config.inc which will override the default settings.

Step 4. Reissuing existing Nginx site’s SSL certificates using ZeroSSL instead of Letsencrypt as CA provider.

Once you have switched to ZeroSSL defaults from above step 1-3, you will need to reissue your SSL certificates for existing Centmin Mod Nginx sites on your server. You do this via acmetool.sh addon wrapper script’s reissue-only option specifying your existing already created Centmin Mod Nginx site’s domain name (without the www) or subdomain name. Replace yourdomain.com with your Centmin Mod Nginx site’s domain name or subdomain name. The reissue-only option will only touch your existing Centmin Mod Nginx site’s SSL certificate configuration leaving the rest of your Nginx HTTPS vhost configuration intact.

Code (Text):

/usr/local/src/centminmod/addons/acmetool.sh reissue-only yourdomain.com live

You can then run acmetool.sh checkdates option to list all SSL certificates issued and configured at Nginx level for the current Centmin Mod server. Example below:

Code (Text):

/usr/local/src/centminmod/addons/acmetool.sh checkdates
output

Code (Text):

/usr/local/src/centminmod/addons/acmetool.sh checkdates
----------------------------------------------
nginx installed
----------------------------------------------

/usr/local/nginx/conf/ssl/zerossl.domain.com/zerossl.domain.com-acme.cer
SHA1 Fingerprint=06FE84519E09ACB75BBE11EDF26F7D41D0Bxxxxx
certificate expires in 83 days on 25 Dec 2021

/usr/local/nginx/conf/ssl/letsencrypt.domain.com/letsencrypt.domain.com-acme.cer
SHA1 Fingerprint=423A55D99E8BEEBBC4C42C82E2C8683684Cxxxxx
certificate expires in 87 days on 29 Dec 2021

----------------------------------------------
acme.sh obtained
----------------------------------------------

/root/.acme.sh/zerossl.domain.com/zerossl.domain.com.cer
SHA1 Fingerprint=06FE84519E09ACB75BBE11EDF26F7D41D0Bxxxxx
[ below certifcate transparency link is only valid ~1hr after issuance ]
https://crt.sh/?sha1=06FE84519E09ACB75BBE11EDF26F7D41D0Bxxxxx
certificate expires in 83 days on 25 Dec 2021

/root/.acme.sh/letsencrypt.domain.com/letsencrypt.domain.com.cer
SHA1 Fingerprint=423A55D99E8BEEBBC4C42C82E2C8683684Cxxxxx
[ below certifcate transparency link is only valid ~1hr after issuance ]
https://crt.sh/?sha1=423A55D99E8BEEBBC4C42C82E2C8683684Cxxxxx
certificate expires in 87 days on 29 Dec 2021

 

I've also posted a blog post summary of how Centmin Mod has handled this at https://blog.centminmod.com/2021/10...oot-ca-x3-certificate-expiration-on-centos-7/

 

Cloudflare Universal SSL Edge Certificates Switch From Letsencrypt To Digicert CA

The Letsencrypt DST Root CA X3 expiration on September 30, 2021 may also impact Cloudflare orange cloud proxy enabled users as Cloudflare's Universal SSL provides free SSL certificates through 2 CA SSL providers, Digicert or Letsencrypt.

For Cloudflare Universal SSL users who have Cloudflare edge certificates issued by Letsencrypt, their web site's visitors on very old browsers or devices i.e. Ubuntu 16, iOS <10, Android <7.1.1, Windows XP/7 or macOS <10.12.1 their web browsers like Chrome may report that the connection may not be secure or may not be private with the following errors: NET::ERR_CERT_AUTHORITY_INVALID or NET::ERR_CERT_DATE_INVALID.

There are a number of solutions for this:

1. Contact Cloudflare tech support and request that they switch your Cloudflare Universal SSL edge certificates from Letsencypt CA provided to Digicert CA provided SSL certificates.
2. Upgrade to Cloudflare Advanced Certificate Management (ACM) product at $10/month and you can create your own custom Cloudflare edge SSL certificate and choose Digicert as your CA provider.
3. Or you can switch your Cloudflare Universal SSL edge certificates from Letsencrypt to Digicert CA provided via the Cloudflare API via the below curl command ran while logged into your SSH session on your server or local computer (Linux).

There are 4 commands below, where last curl command is a single line command over 5 lines. You need to populate 3 session variables:

• cfzoneid variable replace value with your Cloudflare domain's zone id (found on Cloudflare dashboard overview page's bottom right side column)
• cfemail variable replace value Cloudflare account's registered email address
• cfglobalkey variable replace value with your Cloudflare account's Global API Key found at https://dash.cloudflare.com/profile/api-tokens

Code (Text):

cfzoneid='your_domain_zone_id'
cfemail='your_cloudflare_account_email'
cfglobalkey='your_cloudflare_account_global_api_key'

curl -4sX PATCH "https://api.cloudflare.com/client/v4/zones/$cfzoneid/ssl/universal/settings" \
     -H "X-Auth-Email: $cfemail" \
     -H "X-Auth-Key: $cfglobalkey" \
     -H "Content-Type: application/json" \
     --data '{"certificate_authority":"digicert"}' | tee /root/centminlogs/cf-universal-ssl-ca-switch.log

After populating SSH session variables and running the curl command, you should have the response from Cloudflare API outputted to your screen (example below) as well as saved copy in JSON format at /root/centminlogs/cf-universal-ssl-ca-switch.log. If you're not using Centmin Mod LEMP stack, change /root/centminlogs directory location appropriately.

Example curl output

Code (Text):

{"result":{"enabled":true,"certificate_authority":"digicert"},"success":true,"errors":[],"messages":[]}

Using jq to show pretty JSON formatted saved output of file at /root/centminlogs/cf-universal-ssl-ca-switch.log

Code (Text):

jq -r . /root/centminlogs/cf-universal-ssl-ca-switch.log
{
  "result": {
    "enabled": true,
    "certificate_authority": "digicert"
  },
  "success": true,
  "errors": [],
  "messages": []
}