Troubleshoot Invalid SSL certificate error 526, 403, 401

EDIT: Problem was cloudflare bot fight mode. Solution in comment #7.

I have authenticated origin pull enabled, and I'm using Cloudflare. I woke up today to learn that my website has been down for the past 8 hours.
I think the original cert was created on my Centos7 server via

Code:

4. issue live cert with HTTPS default

And I transferred it to the new Alma9 server during the migration. Perhaps I missed a step or there's a missing step?

I see there's a "Centmin Mod Self-Signed SSL Fallback"? Do I need to manually enable this (how?) so that my websites won't go down if the letsencrypt renewal fails? It doesn't say here.

It would be nice if there were some warning that the SSL renew failed and will expire soon.

Test result for forum.humanmicrobiome.info: Warning

SSL Server Test: forum.humanmicrobiome.info (Powered by Qualys SSL Labs)

Acmetool sh debug log 6.63 MB file on MEGA

Code:

grep acme /var/log/cron* | sed -e "s|$(hostname -s)|host|g"

/var/log/cron:Nov 22 00:07:01 host CROND[359462]: (root) CMD ("/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh" >/dev/null 2>&1)
/var/log/cron:Nov 22 00:07:17 host CROND[359439]: (root) CMDEND ("/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh" >/dev/null 2>&1)
/var/log/cron-20241027:Oct 20 00:07:01 host CROND[282934]: (root) CMD ("/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh" >/dev/null 2>&1)

The one below that says "expires in -265 days" is still working.

Code:

echo y | /usr/local/src/centminmod/addons/acmetool.sh checkdates
----------------------------------------------
nginx installed
----------------------------------------------

/usr/local/nginx/conf/ssl/forum.humanmicrobiome.info/forum.humanmicrobiome.info-acme.cer
SHA1 Fingerprint=6A178B***2DB7EEB
certificate expires in 0 days on 22 Nov 2024

/usr/local/nginx/conf/ssl/listm.humanmicrobes.org/listm.humanmicrobes.org-acme.cer
SHA1 Fingerprint=BEE29A***7C33E
certificate expires in -265 days on 2 Mar 2024
----------------------------------------------
acme.sh obtained
----------------------------------------------

/root/.acme.sh/forum.humanmicrobiome.info/forum.humanmicrobiome.info.cer
SHA1 Fingerprint=6A178B1***B7EEB
[ below certifcate transparency link is only valid ~1hr after issuance ]
https://crt.sh/?sha1=6A178B***DB7EEB
certificate expires in 0 days on 22 Nov 2024
Letsencrypt validation method: Le_Webroot='/home/nginx/domains/forum.humanmicrobiome.info/public'

/root/.acme.sh/listm.humanmicrobes.org/listm.humanmicrobes.org.cer
SHA1 Fingerprint=BEE29A***87C33E
[ below certifcate transparency link is only valid ~1hr after issuance ]
https://crt.sh/?sha1=BEE29***D487C33E
certificate expires in -265 days on 2 Mar 2024
Letsencrypt validation method: Le_Webroot='/home/nginx/domains/listm.humanmicrobes.org/public'

Code:

"/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh"
[Fri Nov 22 09:56:32 AM PST 2024] ===Starting cron===
[Fri Nov 22 09:56:32 AM PST 2024] Renew: 'forum.humanmicrobiome.info'
[Fri Nov 22 09:56:32 AM PST 2024] Renew to Le_API=https://acme-v02.api.letsencrypt.org/directory
[Fri Nov 22 09:56:33 AM PST 2024] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Fri Nov 22 09:56:33 AM PST 2024] Run pre hook:'/usr/local/src/centminmod/tools/pre-acme-hooks.sh all-check forum.humanmicrobiome.info'
Nginx root path: /home/nginx/domains/forum.humanmicrobiome.info/public
Le_Webroot: /home/nginx/domains/forum.humanmicrobiome.info/public
The root paths match. Proceeding with the acme.sh operation.
[Fri Nov 22 09:56:33 AM PST 2024] Single domain='forum.humanmicrobiome.info'
[Fri Nov 22 09:56:36 AM PST 2024] Getting webroot for domain='forum.humanmicrobiome.info'
[Fri Nov 22 09:56:36 AM PST 2024] Verifying: forum.humanmicrobiome.info
[Fri Nov 22 09:56:37 AM PST 2024] Pending, The CA is processing your order, please just wait. (1/30)
[Fri Nov 22 09:56:40 AM PST 2024] Invalid status, forum.humanmicrobiome.info:Verify error detail:2606:4700:3033::6815:17b6: Invalid response from https://forum.humanmicrobiome.info/.well-known/acme-challenge/n953pICLOcAj_cdC7wHTklNxOW_dOOVakMrFl5CPAHM: 526
[Fri Nov 22 09:56:40 AM PST 2024] Please check log file for more details: /root/centminlogs/acmetool.sh-debug-log-290324-031134.log
[Fri Nov 22 09:56:41 AM PST 2024] Error renew forum.humanmicrobiome.info.
[Fri Nov 22 09:56:41 AM PST 2024] Renew: 'listm.humanmicrobes.org'
[Fri Nov 22 09:56:41 AM PST 2024] Renew to Le_API=https://acme-v02.api.letsencrypt.org/directory
[Fri Nov 22 09:56:42 AM PST 2024] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Fri Nov 22 09:56:42 AM PST 2024] Run pre hook:'/usr/local/src/centminmod/tools/pre-acme-hooks.sh all-check listm.humanmicrobes.org'
Nginx root path: /home/nginx/domains/listm.humanmicrobes.org/public
Le_Webroot: /home/nginx/domains/listm.humanmicrobes.org/public
The root paths match. Proceeding with the acme.sh operation.
[Fri Nov 22 09:56:42 AM PST 2024] Single domain='listm.humanmicrobes.org'
[Fri Nov 22 09:56:45 AM PST 2024] Getting webroot for domain='listm.humanmicrobes.org'
[Fri Nov 22 09:56:45 AM PST 2024] Verifying: listm.humanmicrobes.org
[Fri Nov 22 09:56:46 AM PST 2024] Pending, The CA is processing your order, please just wait. (1/30)
[Fri Nov 22 09:56:49 AM PST 2024] Invalid status, listm.humanmicrobes.org:Verify error detail:2606:4700:3037::ac43:b476: Invalid response from https://listm.humanmicrobes.org/.well-known/acme-challenge/dw743mJ-Zt4ORAAn3ykjOILnjAB5WXdntvMlGKSQXVg: 401
[Fri Nov 22 09:56:49 AM PST 2024] Please check log file for more details: /root/centminlogs/acmetool.sh-debug-log-290324-031134.log
[Fri Nov 22 09:56:50 AM PST 2024] Error renew listm.humanmicrobes.org.
[Fri Nov 22 09:56:50 AM PST 2024] ===End cron===

Code:

echo | openssl s_client -connect forum.humanmicrobiome.info:443

Code:

find /root/centminlogs/ -type f -name 'acme*.log' -printf '%TY-%Tm-%Td %TH:%TM:%TS %p\n' | sort | awk '{print $3}' | xargs -d '\n' grep -i 'errordetail'

Gives a bunch of

Code:

errordetail='2606:4700:3033::6815:17b6: Invalid response from https://forum.humanmicrobiome.info/.well-known/acme-challenge/n953pICLOcAj_cdC7wHTklNxOW_dOOVakMrFl5CPAHM: 526'

Maybe it's the same issue as this thread? https://community.centminmod.com/th...-reissue-verify-error-invalid-response.20698/

If so, it sounds tedious to do for all of my domains, and I don't understand why it would happen in the first place, and thus how to prevent it.

I just saw in another thread you say not to post the full output of "checkdates", but in the main instructions thread there's no mention of that.

I see in another thread you say to check "staticfiles.conf". My forum has it enabled, my listmonk site has it disabled due to it causing issues.

---

Per Eva's suggestion here:

Code:

curl -I http://forum.humanmicrobiome.info
HTTP/1.1 403 Forbidden

Code:

curl -I https://forum.humanmicrobiome.info
HTTP/2 403

Forum nginx config <snip>
Listmonk nginx config <snip>

 

Per Eva's comment here, it seems that the ".well-known" part of the staticfiles.conf is important. So I created a new file called "letsencrypt.conf" in /usr/local/nginx/conf.

With contents taken from the staticfiles.conf

Code:

    # prepare for letsencrypt
    # https://community.centminmod.com/posts/17774/
    location ~ /.well-known { location ~ /.well-known/acme-challenge/(.*) { more_set_headers    "Content-Type: text/plain"; } }

And then in both places in the domain.ssl.config where I commented out the static files lines:

Code:

 #  include /usr/local/nginx/conf/staticfiles.conf;

I added:

Code:

   include /usr/local/nginx/conf/letsencrypt.conf;

However, that for sure doesn't solve the issue, and having those commented out might be the reason my listmonk domain is still working with an expired ssl cert?

I tried commenting out those lines in my forum config, then restarting nginx, but it didn't fix the 526 issue.

 

I commented out these lines:

Code:

  # cloudflare authenticated origin pull cert community.centminmod.com/threads/13847/
 # ssl_client_certificate /usr/local/nginx/conf/ssl/cloudflare/forum.humanmicrobiome.info/origin.crt;
#  ssl_verify_client on;

And ran ngxrestart.

Curling the domain still gets 403. And this command still results in the same errors:

Code:

"/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh"

 

The WAF tab on the left side of the main dashboard (Account-level web application firewall (WAF)) is not activated, as it requires a purchase.

Nothing stands out to me under "analytics & logs" or "security center".

I noticed that "bot fight mode" was enabled for my forum (not for my listmonk) and disabling it allows me to curl my domain now.
But this command still results in the 526 and 401 errors:

Code:

"/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh"

Under "domain -> rules" I don't have any custom rules enabled besides these three cache rules that come with the digitalpoint cloudflare xenforo addon. I disabled them and still get the same errors.

I have the two page rules that also come with that addon. I disabled them and still get the same errors.

And my listmonk site is on another domain, and doesn't have any of that and is still failing (with 401). Maybe the 401 will resolve if I uncomment the letsencrypt lines but I'm scared to do so as I don't want another website going down.

ChatGPT told me I should look for the "/.well-known/" folder in "public" and/or try to create it there. Indeed it's not there, but I'm assuming it gets created temporarily?

I reran the "lets debug" and instead of 403 it's now 526 Test result for forum.humanmicrobiome.info: Warning.

 

Ok, so it looks like the problem was the "bot fight mode", and the solution is to temporarily change the Cloudflare setting from "full strict" to "full". Troubleshooting Cloudflare 5XX errors | Cloudflare Support docs

Uncommenting the "letsencrypt" line in my listmonk site's config also solved the 401 error and renewed the certificate.

The takeaway for me is that there needs to be an alert to solve this issue before it gets to the point of the site going down for days.

I can't remember if it was with centmin mod, but there was a time I would get emails about the status of my letsencrypt certificate status.

 

I think I was getting them directly from letsencrypt at one point. Maybe on my old server where I had it through docker. So there should be a way to do that.

But it's easy enough to edit

Code:

/root/.acme.sh/account.conf

And add:

Code:

SAVED_MAIL_BIN='mail'
SAVED_MAIL_FROM='root@localhost'
SAVED_MAIL_TO='root@localhost' # or whatever email you want
NOTIFY_HOOK='mail'
NOTIFY_LEVEL='2' # can temporarily set to 3 for testing
NOTIFY_MODE='0'

Then run this to test it:

Code:

"/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh"  

I would suggest adding this to one of your "get started" guides. I added it to mine.

Thanks for the help!