Issue with acmetool.sh renewal

I've had auto renewals working for a couple years and now all of a sudden my sites wouldn't load due to expired certs.

---

One domain seemed to renew properly when I tried manually, but the other simply won't work. I will upload the log file. "example1.com" is my now working domain and "example2.com" is the broken one that won't renew.

Edit: Well this is NOT the error I was getting, but I tried running

Code:

"/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh"

and now it's throwing the error:

Code:

Create new order error. Le_OrderFinalize not found. {
  "type": "urn:ietf:params:acme:error:rateLimited",
  "detail": "Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/",
  "status": 429
}

 

LetsDebug log: Let's Debug Test result for example2.com using http-01 - Pastebin.com
Note: I have always used my domains with LetsEncrypt + Cloudflare Full (strict) SSL. For these test I have it paused and in development mode.


I believe I originally issued the cert on an already created vhost, but it was such I long time ago I could be wrong.

acmetool.sh-debug-log-160821-160808 - Pastebin.com
acmesh-issue_160821-160808 - Pastebin.com

If I run

Code:

./acmetool.sh issue example2.com live

I get the linked logs above and this error in the console:

Code:

example2.com:Verify error:Fetching http://example2.com/.well-known/acme-challenge/AvykzxZtgZLC1kLgPhv06BK0fY09nZEO-dv-ykvUzuE: Redirect loop detected

checkdates command outputs:

Code:

/root/.acme.sh/example2.com/product-testers.com.cer
SHA1 Fingerprint=3BC1C1BAB386F7218CC579BDC1D7752784515348
[ below certifcate transparency link is only valid ~1hr after issuance ]
https://crt.sh/?sha1=3BC1C1BAB386F7218CC579BDC1D7752784515348
certificate expires in -22 days on 25 Jul 2021

and the port 443 test:

Code:

CONNECTED(00000003)
139876616435600:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:769:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 289 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1629130573
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)

cron looks like it is running regularly every day:

Code:

/var/log/cron-20210815:Aug 13 00:00:01 host CROND[11086]: (root) CMD ("/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh" > /dev/null)
/var/log/cron-20210815:Aug 14 00:00:01 host CROND[16668]: (root) CMD ("/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh" > /dev/null)
/var/log/cron-20210815:Aug 15 00:00:01 host CROND[22360]: (root) CMD ("/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh" > /dev/null)

 

Thanks for the reply

I only have example2.com.ssl.conf for that domain. There's no example2.com.conf
Contents:

Code:

#x# HTTPS-DEFAULT
 server {
 
   server_name example2.com www.example2.com;
   return 302 https://example2.com$request_uri;
   root /home/nginx/domains/example2.com/public;
   include /usr/local/nginx/conf/staticfiles.conf;
 }

 server {
   listen 80;
   listen [::]:80;
   server_name example2.com www.example2.com;
   return 301 https://example2.com$request_uri;
 }
 
 server {
  listen 80;
  listen [::]:80;
  server_name example2.com www.example2.com;
  return 404;
  }

server {
  listen 443 ssl http2;
  listen [::]:443 ssl http2;
  server_name example2.com www.example2.com;

  include /usr/local/nginx/conf/ssl/example2.com/example2.com.crt.key.conf;
  include /usr/local/nginx/conf/ssl_include.conf;

  #http2_max_field_size 16k;
  #http2_max_header_size 32k;
  # dual cert supported ssl ciphers
  ssl_ciphers     EECDH+CHACHA20-draft:EECDH+CHACHA20:EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA256:EECDH+ECDSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+SHA384:EECDH+AES128:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!CAMELLIA;
  ssl_prefer_server_ciphers   on;
  #add_header Alternate-Protocol  443:npn-spdy/3;
  add_header Strict-Transport-Security "max-age=31536000;";
  #add_header X-Frame-Options SAMEORIGIN;
  #add_header X-Xss-Protection "1; mode=block" always;
  #add_header X-Content-Type-Options "nosniff" always;
  #spdy_headers_comp 5;
  ssl_buffer_size 1369;
  ssl_session_tickets on;
 
  # enable ocsp stapling
  #resolver 8.8.8.8 8.8.4.4 valid=10m;
  #resolver_timeout 10s;
  #ssl_stapling on;
  #ssl_stapling_verify on;

# ngx_pagespeed & ngx_pagespeed handler
include /usr/local/nginx/conf/pagespeed.conf;
include /usr/local/nginx/conf/pagespeedhandler.conf;
include /usr/local/nginx/conf/pagespeedstatslog.conf;

  # limit_conn limit_per_ip 16;
  # ssi  on;

  access_log /home/nginx/domains/example2.com/log/access.log combined buffer=256k flush=60m;
  error_log /home/nginx/domains/example2.com/log/error.log;

  root /home/nginx/domains/example2.com/public;

  location / {

# block common exploits, sql injections etc
#include /usr/local/nginx/conf/block.conf;

  # Enables directory listings when index file not found
  #autoindex  on;

  # Shows file listing times as local time
  #autoindex_localtime on;

  # Enable for vBulletin usage WITHOUT vbSEO installed
  # More example Nginx vhost configurations at
  # http://centminmod.com/nginx_configure.html
  #try_files    $uri $uri/ /index.php;

  }
 
  # enforce NO www
    if ($host ~* ^www\.(.*))
    {
    set $host_without_www $1;
    rewrite ^/(.*)$ $scheme://$host_without_www/$1 permanent;
    }

# unless the request is for a valid file, send to bootstrap
    if (!-e $request_filename)
    {
    rewrite ^(.+)$ /index.php?q=$1 last;
    }

  include /usr/local/nginx/conf/staticfiles.conf;
  include /usr/local/nginx/conf/php.conf;
  include /usr/local/nginx/conf/drop.conf;
  #include /usr/local/nginx/conf/errorpage.conf;
  include /usr/local/nginx/conf/vts_server.conf;
}

I currently have Cloudflare still in development mode and switched from Full (strict) SSL to OFF.
In the order of the 4 commands you listed:

Code:

[07:26][root@atlas ~]# curl -I https://example2.com
HTTP/1.1 301 Moved Permanently
Date: Tue, 17 Aug 2021 07:26:36 GMT
Connection: keep-alive
Cache-Control: max-age=3600
Expires: Tue, 17 Aug 2021 08:26:36 GMT
Location: http://example2.com/
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=0G1KKFbEsXp4x%2BycsxmGBJBFFlZi3%2F8riWea5Axc%2BN0kuln8AcD%2FB%2Fz%2Bwro5bQ7n3%2FHvuJwyc0decGcBBV7dNNOsTr3jG61%2BQtn9S7QYoIiRbtP0UK0rmRBqV6LRzUK7%2FLekASKB"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 68012cd89c43e02b-DFW

Code:

[07:26][root@atlas ~]# curl -I https://www.example2.com
HTTP/1.1 301 Moved Permanently
Date: Tue, 17 Aug 2021 07:27:04 GMT
Connection: keep-alive
Cache-Control: max-age=3600
Expires: Tue, 17 Aug 2021 08:27:04 GMT
Location: http://www.example2.com/
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=OX8BZbA4TmJpAPvxx2SUJyNLfWOyJt4nP7VFM1QQoK6jrJRorjT6aTfVARM4t9vl%2F7G7Ux0SEqY2EsDf9TuIA2xdCXSohOVTLXmAf9MkhKomJi7FZ0%2FcCfMDG53nsoCIWDXIZEOcCCiw%2BQ%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 68012d879cbf0e5e-DFW

Code:

[07:27][root@atlas ~]# curl -I http://example2.com
HTTP/1.1 301 Moved Permanently
Date: Tue, 17 Aug 2021 07:27:53 GMT
Connection: keep-alive
Cache-Control: max-age=3600
Expires: Tue, 17 Aug 2021 08:27:53 GMT
Location: https://example2.com/
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=LFzNqqBG6BdaaS7%2F8BWjiQoIf1fim7iXgb8utllunm9vKgwK4CFvBvZTTy96vwCgPicIsY6bJ%2BkRj0qXssWUMcJz92873aPS2fmhXjju%2FOB1Tp3zwyQCgfdqJJZ0NooAE2Xy1AAI"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 68012eb91d37285f-DFW

Code:

[07:28][root@atlas ~]# curl -I http://www.example2.com
HTTP/1.1 521
Date: Tue, 17 Aug 2021 07:28:26 GMT
Content-Length: 0
Connection: keep-alive
cache-control: no-store, no-cache
CF-Cache-Status: DYNAMIC
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=bbV5pQ9N1hEv0s%2BGVU2mJOsUvdDQjAUh78w%2F4mtRgssk3hj0%2Fyu3hwtiy6VQN3XK8aA%2F%2B0AUIbEGmdgkDpbhqqpDMKHt4M44SkibtqYwcwSLGXXdY2H%2FYf9l3v9Opr8i8oGUDvJJMTGpMw%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 68012f884c340f22-DFW

 

OK so my ssl.conf file now looks like:

Code:

#x# HTTPS-DEFAULT
 server {
   listen 80;
   listen [::]:80;
   server_name example2.com www.example2.com;
   return 302 https://example2.com$request_uri;
   root /home/nginx/domains/example2.com/public;
   include /usr/local/nginx/conf/staticfiles.conf;
 }

server {
  listen 443 ssl http2;
  listen [::]:443 ssl http2;
  server_name example2.com www.example2.com;

  include /usr/local/nginx/conf/ssl/example2.com/example2.com.crt.key.conf;
  include /usr/local/nginx/conf/ssl_include.conf;

  #http2_max_field_size 16k;
  #http2_max_header_size 32k;
  # dual cert supported ssl ciphers
  ssl_ciphers     EECDH+CHACHA20-draft:EECDH+CHACHA20:EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA256:EECDH+ECDSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+SHA384:EECDH+AES128:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!CAMELLIA;
  ssl_prefer_server_ciphers   on;
  #add_header Alternate-Protocol  443:npn-spdy/3;
  add_header Strict-Transport-Security "max-age=31536000;";
  #add_header X-Frame-Options SAMEORIGIN;
  #add_header X-Xss-Protection "1; mode=block" always;
  #add_header X-Content-Type-Options "nosniff" always;
  #spdy_headers_comp 5;
  ssl_buffer_size 1369;
  ssl_session_tickets on;
 
  # enable ocsp stapling
  #resolver 8.8.8.8 8.8.4.4 valid=10m;
  #resolver_timeout 10s;
  #ssl_stapling on;
  #ssl_stapling_verify on;

# ngx_pagespeed & ngx_pagespeed handler
include /usr/local/nginx/conf/pagespeed.conf;
include /usr/local/nginx/conf/pagespeedhandler.conf;
include /usr/local/nginx/conf/pagespeedstatslog.conf;

  # limit_conn limit_per_ip 16;
  # ssi  on;

  access_log /home/nginx/domains/example2.com/log/access.log combined buffer=256k flush=60m;
  error_log /home/nginx/domains/example2.com/log/error.log;

  root /home/nginx/domains/example2.com/public;

  location / {

# block common exploits, sql injections etc
#include /usr/local/nginx/conf/block.conf;

  # Enables directory listings when index file not found
  #autoindex  on;

  # Shows file listing times as local time
  #autoindex_localtime on;

  # Enable for vBulletin usage WITHOUT vbSEO installed
  # More example Nginx vhost configurations at
  # http://centminmod.com/nginx_configure.html
  #try_files    $uri $uri/ /index.php;

  }
 
  # enforce NO www
    if ($host ~* ^www\.(.*))
    {
    set $host_without_www $1;
    rewrite ^/(.*)$ $scheme://$host_without_www/$1 permanent;
    }

# unless the request is for a valid file, send to bootstrap
    if (!-e $request_filename)
    {
    rewrite ^(.+)$ /index.php?q=$1 last;
    }

  include /usr/local/nginx/conf/staticfiles.conf;
  include /usr/local/nginx/conf/php.conf;
  include /usr/local/nginx/conf/drop.conf;
  #include /usr/local/nginx/conf/errorpage.conf;
  include /usr/local/nginx/conf/vts_server.conf;
}

LetsDebug spits out those same 4x bad redirect errors

Code:

BadRedirect
ERROR
Sending an ACME HTTP validation request to example2.com results in an unacceptable redirect. This is most likely a misconfiguration of your web server or your web application.
Too many (10) redirects, last redirect was to: http://example2.com/.well-known/acme-challenge/letsdebug-test

Code:

IssueFromLetsEncrypt
ERROR
A test authorization for example2.com to the Let's Encrypt staging service has revealed issues that may prevent any certificate for this domain being issued.
Fetching http://example2.com/.well-known/acme-challenge/uNVve8cplxsTy5hhnDsPHnN7AOjW82Y1W9vltvXlIRs: Redirect loop detected

I still have the expired cert when I run:

Code:

echo y | /usr/local/src/centminmod/addons/acmetool.sh checkdates

Is the correct way to force a renew before 60 days?:

Code:

./acmetool.sh reissue example2.com lived

 

No luck after commenting that code out
Still getting the redirect loop in Letsdebug.

Would it be easier to completely remove SSL from the domain? Maybe re-issue is broken, or will a new cert also have this problem?

 

Code:

[17:12][root@atlas ~]# curl -I https://example2.com
HTTP/1.1 301 Moved Permanently
Date: Mon, 23 Aug 2021 17:12:35 GMT
Connection: keep-alive
Cache-Control: max-age=3600
Expires: Mon, 23 Aug 2021 18:12:35 GMT
Location: http://example2.com/
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=vAc%2BZEEFBr6AMGouuX11wAgRKJM5cj6GKdExOl0%2BCvw2XKt%2FhBqcasxXXEydIb%2BvfN57zqNIfCtRDtRO3l2p5v9fOnJziMR%2BtelDIaB1Is5jrIVNk0OM4yVenW%2FZ9%2BMEo2QMsf%2FZ"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 6835f7759fa5c812-DFW

Code:

[17:12][root@atlas ~]# curl -I https://www.example2.com
HTTP/1.1 301 Moved Permanently
Date: Mon, 23 Aug 2021 17:12:57 GMT
Connection: keep-alive
Cache-Control: max-age=3600
Expires: Mon, 23 Aug 2021 18:12:57 GMT
Location: http://www.example2.com/
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=9VtbwlTPaPoguOifL5ZJhd%2FC9nCc3ojpPPScrvmpDEnMTDJyxDfCMgfEs4f8F8f%2BQu7G1Aj9Cxzpjui4jEZCvODWtX2BEdOlHFSkKf05TE4HYuiRiNZHlnNunaZizAPDbvNAIoWMSKwyWQ%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 6835f800aeb00e62-DFW

Code:

[17:12][root@atlas ~]# curl -I http://example2.com
HTTP/1.1 301 Moved Permanently
Date: Mon, 23 Aug 2021 17:13:07 GMT
Connection: keep-alive
Cache-Control: max-age=3600
Expires: Mon, 23 Aug 2021 18:13:07 GMT
Location: https://example2.com/
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=5NrITMaQO89SXL1oMy8ryADF19KenPh%2BnoRVi%2F2Zx1bJmCTfhpRDTd3lmP4bCrjc2SK%2BpM85jflf9kNhSZKXt2L6u1v1yslvV72zoujjj5a75o8gvpy%2FpFMeMJVbcHZF2%2BS96qSL"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 6835f83e1fb80c0f-DFW

Code:

[17:13][root@atlas ~]# curl -I http://www.example2.com
HTTP/1.1 302 Found
Date: Mon, 23 Aug 2021 17:13:14 GMT
Content-Type: text/html
Connection: keep-alive
location: https://example2.com/
CF-Cache-Status: DYNAMIC
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=C1EO7u36W91LWq2u6VjcyiBBqr%2Fpq0WJlEltbT6rI7oYQ4OsSTZQ2kuROgeoYN06f3df8xBQot%2Fk5L%2B%2Bg7QzaoZ22rV%2B9uy2KcmubSzZjcMO%2FwvRJQd4a0X2EFUg44nA2JXPl0FDS3sZaw%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 6835f867a97b0ba7-DFW

 

nginx_pagespeed wasn't the issue (tried that first).

- I did have an "Always use HTTPS" Page Rule setup for: http://example2.com/*
- Tried re-issuing the cert while in both Cloudflare SSL Off and Full (Strict) modes + Development Mode

Not sure which exact step fixed it, but I disabled the Page Rule and put it into Full mode (not strict). This allowed me to reissue successfully and then I re-activated the Page Rule and Full (Strict). Seems to be working now, lets see what happens when the next cron is ready to run

Thanks for the help, and I hope that we documented it well enough so it can help someone else!