Want to subscribe to topics you're interested in?
Become a Member

Letsencrypt LE cert reissue failing

Discussion in 'Install & Upgrades or Pre-Install Questions' started by fly, Apr 18, 2023.

  1. fly

    fly Member

    109
    16
    18
    Jul 27, 2019
    Ratings:
    +28
    Local Time:
    11:30 PM
    Please fill in any relevant information that applies to you:
    • CentOS Version: CentOS 7 64bit
    • Centmin Mod Version Installed: 130.00beta01
    • Nginx Version Installed: i.e. 1.23.3
    • PHP Version Installed: 7.2.34
    • MariaDB MySQL Version Installed: 10.3
    • When was last time updated Centmin Mod code base ? : Today
    • Persistent Config: Do you have any persistent config file options set in /etc/centminmod/custom_config.inc ? You can check via this command:
      Code (Text):
      cat /etc/centminmod/custom_config.inc
      

      Post output in CODE tags.
    Code:
    LETSENCRYPT_DETECT='y'
    MARCH_TARGETNATIVE='n'
    
    I'm trying to reissue a cert for a domain (really trying to add a domain), but the reissue fails even if I only use the original domain. This is the command I'm using:

    Code:
     /root/.acme.sh/acme.sh --force --issue --days 60 -d domain.com -w /home/nginx/domains/domain.com/public -k 2048 --useragent centminmod-centos-acmesh-webroot --log /root/centminlogs/acmetool.sh-debug-log-domain.com.log --log-level 2
    
    The error I get back is a 404:
    Code:
    [Mon Apr 17 20:17:14 UTC 2023] domain.com:Verify error:1.2.3.4: Invalid response from https://domain.com/.well-known/acme-challenge/DAk5MBLstuffgoesheretoo: 404
    [Mon Apr 17 20:17:15 UTC 2023] Please check log file for more details: /root/centminlogs/acmetool.sh-debug-log-domain.com.log
    What am I missing here?

    edit: BTW, the log has nothing obvious in it to me.

     
  2. fly

    fly Member

    109
    16
    18
    Jul 27, 2019
    Ratings:
    +28
    Local Time:
    11:30 PM
  3. eva2000

    eva2000 Administrator Staff Member

    55,896
    12,281
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,868
    Local Time:
    1:30 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    First try running your intended SSL certificate domain through the letsdebug.net online testing tool to check for potential errors with HTTP-01 validation. If you created Centmin Mod 123.09beta01 or higher Nginx site with Letsencrypt via centmin.sh menu option 2, 22 or nv command line, you now also have an automatic letsdebug.net API check log saved at /root/centminlogs/letsdebug-yourdomain.com-${DT}.log where yourdomain.com is domain specified during nginx vhost creation and DT is date/timestamp. Inspecting the /root/centminlogs/letsdebug-yourdomain.com-${DT}.log log will also give you clues as to why letsencrypt SSL certificate issuance failed.

    How was the initial letsencrypt ssl certificate obtained ? Which method ?
    • Was the domain nginx vhost alreadying created prior or new domain nginx vhost site setup for first time ?
    • Via centmin.sh menu option 2, 22, /usr/bin/nv ?
    • If you ran centmin.sh menu option 2 or 22, which letsencrypt option did you select from
      Code (Text):
      -------------------------------------------------------------
      Setup full Nginx vhost + Wordpress + WP Plugins
      -------------------------------------------------------------
      
      Enter vhost domain name you want to add (without www. prefix): acme3.domain1.com
      
      Create a self-signed SSL certificate Nginx vhost? [y/n]: n
      Get Letsencrypt SSL certificate Nginx vhost? [y/n]: y
      
      You have 4 options:
      1. issue staging test cert with HTTP + HTTPS
      2. issue staging test cert with HTTPS default
      3. issue live cert with HTTP + HTTPS
      4. issue live cert with HTTPS default
      Enter option number 1-4: 1
      
    • Via addons/acmetool.sh ? which specific command ? examples
      Code (Text):
      ./acmetool.sh issue acme.domain.com
      
      Code (Text):
      ./acmetool.sh issue acme.domain.com live
      
      Code (Text):
      ./acmetool.sh issue acme.domain.com d
      
      Code (Text):
      ./acmetool.sh issue acme.domain.com lived
      
    • What was order of steps you did ? Did you run centmin.sh menu option 2 first with letsencrypt ? Then did you run addons/acmetool.sh afterwards ?

    Centmin Mod Self-Signed SSL Fallback



    If you're seeing a Centmin Mod's self-signed ssl certificate instead of letsencrypt ssl certificate, then that's acmetool.sh and centminmod's fallback if letsencrypt verification fails to obtain letsencrypt ssl cert, it falls back to centmin mod self-signed ssl certificate on https port 443 side so to preserve the https nginx vhost

    Troubleshooting



    There are various steps you can do to troubleshoot failed letsencrypt issuances, renews, reissues etc.
    • acmetool.sh logs all command line or shell menu runs to log files at /root/centminlogs. To troubleshoot, copy the contents of the log run and post contents of log to pastebin.com or gist.github.com and share link in this thread. To find the log list the logs in ascending date order
      Code (Text):
      ls -lahrt /root/centminlogs
      .
    • You can also do a quick grep filter on all previous and current acmetool.sh runs of the underlying acme.sh client for errors listed in errordetails field of each log using the command below:
      Code (Text):
      find /root/centminlogs/ -type f -name 'acme*.log' -printf '%TY-%Tm-%Td %TH:%TM:%TS %p\n' | sort | awk '{print $3}' | xargs -d '\n' grep -i 'errordetail'
      
    • For direct acmetool.sh runs, there should be a 2nd & 3rd & 4th log in format /root/centminlogs/centminmod_${DT}_nginx_addvhost_nv.log and /root/centminlogs/acmetool.sh-debug-log-$DT.log and /root/centminlogs/acmesh-issue_*.log or /root/centminlogs/acmesh-reissue_*.log which would need to be included via separate pastebin.com or gist.github.com post.
    • Enable acmetool.sh debug mode. In persistent config file at /etc/centminmod/custom_config.inc (create it if doesn't exist) add and enable acmetool.sh debug mode which gives much more verbose letsencrypt issuance process information when you re-run acmetool.sh or centmin.sh menu options 2, 22 or /usr/bin/nv command lines.
      Code (Text):
      ACMEDEBUG='y'
    If acme.sh auto renewals didn't happen, check output for the following commands
    Code (Text):
    grep acme /var/log/cron* | sed -e "s|$(hostname -s)|host|g"
    

    Code (Text):
    echo y | /usr/local/src/centminmod/addons/acmetool.sh checkdates
    

    Code (Text):
    "/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh"
    

    Code (Text):
    echo | openssl s_client -connect yourdomain.com:443
    

    Without the answers to above questions and logs, there is nothing to help troubleshoot.

    SSLLabs Test



    Also run your HTTPS domain site through SSLLabs tester at SSL Server Test (Powered by Qualys SSL Labs) if it says untrusted SSL cert and prompts to continue the test, continue the test.

    Cloudflare



    If you use Cloudflare, instead of the default Letsencrypt web root validation, you can use Cloudflare's DNS API for Letsencrypt DNS validation for your domain. See the outline at bottom of page at Letsencrypt Free SSL Certificates
     
  4. fly

    fly Member

    109
    16
    18
    Jul 27, 2019
    Ratings:
    +28
    Local Time:
    11:30 PM
    Thanks for the reply!

    * letsdebug says that everything is fine (even in the log in the centminlogs folder).

    * Cert was originally created via option 2, with the acme option of 4 (HTTPS default)

    * The acme errors all show the same thing as listed above: Invalid response 404

    * As far as I can tell, debug doesn't show anything more
     
  5. eva2000

    eva2000 Administrator Staff Member

    55,896
    12,281
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,868
    Local Time:
    1:30 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    When you create a new nginx vhost domain via centmin.sh menu option 2 or menu option 22 or via /usr/bin/nv cli command line, you will create the Nginx vhost files and directories. You will get an outputted the path location where it will create the domain name's vhost conf file named newdomain.com.conf (and newdomain.com.ssl.conf if you selected yes to self signed SSL)
    • Nginx vhost conf path will be at /usr/local/nginx/conf/conf.d/newdomain.com.conf
    • Nginx HTTP/2 SSL vhost conf path will be at /usr/local/nginx/conf/conf.d/newdomain.com.ssl.conf
    • Nginx Self-Signed SSL Certificate Directory at /usr/local/nginx/conf/ssl/newdomain.com
    • Vhost public web root will be at /home/nginx/domains/newdomain.com/public
    • Vhost log directory will be at /home/nginx/domains/newdomain.com/log
    Please post the contents of /usr/local/nginx/conf/conf.d/newdomain.com.conf and if applicable /usr/local/nginx/conf/conf.d/newdomain.com.ssl.conf wrapped in CODE tags (outlined at How to use forum BBCODE code tags)

    what is output of these commands in ssh
    Code (Text):
    curl -I https://domain.com
    

    Code (Text):
    curl -I https://www.domain.com
    

    Code (Text):
    curl -I http://domain.com
    

    Code (Text):
    curl -I http://www.domain.com
    

    wrap output in CODE tags
     
  6. fly

    fly Member

    109
    16
    18
    Jul 27, 2019
    Ratings:
    +28
    Local Time:
    11:30 PM
    I'm not certain we're on the same page here. The domain is currently secured with a valid working LE SSL certificate created originally by Cenmin 2. My attempt to reissue it (ideally to add another domain) is failing.

    Thinking that maybe I did something wrong, I created a new temp domain with Centmin 2. That also created a valid working LE cert. I then did a text comparison of the two domain.com.ssl.conf files. They were identical, with the exception of the different domain names.
     
  7. eva2000

    eva2000 Administrator Staff Member

    55,896
    12,281
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,868
    Local Time:
    1:30 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    oh i see that is because your additional domain isn't in nginx vhost server_name so it isn't resolving to the correct public web root = hence 404 not found. Make sure the additional domain names are listed in nginx vhost config domain.com.ssl.conf within server_names directive and have appropriate DNS entries and restart nginx service
     
  8. fly

    fly Member

    109
    16
    18
    Jul 27, 2019
    Ratings:
    +28
    Local Time:
    11:30 PM
    I tried adding the new domain (with the domain in server_names), it failed. So I've simplified it. I'm just trying to reissue the current cert and its failing. Once I get the current cert reissued successfully, I'll go back and add the other domain. So *all* I'm currently trying to do is reissue the exact same cert.

    For the record, this is the command I'm using (maybe I have it wrong?).
    Code:
     /root/.acme.sh/acme.sh --force --issue --days 60 -d domain.com -w /home/nginx/domains/domain.com/public -k 2048 --useragent centminmod-centos-acmesh-webroot --log /root/centminlogs/acmetool.sh-debug-log-domain.com.log --log-level 2
     
  9. eva2000

    eva2000 Administrator Staff Member

    55,896
    12,281
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,868
    Local Time:
    1:30 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  10. fly

    fly Member

    109
    16
    18
    Jul 27, 2019
    Ratings:
    +28
    Local Time:
    11:30 PM
    Okay, here ya go. In case its actually relavant, this vhost is installed as a subdomain of 'dev'.

    Code:
    #x# HTTPS-DEFAULT
     server {
       listen   80;
    #x#
       server_name dev.domain.com www.dev.domain.com;
       return 302 https://dev.domain.com$request_uri;
       root /home/nginx/domains/dev.domain.com/public;
       include /usr/local/nginx/conf/staticfiles.conf;
     }
    
    #
    
    server {
      listen 443 ssl http2 reuseport;
    
      server_name dev.domain.com www.dev.domain.com;
    
      include /usr/local/nginx/conf/ssl/dev.domain.com/dev.domain.com.crt.key.conf;
      include /usr/local/nginx/conf/ssl_include.conf;
    
      # cloudflare authenticated origin pull cert community.centminmod.com/threads/13847/
      #ssl_client_certificate /usr/local/nginx/conf/ssl/cloudflare/dev.domain.com/origin.crt;
      #ssl_verify_client on;
    
    
    
      # mozilla recommended
      ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
      ssl_prefer_server_ciphers   on;
      #add_header Alternate-Protocol  443:npn-spdy/3;
    
      # before enabling HSTS line below read centminmod.com/nginx_domain_dns_setup.html#hsts
      #add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";
      #add_header X-Frame-Options SAMEORIGIN;
      add_header X-Xss-Protection "1; mode=block" always;
      add_header X-Content-Type-Options "nosniff" always;
      #add_header Referrer-Policy "strict-origin-when-cross-origin";
      #add_header Permissions-Policy "accelerometer=(), camera=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), payment=(), usb=()";
      #spdy_headers_comp 5;
      ssl_buffer_size 1369;
      ssl_session_tickets on;
    
      # enable ocsp stapling
      resolver 8.8.8.8 8.8.4.4 1.1.1.1 1.0.0.1 valid=10m;
      resolver_timeout 10s;
      ssl_stapling on;
      ssl_stapling_verify on;
    
    # ngx_pagespeed & ngx_pagespeed handler
    #include /usr/local/nginx/conf/pagespeed.conf;
    #include /usr/local/nginx/conf/pagespeedhandler.conf;
    #include /usr/local/nginx/conf/pagespeedstatslog.conf;
    
      # limit_conn limit_per_ip 16;
      # ssi  on;
    
      access_log /home/nginx/domains/dev.domain.com/log/access.log combined buffer=256k flush=5m;
      #access_log /home/nginx/domains/dev.domain.com/log/access.json main_json buffer=256k flush=5m;
      error_log /home/nginx/domains/dev.domain.com/log/error.log;
    
      include /usr/local/nginx/conf/autoprotect/dev.domain.com/autoprotect-dev.domain.com.conf;
      root /home/nginx/domains/dev.domain.com/public/app;
      # uncomment cloudflare.conf include if using cloudflare for
      # server and/or vhost site
      #include /usr/local/nginx/conf/cloudflare.conf;
      include /usr/local/nginx/conf/503include-main.conf;
    
      location / {
    
        include /usr/local/nginx/conf/503include-only.conf;
        index index.html;
        try_files $uri $uri/ /index.html?$args;
    
    }
    
      include /usr/local/nginx/conf/php.conf;
    
      include /usr/local/nginx/conf/pre-staticfiles-local-dev.domain.com.conf;
      include /usr/local/nginx/conf/pre-staticfiles-global.conf;
      include /usr/local/nginx/conf/staticfiles.conf;
      include /usr/local/nginx/conf/drop.conf;
      #include /usr/local/nginx/conf/errorpage.conf;
      include /usr/local/nginx/conf/vts_server.conf;
    }
    
    Code:
    curl -I https://dev.domain.com
    
    HTTP/1.1 200 OK
    Date: Wed, 19 Apr 2023 20:14:50 GMT
    Content-Type: text/html; charset=utf-8
    Content-Length: 4711
    Last-Modified: Wed, 22 Mar 2023 21:02:43 GMT
    Connection: keep-alive
    Vary: Accept-Encoding
    ETag: "641b6cf3-1267"
    Server: nginx centminmod
    X-Powered-By: centminmod
    X-Xss-Protection: 1; mode=block
    X-Content-Type-Options: nosniff
    Accept-Ranges: bytes
    
    Code:
    curl -I http://dev.domain.com
    
    HTTP/1.1 302 Moved Temporarily
    Date: Wed, 19 Apr 2023 20:19:53 GMT
    Content-Type: text/html
    Content-Length: 138
    Connection: keep-alive
    Location: https://dev.domain.com/
    Server: nginx centminmod
    X-Powered-By: centminmod
    
    Code:
    curl -I https://www.dev.domain.com
    
    curl: (51) Unable to communicate securely with peer: requested domain name does not match the server's certificate.
    
    Code:
     curl -I http://www.dev.domain.com
    
    HTTP/1.1 302 Moved Temporarily
    Date: Wed, 19 Apr 2023 20:22:58 GMT
    Content-Type: text/html
    Content-Length: 138
    Connection: keep-alive
    Location: https://dev.domain.com/
    Server: nginx centminmod
    X-Powered-By: centminmod
    
     
  11. eva2000

    eva2000 Administrator Staff Member

    55,896
    12,281
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,868
    Local Time:
    1:30 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Oh you mean 2nd level subdomain sub.dev.domain.com? Did you make sure to add that domain to both server name directives for 302 redirect and https port 443 server contexts?
     
  12. fly

    fly Member

    109
    16
    18
    Jul 27, 2019
    Ratings:
    +28
    Local Time:
    11:30 PM
    No the domain is dev.domain.com and it was all handled by Centmin 2.
     
  13. eva2000

    eva2000 Administrator Staff Member

    55,896
    12,281
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,868
    Local Time:
    1:30 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Not seeing anything obvious but try commenting out /usr/local/nginx/conf/drop.conf include file and restart nginx service
     
  14. fly

    fly Member

    109
    16
    18
    Jul 27, 2019
    Ratings:
    +28
    Local Time:
    11:30 PM
    Yeah, that didn't help either.

    I tried adding a new vhost for testing purposes. I can add, reissue, etc certs for this new test vhost without issue. So there's something very wrong with the dev. vhost that we're talking about here. I may just try blowing it away and recreating it at this point.
     
  15. eva2000

    eva2000 Administrator Staff Member

    55,896
    12,281
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,868
    Local Time:
    1:30 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    If you're still trying with acme.sh force reissue direct manual command, then last piece is me looking at the full /root/centminlogs/acmetool.sh-debug-log-domain.com.log debug log
     
  16. fly

    fly Member

    109
    16
    18
    Jul 27, 2019
    Ratings:
    +28
    Local Time:
    11:30 PM
    Recreating the vhost was certainly the nuclear option, but it worked. Thanks again for your help good sir!
     
  17. eva2000

    eva2000 Administrator Staff Member

    55,896
    12,281
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,868
    Local Time:
    1:30 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Interesting so something in your previous config held up the issuance!
     
  18. fly

    fly Member

    109
    16
    18
    Jul 27, 2019
    Ratings:
    +28
    Local Time:
    11:30 PM
    Yeah, and you saw the config above. It makes zero sense.
     
  19. eva2000

    eva2000 Administrator Staff Member

    55,896
    12,281
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,868
    Local Time:
    1:30 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    I find the full /root/centminlogs/acmetool.sh-debug-log-domain.com.log debug log as the best source to use for troubleshooting these issues usually :)
     
  20. fly

    fly Member

    109
    16
    18
    Jul 27, 2019
    Ratings:
    +28
    Local Time:
    11:30 PM
    If you actually still want them, I'd be happy to share. Otherwise, we can consider this resolved. ;)