Learn about Centmin Mod LEMP Stack today
Become a Member

SSL Letsencrypt My site went down, SSL expired

Discussion in 'Domains, DNS, Email & SSL Certificates' started by sepulchre, Jun 21, 2019.

  1. sepulchre

    sepulchre Member

    168
    22
    18
    Dec 22, 2014
    Ratings:
    +28
    Local Time:
    9:26 AM
    Hey,

    my site went down right now and I don't know what to do. I thought the SSL will renew itself automatically? What happened here?

    Code:
    SEC_ERROR_EXPIRED_CERTIFICATE
    
    HTTP Strict Transport Security: true
    HTTP Public Key Pinning: false
    
    Using letsdebug.net
    Code:
    Test result for xxx.com using http-01
    AAAANotWorking
    Error
    xxx.com has an AAAA (IPv6) record (2a01:4f8:1c0c:700f::1) but a test request to this address over port 80 did not succeed.
    Your web server must have at least one working IPv4 or IPv6 address. You should either ensure that validation requests to this domain succeed over IPv6, or remove its AAAA record.
    Get http://xxx.com/.well-known/acme-challenge/letsdebug-test: dial tcp [2a01:4f8:1c0c:700f::1]:80: connect: connection refused
    
    Trace:
    @0ms: Making a request to http://xxx.com/.well-known/acme-challenge/letsdebug-test (using initial IP 2a01:4f8:1c0c:700f::1)
    @0ms: Dialing 2a01:4f8:1c0c:700f::1
    @80ms: Experienced error: dial tcp [2a01:4f8:1c0c:700f::1]:80: connect: connection refused
    IssueFromLetsEncrypt
    Error
    A test authorization for xxx.com to the Let's Encrypt staging service has revealed issues that may prevent any certificate for this domain being issued.
    Fetching https://xxx.com/.well-known/acme-challenge/Bokv_43OtlHNZ2YRrQLxZNLc1kyJ-ypLLePU9mOAYao: Connection refused 
    What to do?

    I want the site get online again, don't care what happened.

     
    Last edited: Jun 21, 2019
  2. sepulchre

    sepulchre Member

    168
    22
    18
    Dec 22, 2014
    Ratings:
    +28
    Local Time:
    9:26 AM
    From this tutorial:
    SSL - How switch to SSL on centminmod/nginx

    I am trying the 2nd method, as the other methods are not for my case, I cannot get a new SSL to work.

    Doing

    Code:
    ./acmetool.sh acme-menu
    
    And doing the issue, reissue, renew processes, it says all the time

    Code:
    [Fri Jun 21 12:50:50 UTC 2019] MYDOMAIN.com:Verify error:Fetching https://MYDOMAIN.com/.well-known/acme-challenge/Gl0c5-O5CRrx4Ik2hWL1SBgbxK-rtHE7kFUUE0JwZcs: Connection refused
    How can I solve this problem? I have seen here on forum people having the same problem but none explained how to solve this.
     
  3. pamamolf

    pamamolf Well-Known Member

    4,101
    428
    83
    May 31, 2014
    Ratings:
    +838
    Local Time:
    10:26 AM
    Nginx-1.26.x
    MariaDB 10.6.x
    For a reason it seems that your server is refusing the connection from Let's Encrypt...

    Did you check your firewall? Maybe clean the ban ip's list or disable it for a while?
     
  4. sepulchre

    sepulchre Member

    168
    22
    18
    Dec 22, 2014
    Ratings:
    +28
    Local Time:
    9:26 AM
    I have the standard centminmod server, I have all the preconfigured stuff, didn't touch anything.

    So, what to do? How can I disable it for a while?
     
  5. pamamolf

    pamamolf Well-Known Member

    4,101
    428
    83
    May 31, 2014
    Ratings:
    +838
    Local Time:
    10:26 AM
    Nginx-1.26.x
    MariaDB 10.6.x
    Keep in mind that i am not an expert and i am not sure that my recommendation will solve your issue but i think you can try....

    Backup your file first:

    /etc/csf/csf.deny

    Then edit it and remove the banned ip's and restart csf firewall using from ssh: csf -r

    If it will not work you can try to disable it using: csf -x

    To re enable it run: csf -e
     
  6. sepulchre

    sepulchre Member

    168
    22
    18
    Dec 22, 2014
    Ratings:
    +28
    Local Time:
    9:26 AM
    Thanks, I tried and disabled it, still same problem.

    Code:
    -----------------------------------------------------------
    renew & install letsencrypt ssl certificate for MYDOMAIN.com
    -----------------------------------------------------------
    testcert value = live
    /root/.acme.sh/acme.sh --issue -d MYDOMAIN.com -d www.MYDOMAIN.com --days 60 -w /home/nginx/domains/MYDOMAIN.com/public -k 2048 --useragent centminmod-centos7-acmesh-webroot --log /root/centminlogs/acmetool.sh-debug-log-210619-132733.log --log-level 2
    [Fri Jun 21 13:27:40 UTC 2019] Multi domain='DNS:MYDOMAIN.com,DNS:www.MYDOMAIN.com'
    [Fri Jun 21 13:27:40 UTC 2019] Getting domain auth token for each domain
    [Fri Jun 21 13:27:42 UTC 2019] Getting webroot for domain='MYDOMAIN.com'
    [Fri Jun 21 13:27:42 UTC 2019] Getting webroot for domain='www.MYDOMAIN.com'
    [Fri Jun 21 13:27:42 UTC 2019] Verifying: MYDOMAIN.com
    [Fri Jun 21 13:27:45 UTC 2019] MYDOMAIN.com:Verify error:Fetching https://MYDOMAIN.com/.well-known/acme-challenge/kG_L2OMtYCxO5zR9d6jAAwCsMhCtMg0U3QJV-tPRuXU: Connection refused
    [Fri Jun 21 13:27:45 UTC 2019] Please check log file for more details: /root/centminlogs/acmetool.sh-debug-log-210619-132733.log
    LECHECK = 1
    
    log files saved at /root/centminlogs
    -rw-r--r-- 1 root root  46K Jun 21 13:27 acmetool.sh-debug-log-210619-132733.log
    -rw-r--r-- 1 root root 2.6K Jun 21 13:27 acmesh-renew_210619-132733.log
     
  7. pamamolf

    pamamolf Well-Known Member

    4,101
    428
    83
    May 31, 2014
    Ratings:
    +838
    Local Time:
    10:26 AM
    Nginx-1.26.x
    MariaDB 10.6.x
    Did you check the log files for more details?
     
  8. eva2000

    eva2000 Administrator Staff Member

    55,801
    12,271
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,857
    Local Time:
    5:26 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    IPv6 not working on your server either at network or nginx level

    Solution outlined there too easiest just remove dns AAAA record for domain
     
  9. sepulchre

    sepulchre Member

    168
    22
    18
    Dec 22, 2014
    Ratings:
    +28
    Local Time:
    9:26 AM
    I did but I couldn't understand what's wrong.

    IPv6 is active from network's side. And I did everything to get ipv6 working from nginx side.

    I think I have the same issue, what all other people reported before me. When I setup my SSL, I didn't have a AAAA record, and added it after. Now I don't understand how I can renew the SSL certificate.

    So just remove from DNS (Cloudflare in my case)? And then do what? Wait?
     
  10. eva2000

    eva2000 Administrator Staff Member

    55,801
    12,271
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,857
    Local Time:
    5:26 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Yes remove from cloudflare and then run letsencrypt renewal cronjob manually
     
  11. sepulchre

    sepulchre Member

    168
    22
    18
    Dec 22, 2014
    Ratings:
    +28
    Local Time:
    9:26 AM
    I did the renewal via acmetool.

    Does it mean I can put the DNS AAAA records back again and how to configure the SSL now properly to work with ipv6?

    Site is working again, thank you.
     
    Last edited: Jun 22, 2019
  12. eva2000

    eva2000 Administrator Staff Member

    55,801
    12,271
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,857
    Local Time:
    5:26 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    what was exact command used as some will rewrite your nginx vhost which depending which acmetool.sh option used on your vhost could break nginx vhost for future letsencrypt auto renewals

    only acmetool.sh option that doesn't touch nginx vhost is acmetool.sh reissue-only option

    cronjob for acme.sh client listing is
    Code (Text):
    crontab -l | grep acme
    8 0 * * * "/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh" > /dev/null
    

    so re-running manually should renewal an already expired ssl cert from letsencrypt for all domains with letsencrypt issued from the server via centmin.sh menu option 2, 22 or nv command which uses addons/acmetool.sh which relies on acme.sh client
    Code (Text):
    "/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh"

    example for domain.com run where it hasn't expired so skips auto renew
    Code (Text):
    "/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh"
    [Fri Jun 21 16:06:06 UTC 2019] ===Starting cron===
    [Fri Jun 21 16:06:06 UTC 2019] Renew: 'domain.com'
    [Fri Jun 21 16:06:06 UTC 2019] Skip, Next renewal time is: Tue Jul 30 06:33:30 UTC 2019
    [Fri Jun 21 16:06:06 UTC 2019] Add '--force' to force to renew.
    [Fri Jun 21 16:06:06 UTC 2019] Skipped domain.com
    [Fri Jun 21 16:06:06 UTC 2019] ===End cron===
    

    if IPv6 network connectivity is still broken and/or nginx IPv6 isn't configured in nginx vhost for domain, then putting DNS AAAA record back will cause next letsencrypt auto renewal to fail again

    easiest thing to do is just remove DNS AAAA record so letsencrypt only validates auto renewals via DNS A IPv4 record lookup if your IPv6 network connectivity is flaky or nginx IPv6 configuration isn't working.

    For nginx IPv6 see official FAQ item 34 for How To Enable Nginx IPv6 Support ?
     
  13. sepulchre

    sepulchre Member

    168
    22
    18
    Dec 22, 2014
    Ratings:
    +28
    Local Time:
    9:26 AM
    I did the renew one in the menu option and said https is the default one I want.

    What disadvantage do I have now?

    I removed it, it will stay like this. I just did it because of mail configuration stuff, I hope nothing breaks now. I will now regularly keep an eye on this. The certificate will expire in september now. When is usually the auto-renewal, so I can check, if the cron is working or not?
     
  14. eva2000

    eva2000 Administrator Staff Member

    55,801
    12,271
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,857
    Local Time:
    5:26 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    auto renew is 30 days before expiry

    hard to know - you'd have to inspect your nginx vhost

    When you create a new nginx vhost domain via centmin.sh menu option 2 or menu option 22 or via /usr/bin/nv cli command line, you will create the Nginx vhost files and directories. You will get an outputted the path location where it will create the domain name's vhost conf file named newdomain.com.conf (and newdomain.com.ssl.conf if you selected yes to self signed SSL)
    • Nginx vhost conf path will be at /usr/local/nginx/conf/conf.d/newdomain.com.conf
    • Nginx HTTP/2 SSL vhost conf path will be at /usr/local/nginx/conf/conf.d/newdomain.com.ssl.conf
    • Nginx Self-Signed SSL Certificate Directory at /usr/local/nginx/conf/ssl/newdomain.com
    • Vhost public web root will be at /home/nginx/domains/newdomain.com/public
    • Vhost log directory will be at /home/nginx/domains/newdomain.com/log
    Please post the contents of /usr/local/nginx/conf/conf.d/newdomain.com.conf and if applicable /usr/local/nginx/conf/conf.d/newdomain.com.ssl.conf wrapped in CODE tags (outlined at How to use forum BBCODE code tags)
     
  15. sepulchre

    sepulchre Member

    168
    22
    18
    Dec 22, 2014
    Ratings:
    +28
    Local Time:
    9:26 AM
    But I neither used menu option 2 or 22 nor used a new domain name.

    I used the acmetool and entered my current_domain.com. That's it.

    It is possible though I accidentally created a acme.domain.com while I was trying to figure out the problem.

    See:

    upload_2019-6-22_0-33-36.png

    Here the content of mydomain.com.ssl.conf

    Code:
    #x# HTTPS-DEFAULT
     server {
     
       server_name mydomain.com www.mydomain.com;
       return 302 https://mydomain.com$request_uri;
       include /usr/local/nginx/conf/staticfiles.conf;
     }
    
    server {
        server_name mydomain.com www.mydomain.com;
        return 301 https://mydomain.com$request_uri;
     }
    
    server {
      listen 443 ssl http2;
      server_name mydomain.com www.mydomain.com;
     
        ##  redirect https www to https non-www
          if ($host = 'www.mydomain.com' ) {
             return 301 https://mydomain.com$request_uri;
          }
         
    
      include /usr/local/nginx/conf/ssl/mydomain.com/mydomain.com.crt.key.conf;
      include /usr/local/nginx/conf/ssl_include.conf;
    
      http2_max_field_size 16k;
      http2_max_header_size 32k;
      # dual cert supported ssl ciphers
      ssl_ciphers     EECDH+CHACHA20-draft:EECDH+CHACHA20:EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA256:EECDH+ECDSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+SHA384:EECDH+AES128:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!CAMELLIA;
      ssl_prefer_server_ciphers   on;
      #add_header Alternate-Protocol  443:npn-spdy/3;
      add_header Strict-Transport-Security "max-age=31536000;";
      #add_header X-Frame-Options SAMEORIGIN;
      #add_header X-Xss-Protection "1; mode=block" always;
      #add_header X-Content-Type-Options "nosniff" always;
      #spdy_headers_comp 5;
      ssl_buffer_size 1369;
      ssl_session_tickets on;
     
      # enable ocsp stapling
      resolver 8.8.8.8 8.8.4.4 valid=10m;
      resolver_timeout 10s;
      ssl_stapling on;
      ssl_stapling_verify on;
    
    # ngx_pagespeed & ngx_pagespeed handler
    #include /usr/local/nginx/conf/pagespeed.conf;
    #include /usr/local/nginx/conf/pagespeedhandler.conf;
    #include /usr/local/nginx/conf/pagespeedstatslog.conf;
    
      # limit_conn limit_per_ip 16;
      # ssi  on;
    
      access_log /home/nginx/domains/mydomain.com/log/access.log combined buffer=256k flush=5m;
      error_log /home/nginx/domains/mydomain.com/log/error.log;
    
      root /home/nginx/domains/mydomain.com/public;
    
    location / {
         index index.php index.html index.htm;
         try_files $uri $uri/ /index.php?$uri&$args;
    }
    
    
    
    location /install/data/ {
         internal;
    }
    
    location /install/templates/ {
         internal;
    }
    
    location /internal_data/ {
         internal;
    }
    
    location /library/ {
         internal;
    }
    
    # xenforo 2 uncomment / remove hash from next 3 lines
    location /src/ {
         internal;
    }
    
      include /usr/local/nginx/conf/staticfiles.conf;
      include /usr/local/nginx/conf/php.conf;
      #include /usr/local/nginx/conf/drop.conf;
      #include /usr/local/nginx/conf/errorpage.conf;
      include /usr/local/nginx/conf/vts_server.conf;
    }
     
  16. eva2000

    eva2000 Administrator Staff Member

    55,801
    12,271
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,857
    Local Time:
    5:26 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    exact command you used ? you can type = history to look at history of all your commands you ran in SSH and see the one you used
     
  17. sepulchre

    sepulchre Member

    168
    22
    18
    Dec 22, 2014
    Ratings:
    +28
    Local Time:
    9:26 AM
    The problem is the history doesn't save the stuff I typed when I run centmin.sh or acmetool.sh

    Code:
    604  [21.06.19] 10:36:52   yum clean all
      605  [21.06.19] 10:37:02   yum list updates
      606  [21.06.19] 10:38:25   yum -y update
      607  [21.06.19] 10:43:14   mysqladmin flush-tables && sleep 60 && reboot
      608  [21.06.19] 10:44:54   cd /usr/local/src/centminmod
      609  [21.06.19] 10:44:58   ./centmin.sh
      610  [21.06.19] 10:46:13   cd /usr/local/src/centminmod
      611  [21.06.19] 10:46:22   ./centmin.sh
      612  [21.06.19] 11:11:44   cd /usr/local/src/centminmod
      613  [21.06.19] 11:11:45   ./centmin.sh
      614  [21.06.19] 11:15:18   cd /usr/local/src/centminmod/addons
      615  [21.06.19] 11:16:30   ./acmetool.sh reissue server.mydomain.com lived
      616  [21.06.19] 11:20:51   cd /usr/local/src/centminmod
      617  [21.06.19] 11:20:53   ./centmin.sh
      618  [21.06.19] 11:29:03   dig http2.centminmod.com +shortdig http2.centminmod.com +shortdig http2.centminmod.com +shortdig http2.centminmod.com +short
      619  [21.06.19] 11:29:22   dig http2.centminmod.com +short   
      620  [21.06.19] 11:29:48   dig http2.centminmod.com +short
      621  [21.06.19] 11:38:11   cd /usr/local/src/centminmod/addons
      622  [21.06.19] 11:38:15   ./acmetool.sh reissue acme.domain.com live
      623  [21.06.19] 11:41:00   nprestart
      624  [21.06.19] 11:41:57   ./acmetool.sh acme-menu
      625  [21.06.19] 12:33:08   cd /usr/local/src/centminmod
      626  [21.06.19] 12:37:09   ./centmin.sh
      627  [21.06.19] 12:50:09   cd /usr/local/src/centminmod/addons
      628  [21.06.19] 12:50:14   ./acmetool.sh acme-menu
      629  [21.06.19] 13:24:31   csf -r
      630  [21.06.19] 13:24:58   cd /usr/local/src/centminmod/addons
      631  [21.06.19] 13:25:00   ./acmetool.sh acme-menu
      632  [21.06.19] 13:26:09   csf -x
      633  [21.06.19] 13:26:17   csf -r
      634  [21.06.19] 13:26:22   cd /usr/local/src/centminmod/addons
      635  [21.06.19] 13:26:25   ./acmetool.sh acme-menu
      636  [21.06.19] 13:28:14   csf -e
      637  [21.06.19] 13:28:19   csf -r
      638  [21.06.19] 13:52:54   cd /usr/local/src/centminmod/addons
      639  [21.06.19] 13:52:56   ./acmetool.sh acme-menu
      640  [21.06.19] 20:57:02   cd /usr/local/src/centminmod/addons
      641  [21.06.19] 20:57:09   ./acmetool.sh acme-menu
      642  [21.06.19] 20:59:22   crontab -l | grep acme
      643  [21.06.19] 21:00:30   "/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh"
      644  [21.06.19] 21:01:36   ./acmetool.sh acme-menu
      645  [21.06.19] 21:02:18   ./acmetool.sh reissue-only domain.com live
      646  [21.06.19] 21:03:06   ./acmetool.sh acme-menu
      647  [21.06.19] 21:04:01   nprestart
      648  [21.06.19] 21:04:38   ./acmetool.sh acme-menu
      649  [22.06.19] 00:21:14   history