SSL Letsencrypt My site went down, SSL expired

sepulchre · Jun 21, 2019

Hey,

my site went down right now and I don't know what to do. I thought the SSL will renew itself automatically? What happened here?

Code:

SEC_ERROR_EXPIRED_CERTIFICATE

HTTP Strict Transport Security: true
HTTP Public Key Pinning: false

Using letsdebug.net

Code:

Test result for xxx.com using http-01
AAAANotWorking
Error
xxx.com has an AAAA (IPv6) record (2a01:4f8:1c0c:700f::1) but a test request to this address over port 80 did not succeed.
Your web server must have at least one working IPv4 or IPv6 address. You should either ensure that validation requests to this domain succeed over IPv6, or remove its AAAA record.
Get http://xxx.com/.well-known/acme-challenge/letsdebug-test: dial tcp [2a01:4f8:1c0c:700f::1]:80: connect: connection refused

Trace:
@0ms: Making a request to http://xxx.com/.well-known/acme-challenge/letsdebug-test (using initial IP 2a01:4f8:1c0c:700f::1)
@0ms: Dialing 2a01:4f8:1c0c:700f::1
@80ms: Experienced error: dial tcp [2a01:4f8:1c0c:700f::1]:80: connect: connection refused
IssueFromLetsEncrypt
Error
A test authorization for xxx.com to the Let's Encrypt staging service has revealed issues that may prevent any certificate for this domain being issued.
Fetching https://xxx.com/.well-known/acme-challenge/Bokv_43OtlHNZ2YRrQLxZNLc1kyJ-ypLLePU9mOAYao: Connection refused

What to do?

I want the site get online again, don't care what happened.

sepulchre · Jun 21, 2019

From this tutorial:
SSL - How switch to SSL on centminmod/nginx

I am trying the 2nd method, as the other methods are not for my case, I cannot get a new SSL to work.

Doing
Code:
./acmetool.sh acme-menu
And doing the issue, reissue, renew processes, it says all the time
Code:
[Fri Jun 21 12:50:50 UTC 2019] MYDOMAIN.com:Verify error:Fetching https://MYDOMAIN.com/.well-known/acme-challenge/Gl0c5-O5CRrx4Ik2hWL1SBgbxK-rtHE7kFUUE0JwZcs: Connection refused
How can I solve this problem? I have seen here on forum people having the same problem but none explained how to solve this.

pamamolf · Jun 21, 2019

For a reason it seems that your server is refusing the connection from Let's Encrypt...

Did you check your firewall? Maybe clean the ban ip's list or disable it for a while?

sepulchre · Jun 21, 2019

pamamolf said: ↑

Did you check your firewall? Maybe clean the ban ip's list or disable it for a while?
Click to expand...

I have the standard centminmod server, I have all the preconfigured stuff, didn't touch anything.

So, what to do? How can I disable it for a while?

pamamolf · Jun 21, 2019

Keep in mind that i am not an expert and i am not sure that my recommendation will solve your issue but i think you can try....

Backup your file first:

/etc/csf/csf.deny

Then edit it and remove the banned ip's and restart csf firewall using from ssh: csf -r

If it will not work you can try to disable it using: csf -x

To re enable it run: csf -e

sepulchre · Jun 21, 2019

Thanks, I tried and disabled it, still same problem.

Code:

-----------------------------------------------------------
renew & install letsencrypt ssl certificate for MYDOMAIN.com
-----------------------------------------------------------
testcert value = live
/root/.acme.sh/acme.sh --issue -d MYDOMAIN.com -d www.MYDOMAIN.com --days 60 -w /home/nginx/domains/MYDOMAIN.com/public -k 2048 --useragent centminmod-centos7-acmesh-webroot --log /root/centminlogs/acmetool.sh-debug-log-210619-132733.log --log-level 2
[Fri Jun 21 13:27:40 UTC 2019] Multi domain='DNS:MYDOMAIN.com,DNS:www.MYDOMAIN.com'
[Fri Jun 21 13:27:40 UTC 2019] Getting domain auth token for each domain
[Fri Jun 21 13:27:42 UTC 2019] Getting webroot for domain='MYDOMAIN.com'
[Fri Jun 21 13:27:42 UTC 2019] Getting webroot for domain='www.MYDOMAIN.com'
[Fri Jun 21 13:27:42 UTC 2019] Verifying: MYDOMAIN.com
[Fri Jun 21 13:27:45 UTC 2019] MYDOMAIN.com:Verify error:Fetching https://MYDOMAIN.com/.well-known/acme-challenge/kG_L2OMtYCxO5zR9d6jAAwCsMhCtMg0U3QJV-tPRuXU: Connection refused
[Fri Jun 21 13:27:45 UTC 2019] Please check log file for more details: /root/centminlogs/acmetool.sh-debug-log-210619-132733.log
LECHECK = 1

log files saved at /root/centminlogs
-rw-r--r-- 1 root root  46K Jun 21 13:27 acmetool.sh-debug-log-210619-132733.log
-rw-r--r-- 1 root root 2.6K Jun 21 13:27 acmesh-renew_210619-132733.log

pamamolf · Jun 21, 2019

Did you check the log files for more details?

eva2000 · Jun 21, 2019

sepulchre said: ↑

Test result for xxx.com using http-01 AAAANotWorking Error xxx.com has an AAAA (IPv6) record (2a01:4f8:1c0c:700f::1) but a test request to this address over port 80 did not succeed. Your web server must have at least one working IPv4 or IPv6 address. You should either ensure that validation requests to this domain succeed over IPv6, or remove its AAAA record. Get xxx.com: dial tcp [2a01:4f8:1c0c:700f::1]:80: connect: connection refused
Click to expand...

IPv6 not working on your server either at network or nginx level

Solution outlined there too easiest just remove dns AAAA record for domain

sepulchre · Jun 21, 2019

pamamolf said: ↑

Did you check the log files for more details?
Click to expand...

I did but I couldn't understand what's wrong.

eva2000 said: ↑

IPv6 not working on your server either at network or nginx level
Click to expand...

IPv6 is active from network's side. And I did everything to get ipv6 working from nginx side.

I think I have the same issue, what all other people reported before me. When I setup my SSL, I didn't have a AAAA record, and added it after. Now I don't understand how I can renew the SSL certificate.

eva2000 said: ↑

Solution outlined there too easiest just remove dns AAAA record for domain
Click to expand...

So just remove from DNS (Cloudflare in my case)? And then do what? Wait?

eva2000 · Jun 21, 2019

sepulchre said: ↑

So just remove from DNS (Cloudflare in my case
Click to expand...

Yes remove from cloudflare and then run letsencrypt renewal cronjob manually

sepulchre · Jun 22, 2019

eva2000 said: ↑

Yes remove from cloudflare and then run letsencrypt renewal cronjob manually
Click to expand...

I did the renewal via acmetool.

Does it mean I can put the DNS AAAA records back again and how to configure the SSL now properly to work with ipv6?

Site is working again, thank you.

eva2000 · Jun 22, 2019

sepulchre said: ↑

I did the renewal via acmetool.
Click to expand...

what was exact command used as some will rewrite your nginx vhost which depending which acmetool.sh option used on your vhost could break nginx vhost for future letsencrypt auto renewals

only acmetool.sh option that doesn't touch nginx vhost is acmetool.sh reissue-only option

cronjob for acme.sh client listing is
Code (Text):
crontab -l | grep acme
8 0 * * * "/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh" > /dev/null
so re-running manually should renewal an already expired ssl cert from letsencrypt for all domains with letsencrypt issued from the server via centmin.sh menu option 2, 22 or nv command which uses addons/acmetool.sh which relies on acme.sh client
Code (Text):
"/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh"
example for domain.com run where it hasn't expired so skips auto renew
Code (Text):
"/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh"
[Fri Jun 21 16:06:06 UTC 2019] ===Starting cron===
[Fri Jun 21 16:06:06 UTC 2019] Renew: 'domain.com'
[Fri Jun 21 16:06:06 UTC 2019] Skip, Next renewal time is: Tue Jul 30 06:33:30 UTC 2019
[Fri Jun 21 16:06:06 UTC 2019] Add '--force' to force to renew.
[Fri Jun 21 16:06:06 UTC 2019] Skipped domain.com
[Fri Jun 21 16:06:06 UTC 2019] ===End cron===
sepulchre said: ↑

Does it mean I can put the DNS AAAA records back again and how to configure the SSL now properly to work with ipv6?
Click to expand...

if IPv6 network connectivity is still broken and/or nginx IPv6 isn't configured in nginx vhost for domain, then putting DNS AAAA record back will cause next letsencrypt auto renewal to fail again

easiest thing to do is just remove DNS AAAA record so letsencrypt only validates auto renewals via DNS A IPv4 record lookup if your IPv6 network connectivity is flaky or nginx IPv6 configuration isn't working.

For nginx IPv6 see official FAQ item 34 for How To Enable Nginx IPv6 Support ?

sepulchre · Jun 22, 2019

eva2000 said: ↑

what was exact command used as some will rewrite your nginx vhost which depending which acmetool.sh option used on your vhost could break nginx vhost for future letsencrypt auto renewals
Click to expand...

I did the renew one in the menu option and said https is the default one I want.

What disadvantage do I have now?

eva2000 said: ↑

easiest thing to do is just remove DNS AAAA record so letsencrypt only validates auto renewals via DNS A IPv4 record lookup if your IPv6 network connectivity is flaky or nginx IPv6 configuration isn't working.
Click to expand...

I removed it, it will stay like this. I just did it because of mail configuration stuff, I hope nothing breaks now. I will now regularly keep an eye on this. The certificate will expire in september now. When is usually the auto-renewal, so I can check, if the cron is working or not?

eva2000 · Jun 22, 2019

sepulchre said: ↑

When is usually the auto-renewal, so I can check, if the cron is working or not?
Click to expand...

auto renew is 30 days before expiry

sepulchre said: ↑

What disadvantage do I have now?
Click to expand...

hard to know - you'd have to inspect your nginx vhost

When you create a new nginx vhost domain via centmin.sh menu option 2 or menu option 22 or via /usr/bin/nv cli command line, you will create the Nginx vhost files and directories. You will get an outputted the path location where it will create the domain name's vhost conf file named newdomain.com.conf (and newdomain.com.ssl.conf if you selected yes to self signed SSL)

Nginx vhost conf path will be at /usr/local/nginx/conf/conf.d/newdomain.com.conf

Nginx HTTP/2 SSL vhost conf path will be at /usr/local/nginx/conf/conf.d/newdomain.com.ssl.conf

Nginx Self-Signed SSL Certificate Directory at /usr/local/nginx/conf/ssl/newdomain.com

Vhost public web root will be at /home/nginx/domains/newdomain.com/public

Vhost log directory will be at /home/nginx/domains/newdomain.com/log

Please post the contents of /usr/local/nginx/conf/conf.d/newdomain.com.conf and if applicable /usr/local/nginx/conf/conf.d/newdomain.com.ssl.conf wrapped in CODE tags (outlined at How to use forum BBCODE code tags)

sepulchre · Jun 22, 2019

But I neither used menu option 2 or 22 nor used a new domain name.

I used the acmetool and entered my current_domain.com. That's it.

It is possible though I accidentally created a acme.domain.com while I was trying to figure out the problem.

See:

Here the content of mydomain.com.ssl.conf

Code:

#x# HTTPS-DEFAULT
 server {
 
   server_name mydomain.com www.mydomain.com;
   return 302 https://mydomain.com$request_uri;
   include /usr/local/nginx/conf/staticfiles.conf;
 }

server {
    server_name mydomain.com www.mydomain.com;
    return 301 https://mydomain.com$request_uri;
 }

server {
  listen 443 ssl http2;
  server_name mydomain.com www.mydomain.com;
 
    ##  redirect https www to https non-www
      if ($host = 'www.mydomain.com' ) {
         return 301 https://mydomain.com$request_uri;
      }
     

  include /usr/local/nginx/conf/ssl/mydomain.com/mydomain.com.crt.key.conf;
  include /usr/local/nginx/conf/ssl_include.conf;

  http2_max_field_size 16k;
  http2_max_header_size 32k;
  # dual cert supported ssl ciphers
  ssl_ciphers     EECDH+CHACHA20-draft:EECDH+CHACHA20:EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA256:EECDH+ECDSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+SHA384:EECDH+AES128:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!CAMELLIA;
  ssl_prefer_server_ciphers   on;
  #add_header Alternate-Protocol  443:npn-spdy/3;
  add_header Strict-Transport-Security "max-age=31536000;";
  #add_header X-Frame-Options SAMEORIGIN;
  #add_header X-Xss-Protection "1; mode=block" always;
  #add_header X-Content-Type-Options "nosniff" always;
  #spdy_headers_comp 5;
  ssl_buffer_size 1369;
  ssl_session_tickets on;
 
  # enable ocsp stapling
  resolver 8.8.8.8 8.8.4.4 valid=10m;
  resolver_timeout 10s;
  ssl_stapling on;
  ssl_stapling_verify on;

# ngx_pagespeed & ngx_pagespeed handler
#include /usr/local/nginx/conf/pagespeed.conf;
#include /usr/local/nginx/conf/pagespeedhandler.conf;
#include /usr/local/nginx/conf/pagespeedstatslog.conf;

  # limit_conn limit_per_ip 16;
  # ssi  on;

  access_log /home/nginx/domains/mydomain.com/log/access.log combined buffer=256k flush=5m;
  error_log /home/nginx/domains/mydomain.com/log/error.log;

  root /home/nginx/domains/mydomain.com/public;

location / {
     index index.php index.html index.htm;
     try_files $uri $uri/ /index.php?$uri&$args;
}



location /install/data/ {
     internal;
}

location /install/templates/ {
     internal;
}

location /internal_data/ {
     internal;
}

location /library/ {
     internal;
}

# xenforo 2 uncomment / remove hash from next 3 lines
location /src/ {
     internal;
}

  include /usr/local/nginx/conf/staticfiles.conf;
  include /usr/local/nginx/conf/php.conf;
  #include /usr/local/nginx/conf/drop.conf;
  #include /usr/local/nginx/conf/errorpage.conf;
  include /usr/local/nginx/conf/vts_server.conf;
}

eva2000 · Jun 22, 2019

sepulchre said: ↑

I used the acmetool and entered my current_domain.com. That's it.
Click to expand...

exact command you used ? you can type = history to look at history of all your commands you ran in SSH and see the one you used

sepulchre · Jun 22, 2019

The problem is the history doesn't save the stuff I typed when I run centmin.sh or acmetool.sh

Code:

604  [21.06.19] 10:36:52   yum clean all
  605  [21.06.19] 10:37:02   yum list updates
  606  [21.06.19] 10:38:25   yum -y update
  607  [21.06.19] 10:43:14   mysqladmin flush-tables && sleep 60 && reboot
  608  [21.06.19] 10:44:54   cd /usr/local/src/centminmod
  609  [21.06.19] 10:44:58   ./centmin.sh
  610  [21.06.19] 10:46:13   cd /usr/local/src/centminmod
  611  [21.06.19] 10:46:22   ./centmin.sh
  612  [21.06.19] 11:11:44   cd /usr/local/src/centminmod
  613  [21.06.19] 11:11:45   ./centmin.sh
  614  [21.06.19] 11:15:18   cd /usr/local/src/centminmod/addons
  615  [21.06.19] 11:16:30   ./acmetool.sh reissue server.mydomain.com lived
  616  [21.06.19] 11:20:51   cd /usr/local/src/centminmod
  617  [21.06.19] 11:20:53   ./centmin.sh
  618  [21.06.19] 11:29:03   dig http2.centminmod.com +shortdig http2.centminmod.com +shortdig http2.centminmod.com +shortdig http2.centminmod.com +short
  619  [21.06.19] 11:29:22   dig http2.centminmod.com +short   
  620  [21.06.19] 11:29:48   dig http2.centminmod.com +short
  621  [21.06.19] 11:38:11   cd /usr/local/src/centminmod/addons
  622  [21.06.19] 11:38:15   ./acmetool.sh reissue acme.domain.com live
  623  [21.06.19] 11:41:00   nprestart
  624  [21.06.19] 11:41:57   ./acmetool.sh acme-menu
  625  [21.06.19] 12:33:08   cd /usr/local/src/centminmod
  626  [21.06.19] 12:37:09   ./centmin.sh
  627  [21.06.19] 12:50:09   cd /usr/local/src/centminmod/addons
  628  [21.06.19] 12:50:14   ./acmetool.sh acme-menu
  629  [21.06.19] 13:24:31   csf -r
  630  [21.06.19] 13:24:58   cd /usr/local/src/centminmod/addons
  631  [21.06.19] 13:25:00   ./acmetool.sh acme-menu
  632  [21.06.19] 13:26:09   csf -x
  633  [21.06.19] 13:26:17   csf -r
  634  [21.06.19] 13:26:22   cd /usr/local/src/centminmod/addons
  635  [21.06.19] 13:26:25   ./acmetool.sh acme-menu
  636  [21.06.19] 13:28:14   csf -e
  637  [21.06.19] 13:28:19   csf -r
  638  [21.06.19] 13:52:54   cd /usr/local/src/centminmod/addons
  639  [21.06.19] 13:52:56   ./acmetool.sh acme-menu
  640  [21.06.19] 20:57:02   cd /usr/local/src/centminmod/addons
  641  [21.06.19] 20:57:09   ./acmetool.sh acme-menu
  642  [21.06.19] 20:59:22   crontab -l | grep acme
  643  [21.06.19] 21:00:30   "/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh"
  644  [21.06.19] 21:01:36   ./acmetool.sh acme-menu
  645  [21.06.19] 21:02:18   ./acmetool.sh reissue-only domain.com live
  646  [21.06.19] 21:03:06   ./acmetool.sh acme-menu
  647  [21.06.19] 21:04:01   nprestart
  648  [21.06.19] 21:04:38   ./acmetool.sh acme-menu
  649  [22.06.19] 00:21:14   history

Log in / Register

Forums

Learn about Centmin Mod LEMP Stack today

SSL Letsencrypt My site went down, SSL expired

sepulchre Member

sepulchre Member

pamamolf Premium Member Premium Member

sepulchre Member

pamamolf Premium Member Premium Member

sepulchre Member

pamamolf Premium Member Premium Member

eva2000 Administrator Staff Member

sepulchre Member

eva2000 Administrator Staff Member

sepulchre Member

eva2000 Administrator Staff Member

sepulchre Member

eva2000 Administrator Staff Member

sepulchre Member

eva2000 Administrator Staff Member

sepulchre Member

.

Log in / Register

Useful Searches

Learn about Centmin Mod LEMP Stack today

SSL Letsencrypt My site went down, SSL expired

sepulchre Member

sepulchre Member

pamamolf Premium Member Premium Member

sepulchre Member

pamamolf Premium Member Premium Member

sepulchre Member

pamamolf Premium Member Premium Member

eva2000 Administrator Staff Member

sepulchre Member

eva2000 Administrator Staff Member

sepulchre Member

eva2000 Administrator Staff Member

sepulchre Member

eva2000 Administrator Staff Member

sepulchre Member

eva2000 Administrator Staff Member

sepulchre Member

.