Join the community today
Register Now

Cloudflare 'Astonishing' problem with letsencrypt/cloudflare

Discussion in 'Domains, DNS, Email & SSL Certificates' started by Andreas, Feb 2, 2022.

  1. Andreas

    Andreas New Member

    12
    0
    1
    Jan 20, 2022
    Ratings:
    +0
    Local Time:
    10:28 PM
    Please fill in any relevant information that applies to you:
    • CentOS Version: CentOS 7 64bit
    • Centmin Mod Version Installed: 123.09beta01
    • Nginx Version Installed: 1.21.1
    • PHP Version Installed: 8.1
    • MariaDB MySQL Version Installed: latest
    • When was last time updated Centmin Mod code base ? : installed fresh php 8.1 yesterday

    Hi Guys,


    so, I have an interesting problem here:
    I followed the instructions of guided hacking >

    and installed centminmod on centOS 7 (digital ocean), using php 8.1
    in combination with cloudflare.

    Everything work correct; after the first tutorial video I was able to switch to "full (strict)" mode inside of my cloudlfare dashboard
    (letsencrypt certificate gives the ability to do so).

    Then, in the 2nd tut. video I made a mistake, and figured to delete all and start a new machine, with same settings, and re-install.

    Of course I also deleted my site (domain) account over at cloudflare, before proceeding.


    So, after starting the install process and setting up a new site account at cloudflare
    (with same domain, but now of course with diff. IP),

    I realized two things:

    1) The advanced settings, which I had in my 'old' cloudflare site account,
    had been taken over into my new site account
    (although I deleted my old one before).

    So, I had adv. settings there, and my url was out of reach therefore.
    I had to switch all settings back to default, to be able to reach the site via browser and proceed the install process.

    2) After I finished the installation,
    I should be able to switch from "full" to "full (strict)" inside my cf dashboard -
    and which had worked before (old install).

    But this time, it ain't working any more when I switch to 'full /strict'.


    So, the question now is why?

    I personally think, it is because of the changed IP adress now
    (as I started a new machine).
    I assume, letsencrypt is still operating with the old IP adress, when I first installed it.


    Can someone please tell me, what I can do now?

    Shall I somehow upgrade the ip adress of my letsencrypt certificate?

    Where can I find that?


    (Just btw, as the guy from guided hacking suggests:
    Is there the ability to replace the letsencyrpt certificate with a cloudflare universal certificate?)


    Thanks
    Andy
     
  2. eva2000

    eva2000 Administrator Staff Member

    55,155
    12,249
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,825
    Local Time:
    6:28 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    You'd have to contact Cloudflare tech support for an exact explanation. Did you use same Cloudflare account or created a new Cloudflare account to re-add the new site zone?

    IIRC, if you create 2 Cloudflare accounts and add the same domain zone to both, the later one will override the first Cloudflare's system once the Cloudflare nameservers have propgrated to your domain's DNS.
     
  3. Andreas

    Andreas New Member

    12
    0
    1
    Jan 20, 2022
    Ratings:
    +0
    Local Time:
    10:28 PM
    Hi George,

    no, I used the same cloudflare account for that.

    Just deleted the site account, and then opened up again after approx. 15 minutes.


    If its not cloudflare faults, which I don't know,
    it might have to do with the letsencrypt certificate ip binding, after the first installation.

    However, got contradictive info here:
    Read on another forum, that ssl certificates are not bind to ip adresses, but solely domain names...

    If this is the case, it is even more strange why my l.e. certificate won't work with cloudflare, as it worked on first try...



    Will try to contact cloud flare, but I am not a paying customer.

    All I saw is their own forum (submitted a new topic there), or a general help site.

    Does anyone know whether there is any way to reach them by contact form?

    Thanks again.
     
  4. eva2000

    eva2000 Administrator Staff Member

    55,155
    12,249
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,825
    Local Time:
    6:28 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Hi I see you posted on Cloudflare community forums too Interesting Problem; CA / certificate not working.
    Not 100% sure I understand what you did but do you mean
    1. You deleted your Cloudflare domain's zone and re-added it back again in 15minutes. That's probably not enough time between deleting and re-adding the same domain zone and Cloudflare's system would still have your previous domain zone settings in their system still.
    2. And all your Cloudflare DNS records from BEFORE deletion were still there and re-added when you re-added your same domain name again and pointing to an old server IP address even before you got a chance to update the DNS records with new server IP? Or do you mean advance setting as in Cloudflare Full SSL strict was enabled by default when you re-added your domain zone the 2nd time? Or both old DNS IP from the previous domain zone setup + Cloudflare Full SSL strict was enabled too?
    3. Thus Letsencrypt failed to re-validate and re-issue a new Letsencrypt SSL certificate for your Centmin Mod Nginx origins server on the fresh 2nd install/setup as Cloudflare Full SSL strict was enabled from the first time you added your domain zone? If at this point you switched from CF Full SSL strict to either Full SSL non-strict or Flexible SSL, does your site work? CF Full SSL strict would probably fail due to either first 2 assumptions I outlined because CF Full SSL strict requires a valid Letsencrypt SSL certificate on Centmin Mod Nginx origin side and if your domain zone's DNS records picked up the previous domain zone's server IP for DNS A records, it would be pointing to a different origin server and not your new Centmin Mod origin server's IP. If that is the case then, all you need to do is temporarily switch from Cloudflare Full SSL to Flexible SSL and update your domain zone's DNS A records for domain and www version A DNS records to ensure it's pointing to the correct new origin server's IP and then you can do is sort of partial manual steps from Migrating Existing Nginx Vhost From HTTP to HTTP/2 based HTTPS With Letsencrypt SSL Certificates in that you temporarily disable your /usr/local/nginx/conf/conf.d/domain.com.ssl.conf nginx vhost and recreate the non-https nginx vhost /usr/local/nginx/conf/conf.d/domain.com.conf using the official Nginx vhost generator at Generate Centmin Mod Nginx Vhost - CentminMod.com LEMP Nginx web stack for CentOS (which is step 1 of guide at Migrating Existing Nginx Vhost From HTTP to HTTP/2 based HTTPS With Letsencrypt SSL Certificates).

      Then follow manual steps 2, 3, 4, 5 and 6 of guide at Migrating Existing Nginx Vhost From HTTP to HTTP/2 based HTTPS With Letsencrypt SSL Certificates where step 6 you can re-enable your https /usr/local/nginx/conf/conf.d/domain.com.ssl.conf nginx vhost and disable your non-https nginx vhost /usr/local/nginx/conf/conf.d/domain.com.conf again. Then you should be able to re-enable CF Full SSL strict.

    Using Letsencrypt DNS based domain validation instead



    However, if you're behind Cloudflare and don't want to deal with the default Centmin Mod's Letsencrypt web root authentication, Centmin Mod's Letsencrypt SSL integration also supports using Letsencrypt DNS validation via Cloudflare's DNS API when you setup the optionally enabled variables for Cloudflare API Token is set in persistent config file at /etc/centminmod/custom_config.inc prior to creating your Centmin Mod Nginx HTTPS vhost domain name via centmin.sh menu option 2, 22 or nv command line. See https://community.centminmod.com/th...ull-cloudflare-dns-api-in-123-09beta01.20327/.

    If the Centmin Mod Nginx vhost site has no data and can be totally deleted, then probably easiest to delete the Nginx vhost and re-create it after you setup Cloudflare API Token for Letsencrypt DNS domain validation first.

    To properly remove an Nginx vhost the instructions are on official site at How to delete Nginx vhost account for existing domain/subdomain ? as well as on each Nginx vhost creation's ending output too lists the commands.

    If you used centmin.sh menu option 2, 22 or nv command line to create your Nginx vhost site, you'd also get a log file for each Nginx vhost created which also contains a lists of the commands in 123.09beta01 and higher. Example for http2.domain.com remove log at /root/centminlogs/centminmod_140218-021218_nginx_addvhost_nv-remove-cmds-http2.domain.com.log
    Code (Text):
    ls -lahrt /root/centminlogs/ | grep remove
    -rw-r--r--   1 root root 1.3K Feb 14 02:12 centminmod_140218-021218_nginx_addvhost_nv-remove-cmds-http2.domain.com.log
    
     
    Last edited: Feb 3, 2022
  5. Andreas

    Andreas New Member

    12
    0
    1
    Jan 20, 2022
    Ratings:
    +0
    Local Time:
    10:28 PM
    Thanks for your in-depth reply.

    Yeah,
    you understood it correctly (point 1/2):

    First I deleted the old instance (which had centminmod + vhost; domain installed),
    then I deleted my cloudflare domain account (site).


    After that I just set up a new machine on digital ocean, centOS 7,
    took the same install route as before (centmin 8.1 php),
    and after I initiated the install process for centminmod, which takes quite some time,
    I went over to cloudflare to set up a new domain/site account for my domain, which I had used before
    (using the same e-mail account which I had used before).


    So, then I realized:
    a) advanced settings, which I had made in my 'old' site account, during the install procedure, were still valid.
    > it was not like default, where many settings are simply turned off
    (Guided hacking makes just a handful of upgrades in the cloudflare site account in video tutorial 1; I had followed them all).
    > I had to switch them manually back to default, to be able to reach my site via browser again (which succeeded)

    b) After install. process of centminmod, and vhost generation for my domain,
    I went back over to cloudflare and tried to switch from 'full' (which works) to 'full/strict',
    which resulted in the message on my site:

    No valid SSL certificate.


    When installing for the first time, and taking the same procedure,
    full/strict worked.
    (I refreshed the page afterwards quite many times, and it worked).



    I think I am going to try the following, before I do any further investigation:

    As I have not come far in my install process,
    a) I will delete the vhost again,
    b) I will completely delete the site account at cloudflare

    I will wait an hour or so,
    then I will re-open a site account at cloudflare, for the same domain
    I will check the settings, and turn them back to default, if needed.
    Then I will re-install the vhost.

    Afterwards, I will let you know how it is going.
     
    Last edited: Feb 3, 2022
  6. Andreas

    Andreas New Member

    12
    0
    1
    Jan 20, 2022
    Ratings:
    +0
    Local Time:
    10:28 PM
    Yep, the same problem still current.

    But this time I made a few notes:

    After reopening my cloudflare site account, with the same domain, I saw the following:

    1) IP adress was still saved.


    2) The following settings were on, like when I had deleted the site account
    (default is: they are off):
    - SSL/TLS encrypt mode: full (instead of flexible)
    - Always use https: on
    - Auto. https rewrites: on
    - Auth. origin pulls: on
    - page rules: on (and still saved)
    - Firewall / bot fight mode: on

    I switched them all back to off.


    Then I started to add my vhost nginx domain name in centminmod:

    use letsencrypt was selected (yes) -
    all just like in the guided hacking tutorial video 1.


    I selected (4) for SSL generation.

    Then I observed the following:
    The first time, when I installed my vhost nginx domain, the SSL generation took quite a long time (approx. 2 mins).

    This time, it just took around 30 sec.

    And as you can see in the screenshot, the space used for it, is not much /
    it had been a lot more on initial installation:

    ssl cert generation.PNG


    So may, at some time, the system realizes, there is already a letsencrypt certificate,
    ends the SSL generation, and takes the old generated l.e. certificate -
    which is unfortunately still binned to my old IP (IP of my old machine).

    Does that make any sense?


    ----------------------------------------------------------------------------------------------
    Just btw, the deletion of my prevoius vhost nginx domain was made by using the log commands:

    pure-pw userdel Admin
    rm -rf /usr/local/nginx/conf/conf.d/domain.com.conf
    rm -rf /usr/local/nginx/conf/conf.d/domain.com.ssl.conf
    rm -rf /usr/local/nginx/conf/ssl/domain.com/domain.com.crt
    rm -rf /usr/local/nginx/conf/ssl/domain.com/domain.com.key
    rm -rf /usr/local/nginx/conf/ssl/domain.com/domain.com.csr
    rm -rf /usr/local/nginx/conf/ssl/domain.com
    rm -rf /home/nginx/domains/domain.com
    rm -rf /root/.acme.sh/domain.com
    rm -rf /root/.acme.sh/domain.com_ecc
    rm -rf /usr/local/nginx/conf/pre-staticfiles-local-domain.com.conf
    service nginx restart

    Don't think I made any mistakes here.
     
    Last edited: Feb 3, 2022
  7. eva2000

    eva2000 Administrator Staff Member

    55,155
    12,249
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,825
    Local Time:
    6:28 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    what is the problem exactly ?

    If it's letsencrypt SSL certificate issuance, then try below troubleshooting steps

    First try running your intended SSL certificate domain through the letsdebug.net online testing tool to check for potential errors with HTTP-01 validation. If you created Centmin Mod 123.09beta01 or higher Nginx site with Letsencrypt via centmin.sh menu option 2, 22 or nv command line, you now also have an automatic letsdebug.net API check log saved at /root/centminlogs/letsdebug-yourdomain.com-${DT}.log where yourdomain.com is domain specified during nginx vhost creation and DT is date/timestamp. Inspecting the /root/centminlogs/letsdebug-yourdomain.com-${DT}.log log will also give you clues as to why letsencrypt SSL certificate issuance failed.

    How was the initial letsencrypt ssl certificate obtained ? Which method ?
    • Was the domain nginx vhost alreadying created prior or new domain nginx vhost site setup for first time ?
    • Via centmin.sh menu option 2, 22, /usr/bin/nv ?
    • If you ran centmin.sh menu option 2 or 22, which letsencrypt option did you select from
      Code (Text):
      -------------------------------------------------------------
      Setup full Nginx vhost + Wordpress + WP Plugins
      -------------------------------------------------------------
      
      Enter vhost domain name you want to add (without www. prefix): acme3.domain1.com
      
      Create a self-signed SSL certificate Nginx vhost? [y/n]: n
      Get Letsencrypt SSL certificate Nginx vhost? [y/n]: y
      
      You have 4 options:
      1. issue staging test cert with HTTP + HTTPS
      2. issue staging test cert with HTTPS default
      3. issue live cert with HTTP + HTTPS
      4. issue live cert with HTTPS default
      Enter option number 1-4: 1
      

    Centmin Mod Self-Signed SSL Fallback



    If you're seeing a Centmin Mod's self-signed ssl certificate instead of letsencrypt ssl certificate, then that's acmetool.sh and centminmod's fallback if letsencrypt verification fails to obtain letsencrypt ssl cert, it falls back to centmin mod self-signed ssl certificate on https port 443 side so to preserve the https nginx vhost

    Troubleshooting



    There are various steps you can do to troubleshoot failed letsencrypt issuances, renews, reissues etc.
    • acmetool.sh logs all command line or shell menu runs to log files at /root/centminlogs. To troubleshoot, copy the contents of the log run and post contents of log to pastebin.com or gist.github.com and share link in this thread. To find the log list the logs in ascending date order
      Code (Text):
      ls -lahrt /root/centminlogs
      .
    • For direct acmetool.sh runs, there should be a 2nd & 3rd & 4th log in format /root/centminlogs/centminmod_${DT}_nginx_addvhost_nv.log and /root/centminlogs/acmetool.sh-debug-log-$DT.log and /root/centminlogs/acmesh-issue_*.log or /root/centminlogs/acmesh-reissue_*.log which would need to be included via separate pastebin.com or gist.github.com post.
    • Enable acmetool.sh debug mode. In persistent config file at /etc/centminmod/custom_config.inc (create it if doesn't exist) add and enable acmetool.sh debug mode which gives much more verbose letsencrypt issuance process information when you re-run acmetool.sh or centmin.sh menu options 2, 22 or /usr/bin/nv command lines.
      Code (Text):
      ACMEDEBUG='y'
    What's output for this command?
    Code (Text):
    echo y | /usr/local/src/centminmod/addons/acmetool.sh checkdates
    

    For posting code or output from commands to keep the formatting, you might want to use CODE tags for code How to use forum BBCODE code tags :)

    SSLLabs Test



    Also run your HTTPS domain site through SSLLabs tester at SSL Server Test (Powered by Qualys SSL Labs) if it says untrusted SSL cert and prompts to continue the test, continue the test.
     
  8. Andreas

    Andreas New Member

    12
    0
    1
    Jan 20, 2022
    Ratings:
    +0
    Local Time:
    10:28 PM
    Yes,
    I used 2,
    and all just like in your screenshot.
    [and that was exactly the way I had set it up for the first time].

    I had a look into the letsdebug - domain file,
    and yes, there is something:

    On the first time, I installed vhost nginx domain,
    it shows no problems.

    Then, from 2nd time forward, I have errors:

    - "name": "IssueFromLetsEncrypt",
    "explanation": "A test authorization for mydomain.com to the Let's Encrypt staging service has revealed issues that may prevent any certificate for this domain being issued.",
    "detail": "Fetching https://mydomain.com/.well-known/acme-challenge/__iGBUeXpoMjbPXT3DSfWpLDt8141fEmH0anZ0HnNGI: Redirect loop detected",
    "severity": "Error"
    },

    and

    - "name": "BadRedirect",
    "explanation": "Sending an ACME HTTP validation request to mydomain.com results in an unacceptable redirect. This is most likely a misconfiguration of your web server or your web application.",
    "detail": "Too many (10) redirects,...

    > I don't know what misconfiguratio might be ment;
    I did not do anything else than in the guided hacking vid. tutorial 1.


    When I check letsdebug.net,
    the info is:
    The domain.com is being served through Cloudflare CDN. Etc.
     
    Last edited: Feb 3, 2022
  9. eva2000

    eva2000 Administrator Staff Member

    55,155
    12,249
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,825
    Local Time:
    6:28 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    On Cloudflare end disable these 2 options you have to
    • Always use https: on
    • Auto. https rewrites: on
    Centmin Mod Nginx default HTTPS option will auto do non-https to https redirects. And if you enable Cloudflare side to do the same, you will end up in a redirect loop.
     
  10. Andreas

    Andreas New Member

    12
    0
    1
    Jan 20, 2022
    Ratings:
    +0
    Local Time:
    10:28 PM
    Will do that.
    However, what do you think about the other notification?

    "test authorization for mydomain.com to the Let's Encrypt staging service has revealed issues that may prevent any certificate for this domain being issued"

    ...

    Seems this has to do with letsencrypt itself(?)

     
    Last edited: Feb 3, 2022
  11. eva2000

    eva2000 Administrator Staff Member

    55,155
    12,249
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,825
    Local Time:
    6:28 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Did you select 2 or 4 when prompted below
    Code (Text):
    -------------------------------------------------------------
    Setup full Nginx vhost + Wordpress + WP Plugins
    -------------------------------------------------------------
    
    Enter vhost domain name you want to add (without www. prefix): acme3.domain1.com
    
    Create a self-signed SSL certificate Nginx vhost? [y/n]: n
    Get Letsencrypt SSL certificate Nginx vhost? [y/n]: y
    
    You have 4 options:
    1. issue staging test cert with HTTP + HTTPS
    2. issue staging test cert with HTTPS default
    3. issue live cert with HTTP + HTTPS
    4. issue live cert with HTTPS default
    Enter option number 1-4: 1
    

    2 will use staging untrusted Letsencrypt SSL certificate for tests. You need to select 4 for live trusted Letsencrypt SSL certificate issuance.

    Otherwise, the issue could be this redirect loop.
     
  12. Andreas

    Andreas New Member

    12
    0
    1
    Jan 20, 2022
    Ratings:
    +0
    Local Time:
    10:28 PM
    always selected 4.
     
  13. Andreas

    Andreas New Member

    12
    0
    1
    Jan 20, 2022
    Ratings:
    +0
    Local Time:
    10:28 PM
    I have done as you suggested with both options (always use https/auto https turned off),
    but the problem is still current.


    What I also wonder:
    Is there maybe a way to renew my letsencrypt certificate?
    (So, as I think it may has troubles to work correctly because of using my old ip adress)

    Otherwise, I will use your guide to set up an origin cloudflaret ssl token
    https://community.centminmod.com/th...ull-cloudflare-dns-api-in-123-09beta01.20327/


    One of these both solutions will be definitely the best.
     
    Last edited: Feb 4, 2022
  14. eva2000

    eva2000 Administrator Staff Member

    55,155
    12,249
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,825
    Local Time:
    6:28 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    You can try with Cloudflare DNS API validation too outlined at https://community.centminmod.com/th...with-letsencrypt-cloudflare.22446/#post-91917 which would bypass Letsencrypt web root file domain validation.

     
  15. Andreas

    Andreas New Member

    12
    0
    1
    Jan 20, 2022
    Ratings:
    +0
    Local Time:
    10:28 PM
    RESOLVED :eek:

    You can close the topic :)

    I had to forcefully renew my letsencrypt certificate via certbot


    https://www.cyberciti.biz/faq/how-to-forcefully-renew-lets-encrypt-certificate/
    https://community.letsencrypt.org/t/renew-letsencrypt-certificate/34677/2


    Thanks @eva2000 for the help & suggestions on my way to that solution (y)

     
    Last edited: Feb 4, 2022
  16. eva2000

    eva2000 Administrator Staff Member

    55,155
    12,249
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,825
    Local Time:
    6:28 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    You shouldn't need to do that to be honest and probably cause problems down the line with Centmin Mod's acme.sh implementation. Best way would of been CF DNS API method I outlined above.