Want to subscribe to topics you're interested in?
Become a Member

Woocommerce using Varnish, Hitch SSL, Cloudflare, Letsencrypt, NGINX with sockets

Discussion in 'Ecommerce / Shopping cart usage' started by Atrix, Jul 2, 2020.

  1. Atrix

    Atrix Member

    33
    12
    8
    Oct 7, 2018
    Ratings:
    +33
    Local Time:
    2:40 AM
    1.15.3
    MariaDB 10.1.36
    Forewarning to those embarking from here: I compiled this from memory and a history dump on the server. It may be missing essential lines, if you have any problems let me know and I'll look through my history file to see if I missed anything. 2nd disclaimer, it comes with no warranty, even implied, and I might not have time to help you much if it doesn't work. 3rd disclaimer, some of this might change, the site isn't production yet.

    All that said I'm happy with it so far, quite happy in fact!

    A lot of the ideas for this is taken from this blog entry and adapted for centminmod, I even quote some parts of it so credit first:
    Faster Web Server Stack Powered by Unix Sockets and PROXY protocol - GetPageSpeed

    ## BUT WHY? ##

    We're going to slightly change nginx to work via sockets for these benefits:
    Sockets (UDS) benefits include:
    * Bypassing network stack’s bottleneck, thus twice as fast with huge workloads
    * Security: UNIX domain sockets are subject to file system permissions, while TCP sockets are not. As a result, it is much easier to regulate which users have access to a UNIX domain socket than it is for a TCP socket
    * Unix sockets allow inter-process communication (IPC) between processes on the same machine.
    * The other major modern goodie aside from UDS is the PROXY protocol which allows forwarding the client's real IP address seamlessly.

    So UDS is exactly designed for what we’re after: running Varnish, NGINX and Hitch and having them talk to each other, on the same machine!

    Why we're using Hitch SSL:
    * NGINX is not capable of forwarding PROXY protocol via http proxy module. It means that server { proxy_pass .... } TLS termination to a Varnish which listens on a PROXY protocol, will not work.
    * However, NGINX is capable of forwarding PROXY protocol via stream module
    * However, nginx SSL stream + Varnish listening on PROXY protocol won’t support HTTP/2 because nginx SSL stream does not know how to negotiate ALPN proto.

    So now you know why we have to use Hitch in order to leverage both UDS and PROXY protocol. Simply because it can do both. In fact, you can think of Hitch as an NGINX with “stream module” which doesn’t have the second issue, and HTTP/2 works fine with it.

    Do not use this setup if you're going for a multiserver setup it's basically optimized for a single server setup.

    Also adding letsencrypt setup in a way that will be auto renewable with a script and be able to authenticate via your domain even if your site isn't live or accessible.
    Also adding cloudflare configurations.

    I saw this guide and wanted to try out centminmod, which I love dearly, with the power of sockets. So here's the guide.

    Final setup request will look like this:
    Browser -> Cloudflare 443 SSL -> Hitch 443 SSL -> (PROXY protocol) -> Varnish socket -> (regular HTTP, sorry) -> NGINX socket

    ## BUT WHY VARNISH? ##
    It's a fast and scriptable and trusted caching system for full page caching that will give you more robust results than anything else I have seen a wordpress plugin achieve. I actually found it easier to configure varnish than some plugins to actually work how they should too and I know the results work without bugs. I've used things like wp fastest cache before or the one that comes with centminmod too which is great if you don't need something more powerful.

    I had troubles with wp fastest cache interacting weird with some modules before like headway but varnish well, it just works once you have it setup in my experience. You might have to edit some rules though to suit your particular needs. It's also tested to be one of the fastest in the industry. I'd only do this if you feel you need it though and the default centminmod one doesn't work for you. For me it comes default that it doesn't full page cache, just a partial cache and I tried making it work with full page cache but didn't succeed. If I did succeed though I'd still not be able to program conf files to do exactly what I want so this is the path for me. Things like caching some AJAX requests but not other ones are valid concerns for me.

    ## INITIAL SITE SETUP ##

    If you haven't already, create a wordpress site using centminmod's menu option
    Code:
    22) Add Wordpress Nginx vhost + Cache Plugin
    
    IMPORTANT: Any time you see site.com in the rest of the guide replace it with the domain you made in this step!

    ## VARNISH INSTALL ##

    Add repo for Varnish 6.3
    file: /etc/yum.repos.d/varnishcache_varnish63.repo
    Code:
    [varnishcache_varnish63]
    name=varnishcache_varnish63
    baseurl=https://packagecloud.io/varnishcache/varnish63/el/7/$basearch
    repo_gpgcheck=1
    gpgcheck=0
    enabled=1
    gpgkey=https://packagecloud.io/varnishcache/varnish63/gpgkey
    sslverify=1
    sslcacert=/etc/pki/tls/certs/ca-bundle.crt
    metadata_expire=300
    
    [varnishcache_varnish63-source]
    name=varnishcache_varnish63-source
    baseurl=https://packagecloud.io/varnishcache/varnish63/el/7/SRPMS
    repo_gpgcheck=1
    gpgcheck=0
    enabled=1
    gpgkey=https://packagecloud.io/varnishcache/varnish63/gpgkey
    sslverify=1
    sslcacert=/etc/pki/tls/certs/ca-bundle.crt
    metadata_expire=300
    
    Uninstall any prior versions of varnish found with:
    Code:
    yum list installed | grep varnish
    
    Use yum remove ____ to uninstall them.

    Install Varnish 6.3
    Code:
    yum install varnish.x86_64
    
    Verify
    Code:
    varnishd -V
    
    Output:
    Code:
    varnishd (varnish-6.3.2 revision 6870fd661a2b42c2e8adad838b5d92a71f27dccd)
    Copyright (c) 2006 Verdens Gang AS
    Copyright (c) 2006-2019 Varnish Software AS
    
    Download this config bundle of vcl files I found, merged with wordpress vcls, and customized and modified for out setup:
    varnish

    extract the varnish folder in it so you end up with /etc/varnish/default.vcl

    generate the secret file
    Code:
    dd if=/dev/random of=/etc/varnish/secret count=1
    

    ## HITCH AND LETSENCRYPT BOT INSTALL ##
    Download & install hitch and certbot
    Code:
    yum install hitch
    yum install certbot python2-certbot-dns-cloudflare
    semanage permissive -a varnishd_t
    semanage permissive -a httpd_t
    openssl dhparam -rand - 2048 | sudo tee /etc/hitch/dhparams.pem
    
    ## CLOUDFLARE API ##
    Get your cloudflare API key from your cloudflare account, they have instructions here:
    Managing API Tokens and Keys
    vi ~/.cloudflare.ini
    Code:
    # Cloudflare API credentials used by Certbot
    dns_cloudflare_email = name@site.com
    dns_cloudflare_api_key = APIKEYGOESHERE
    
    Code:
    chmod 0600 ~/.cloudflare.ini
    
    Use cloudflare DNS to get your valid SSL cert even though your site isn't online or accessible and make it auto renew for you:

    create this file named
    hitch-deploy-hook
    Code:
    #!/bin/bash
    # Full path to pre-generated Diffie Hellman Parameters file
    dhparams=/etc/hitch/dhparams.pem
    
    if [[ "${RENEWED_LINEAGE}" == "" ]]; then
        echo "Error: missing RENEWED_LINEAGE env variable." >&2
        exit 1
    fi
    
    umask 077
    cat ${RENEWED_LINEAGE}/privkey.pem \
    ${RENEWED_LINEAGE}/fullchain.pem \
    ${dhparams} > ${RENEWED_LINEAGE}/hitch-bundle.pem
    
    Code:
    cp hitch-deploy-hook /usr/bin
    chmod a+x /usr/bin/hitch-deploy-hook
    
    create a file ~/cert-letsencrypt.sh and put this in it
    Code:
    certbot certonly --dns-cloudflare --dns-cloudflare-credentials ~/.cloudflare.ini \
      -d site.com -d *.site.com \
      --email name@site.com --non-interactive --agree-tos \
      --deploy-hook="/usr/bin/hitch-deploy-hook" \
      --post-hook="service restart hitch"
    
    run this file
    Code:
    bash ~/cert-letsencrypt.sh
    
    Edit hitch config:
    vi /etc/hitch/hitch.conf
    Code:
    #Run 'man hitch.conf' for a description of all options.
    
    tls-protos = TLSv1.0 TLSv1.1 TLSv1.2
    
    pem-file = "/etc/letsencrypt/live/site.com/hitch-bundle.pem"
    
    frontend = {
            host = "*"
            port = "443"
    }
    
    backend = "/var/run/varnish/varnish.sock"
    workers = 4 # number of CPU cores
    
    daemon = on
    
    # We strongly recommend you create a separate non-privileged hitch
    # user and group
    user = "hitch"
    group = "hitch"
    
    # Enable to let clients negotiate HTTP/2 with ALPN:
    alpn-protos = "h2, http/1.1"
    
    # Varnish is our backend and it listens over PROXY
    write-proxy-v2 = on             # Write PROXY header
    
    modify group permissions
    Code:
     usermod -a -G varnish hitch
     service restart hitch
    
    Edit varnish startup
    We need to tell Varnish to listen on a socket (Hitch SSL will use it to talk to Varnish). That’s all where it will listen in our setup, being the caching layer of our app. However, we also want to keep it on a private HTTP port (6081) – this is going to be used only by external cache purging apps which can be easily configured with TCP details.

    Code:
    systemctl edit varnish
    
    Paste this:
    Code:
    [Service]
    ExecStart=
    ExecStart=/usr/sbin/varnishd -f /etc/varnish/default.vcl -s malloc,256m \
        -a /var/run/varnish/varnish.sock,PROXY,user=varnish,group=hitch,mode=660 \
        -a 127.0.0.1:6081 \
        -p feature=+http2
    

    Take care of socket permissions and files for startup and running
    Code:
    #Since we’re going to store socket files in a dedicated directory, /var/run/varnish, we will “tell” SELinux what we’re going to use the directory for:
    semanage fcontext -a -t varnishd_var_run_t "/var/run/varnish(/.*)?"
    
    #And we also make sure that our sockets directory is created after reboot:
    cat << _EOF_ >> /etc/tmpfiles.d/varnish.conf
      d /run/varnish 755 varnish varnish
    _EOF_
    
    #And we also create it right away via:
    systemd-tmpfiles --create varnish.conf
    
    #Make sure to setup correct SELinux label to our directory:
    semanage fcontext -a -t httpd_var_run_t "/var/run/nginx(/.*)?"
    
    #And ensure that the directory is created at boot time:
    cat << _EOF_ >> /etc/tmpfiles.d/nginx.conf
      d /run/nginx 750 nginx varnish
    _EOF_
    
    #And we also create it right away via:
    systemd-tmpfiles --create nginx.conf
    

    ## NGINX CONFIGS ##
    Nginx configuration for our hitch ssl, varnish setup
    Add this to line 59 of /usr/local/nginx/conf/nginx.conf
    Code:
    real_ip_header X-Forwarded-For;
    set_real_ip_from unix:;
    real_ip_recursive on;
    
    rename /usr/local/nginx/conf.d/site.com.ssl.conf to /usr/local/nginx/conf.d/site.com.ssl.conf-disabled
    This is because nginx will only be serving http requests because that is all varnish understands. Hitch does ALL the SSL work.

    I didn't modify a ton but for easier instructions here's the full config file for ssl
    vi /usr/local/nginx/conf.d/site.com.conf
    Code:
    # Centmin Mod Getting Started Guide
    # must read http://centminmod.com/getstarted.html
    
    # redirect from non-www to www
    # uncomment, save file and restart Nginx to enable
    # if unsure use return 302 before using return 301
    #server {
    #            listen   80;
    #            server_name site.com;
    #            return 301 $scheme://www.site.com$request_uri;
    #       }
    server {
       listen   80;
       #listen [::]:80 default_server;
       server_name site.com www.site.com;
       return 302 https://$server_name$request_uri;
    }
    server {
        listen unix:/var/run/nginx.sock;
        server_name www.site.com;
        return 301 https://$server_name$request_uri;
    }
    
    server {
      listen unix:/var/run/nginx.sock;
      server_name site.com www.site.com;
    
    # ngx_pagespeed & ngx_pagespeed handler
    #include /usr/local/nginx/conf/pagespeed.conf;
    #include /usr/local/nginx/conf/pagespeedhandler.conf;
    #include /usr/local/nginx/conf/pagespeedstatslog.conf;
    
      #add_header X-Frame-Options SAMEORIGIN;
      add_header X-Xss-Protection "1; mode=block" always;
      add_header X-Content-Type-Options "nosniff" always;
      #add_header Referrer-Policy "strict-origin-when-cross-origin";
      #add_header Feature-Policy "accelerometer 'none'; camera 'none'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; payment 'none'; usb 'none'";
    
      # limit_conn limit_per_ip 16;
      # ssi  on;
    
      access_log /home/nginx/domains/site.com/log/access.log combined buffer=256k flush=5m;
      error_log /home/nginx/domains/site.com/log/error.log;
    
      include /usr/local/nginx/conf/autoprotect/site.com/autoprotect-site.com.conf;
      root /home/nginx/domains/site.com/public;
      # uncomment cloudflare.conf include if using cloudflare for
      # server and/or vhost site
      #include /usr/local/nginx/conf/cloudflare.conf;
      include /usr/local/nginx/conf/503include-main.conf;
    
      #include /usr/local/nginx/conf/wpincludes/site.com/wpcacheenabler_site.com.conf;
      include /usr/local/nginx/conf/wpincludes/site.com/wpsupercache_site.com.conf;
      # https://community.centminmod.com/posts/18828/
      #include /usr/local/nginx/conf/wpincludes/site.com/rediscache_site.com.conf;
    
      location / {
      include /usr/local/nginx/conf/503include-only.conf;
     
    
      # Enables directory listings when index file not found
      #autoindex  on;
    
      # for wordpress super cache plugin
      try_files /wp-content/cache/supercache/$http_host/$cache_uri/index.html $uri $uri/ /index.php?q=$uri&$args;
    
      # for wp cache enabler plugin
      #try_files $cache_enabler_uri $uri $uri/ $custom_subdir/index.php?$args;
    
      # Wordpress Permalinks
      #try_files $uri $uri/ /index.php?q=$uri&$args;
    
      # Nginx level redis Wordpress
      # https://community.centminmod.com/posts/18828/
      #try_files $uri $uri/ /index.php?$args;
    
      }
    
    location ~* /(wp-login\.php) {
        limit_req zone=xwplogin burst=1 nodelay;
        #limit_conn xwpconlimit 30;
        #auth_basic "Private";
        #auth_basic_user_file /home/nginx/domains/site.com/htpasswd_wplogin;
        include /usr/local/nginx/conf/php-wpsc.conf;
        # https://community.centminmod.com/posts/18828/
        #include /usr/local/nginx/conf/php-rediscache.conf;
    }
    
    location ~* /(xmlrpc\.php) {
        limit_req zone=xwprpc burst=45 nodelay;
        #limit_conn xwpconlimit 30;
        include /usr/local/nginx/conf/php-wpsc.conf;
        # https://community.centminmod.com/posts/18828/
        #include /usr/local/nginx/conf/php-rediscache.conf;
    }
    
    location ~* /wp-admin/(load-scripts\.php) {
        limit_req zone=xwprpc burst=5 nodelay;
        #limit_conn xwpconlimit 30;
        include /usr/local/nginx/conf/php-wpsc.conf;
        # https://community.centminmod.com/posts/18828/
        #include /usr/local/nginx/conf/php-rediscache.conf;
    }
    
    location ~* /wp-admin/(load-styles\.php) {
        limit_req zone=xwprpc burst=5 nodelay;
        #limit_conn xwpconlimit 30;
        include /usr/local/nginx/conf/php-wpsc.conf;
        # https://community.centminmod.com/posts/18828/
        #include /usr/local/nginx/conf/php-rediscache.conf;
    }
    location = /favicon.ico {
      log_not_found off;
    }
    
      include /usr/local/nginx/conf/wpincludes/site.com/wpsecure_site.com.conf;
      include /usr/local/nginx/conf/php-wpsc.conf;
     
      # https://community.centminmod.com/posts/18828/
      #include /usr/local/nginx/conf/php-rediscache.conf;
      include /usr/local/nginx/conf/pre-staticfiles-local-site.com.conf;
      include /usr/local/nginx/conf/pre-staticfiles-global.conf;
      include /usr/local/nginx/conf/staticfiles.conf;
      include /usr/local/nginx/conf/drop.conf;
      #include /usr/local/nginx/conf/errorpage.conf;
      include /usr/local/nginx/conf/vts_server.conf;
    }
    

    create this file to restart your services, service nginx restart doesn't work becuase it doesn't remove the socket file for some reason
    I haven't found time for a full solution to that yet but stopping the service, removing the socket file works
    and this is a convenient way to restart your stack anyway
    on my todo list before I get to production level though

    create file ~/restartservices.sh
    Code:
    service nginx stop
    rm /var/run/nginx.sock
    service nginx start
    service php-fpm restart
    service varnish restart
    service hitch restart
    
    Ok! Server setup is done now for the wordpress side.
    ## WORDPRESS ##
    We need a few lines in your wp-config.php file since the requests are http and we're serving https wordpress needs to be forced or you'll get infinite loopbacks as wordpress tries to redirect to http but nginx redirects to https.
    We also set some headers so it's cloudflare friendly
    Code:
    //define( 'WP_DEBUG', false );
    //Cloudflare Flexible SSL redirect loop fix
    if ( isset( $_SERVER['HTTP_X_FORWARDED_PROTO'] ) && $_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https') $_SERVER['HTTPS']='on';
    
    define( 'JETPACK_SIGNATURE__HTTPS_PORT', 0 );
    

    Next disable any other caching modules.

    Then install these by going to /home/nginx/site.com/public and running it
    This plugin has been good to me so far no issues purgin after I setup the plugin properly, image here:
    Code:
    wp plugin install vcaching
    wp plugin activate vcaching
    
    Plugin settings screenshot HERE:
    File sharing and storage made simple

    You can copy the logged in cookie string from /etc/varnish/default.vcl

    I also suggest making your own logged in cookie string and replacing it in both locations for security.

    The plugin uses sessions so it can introduce session locking to wordpress which doesn't have it by default but since so many modules use sessions anyway it's not so much a downside as something to acknowledge.
    It's a feature rich module the best I could find and it's free!

    Test it out by hitting purge ALL varnish cache at the top if it works then it's doing purge requests right.

    type ifconfig and get your interface address so we can run a test that bypasses cloudflare and we get accurate headers

    Look for something like this in the output
    Code:
    eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
            inet 111.111.111.111
    
    111.111.111.111 will be different for you, use that IP

    on your work machine, not the server, we are going to bypass cloudflare to get accurate headers
    you'll have to google how to change the hosts file for your system, in windows it is
    C:\Windows\System32\drivers\etc\hosts and add this line
    Code:
    111.111.111.111 site.com
    
    now you will go directly to the server, you can comment it out later to test with cloudflare


    Load up a browser that isn't loaded up with any logins to the site or clear all cookies for the site you are testing.

    Use the network tab and look for response headers

    You'll get a reply like this after reloading the page a few times if everything is working:
    Code:
    accept-ranges: bytes
    age: 13
    content-encoding: br
    content-length: 13934
    content-type: text/html; charset=UTF-8
    date: Thu, 02 Jul 2020 02:15:36 GMT
    link: <https://site.com.com/wp-json/>; rel="https://api.w.org/"
    link: <https://site.com/>; rel=shortlink
    server: nginx centminmod
    status: 200
    vary: Accept-Encoding
    via: 1.1 varnish (Varnish/6.3)
    x-content-type-options: nosniff
    x-powered-by: centminmod
    x-varnish: 65751 786439
    x-vc-cache: HIT
    x-vc-cacheable: YES:Is cacheable, ttl: 1209600.000
    x-vc-debug: true
    x-vc-enabled: true
    x-vc-hash: /?cc=debug#site.com
    x-vc-ttl: 1209600
    x-xss-protection: 1; mode=block
    
    If it's not working then x-vc-cacheable might hint why.


    Notable headers to watch for:
    Code:
    via: 1.1 varnish (Varnish/6.3)
    x-varnish: 164036
    age: 0
    X-VC-Cache: MISS
    
    Age 0 and X-VC-Cache: MISS means it wasn't cached if it's >0 and HIT it's cached.

    some other tools:
    Code:
    varnishlog -c
    
    or you can run
    Code:
    varnishtop
    
    Final steps when it's all working and before you go production:
    in /etc/varnish/lib/static.vcl
    comment out this line 14:
    Code:
           #set req.http.X-VC-Debug = "true";
    
    This will prevent everyone and their dog from knowing what your cache server is doing, you can still get it by adding the word debug somewhere in the url.

    Last is set you SSL/TLS to FULL on cloudflare. Since we have a letsencrypt cert you might even be able to use FUll (strict) but I haven't tested it yet.

    Annnd that's it, enjoy :)


    ## TIPS AND RESULTS ##

    You can use the wordpress vcaching plugin stats page to make sure it's actually caching pages the Cache Hits counter should keep going up.

    My initial testing was pretty positive for the entire stack the best I was able to get was about 85ms TTFB on a page for a served php file that says hello world. For wordpress it'll increase but a fully cached page shouldn't take a ton longer I have a 200ms TTFB for the home page on it now and I currently have a lot of things turned off like opcode caching etc so I can develop quickly.

    You'll probably want to fiddle with it through cloudflare too, it will add time to your TTFB scores but there's a few workarounds like setting a page rule to do site.com/* cache everything, though this could have other effects so use at your discretion. I use cloudflare for security mostly, never had a hacked site since I started using it and it saves you on bandwidth and speeds up your site in other ways.

    ## THANKS ##

    Hope this helps anyone wanting to build woocommerce sites with a more robust caching engine.

    If you're building a big site too I can also suggest using the Scalability Pro plugin to make it work with up to a million products, even more. It's what made us able to consider woo.
     
    Last edited: Jul 2, 2020
  2. ahmed

    ahmed Active Member

    338
    48
    28
    Feb 21, 2017
    Ratings:
    +60
    Local Time:
    10:40 AM
    superb I will try it thank you so much for sharing
     
  3. ahmed

    ahmed Active Member

    338
    48
    28
    Feb 21, 2017
    Ratings:
    +60
    Local Time:
    10:40 AM
    I have a few questions:
    1- why use Hitch and add an extra app? cant we do this in Nginx
    Visitor > Nginx HTTP/2 HTTPS Proxy > Varnish Cache HTTP/2 Enabled > Nginx non-HTTPS backend

    2-I suggest putting the VCL on a direct URL to allow for wget during install
     
  4. ahmed

    ahmed Active Member

    338
    48
    28
    Feb 21, 2017
    Ratings:
    +60
    Local Time:
    10:40 AM
    Do not use this setup if you're going for a multiserver setup it's basically optimized for a single server setup.

    I don't quite get that, do u mean I can not use another domain on the same server?
     
  5. Atrix

    Atrix Member

    33
    12
    8
    Oct 7, 2018
    Ratings:
    +33
    Local Time:
    2:40 AM
    1.15.3
    MariaDB 10.1.36
    1) I actually explain that in the guide. It should end up a faster stack because hitch is light weight and tested to be faster for SSL handling and we're using sockets and nginx doesn't support all the stuff we have to use to do sockets properly.

    2) Fair enough I just don't have a server I'm comfortable to put file downloads on atm since they don't belong to me but my clients.

    If you intended on having a varnish server separate from your nginx server or use separate servers somehow else in the stack. Since we're using unix sockets it has to all be done on a linux file level. Meaning it's designed to have the entire stack run on one server. You can add as many domains to that server as you want though :)
     
  6. ahmed

    ahmed Active Member

    338
    48
    28
    Feb 21, 2017
    Ratings:
    +60
    Local Time:
    10:40 AM
    Thanks a lot, I'm going to grab a cup of coffee and work on that tutorial, did u compare it with cache enabler Nginx customization?
     
  7. Atrix

    Atrix Member

    33
    12
    8
    Oct 7, 2018
    Ratings:
    +33
    Local Time:
    2:40 AM
    1.15.3
    MariaDB 10.1.36
    Good luck hopefully I didn't miss any commands if you find something weird let me know. I didn't compare it directly with any other setups so far. I don't have time to setup multiple environments and run tests unfortunately :(

    What I knew going in is that sockets should have more throughput than TCP stack and that hitch was tested to be faster than nginx for ssl requests and supports more technology with it and that I was settled on using varnish because I've used it before and been very happy with it.
     
    Last edited: Jul 3, 2020
  8. ahmed

    ahmed Active Member

    338
    48
    28
    Feb 21, 2017
    Ratings:
    +60
    Local Time:
    10:40 AM
    for certbot step I guess centmin already had it, why install it?

    Im stuck at that step anyway
    Im aware it is phyto issue I'm working on it


    Code:
    -> Finished Dependency Resolution
    Error: Package: python-requests-2.6.0-9.el7_8.noarch (updates)
               Requires: python-urllib3 >= 1.10.2-1
     You could try using --skip-broken to work around the problem
     You could try running: rpm -Va --nofiles --nodigest
     
  9. Atrix

    Atrix Member

    33
    12
    8
    Oct 7, 2018
    Ratings:
    +33
    Local Time:
    2:40 AM
    1.15.3
    MariaDB 10.1.36
    I didn't have it on mine I think that's optional with centminmod, I tried it before. You can either try to get the one you have working or try installing the way I did as I remember it ran the steps I did for me. Not sure if there's any diff versions. For the dependency issue try just doing a yum search for the package it claims to need and then installing it.
     
  10. eva2000

    eva2000 Administrator Staff Member

    44,780
    10,212
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,828
    Local Time:
    6:40 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Thanks for sharing @Atrix. Interesting way of doing things with Hitch and Varnish Cache. Will let users decide if that is the way they'd want to go. Just know I wouldn't be able to provide any support or help if things go wrong. Folks would be on their own or require Atrix's assistance :)

    As to certbot, you don't need it as Centmin Mod as addons/acmetool.sh with underlying acme.sh client for letsencrypt SSL certificate issuance which supports Cloudflare DNS API domain validation as outlined at Letsencrypt - Official acmetool.sh testing thread for Centmin Mod 123.09beta01 just will need to use acme.sh directly and use equivalent deploy hook and post hooks for acme.sh acmesh-official/acme.sh

    So for Cloudflare DNS API with acme.sh acmesh-official/acme.sh

    I believe it might be something like using Cloudflare API Tokens
    Code (Text):
    export CF_Token="sdfsdfsdfljlbjkljlkjsdfoiwje"
    export CF_Account_ID="xxxxxxxxxxxxx"
    

    for domain1.com letsencrypt SSL wildcard certificate the ~/cert-letsencrypt.sh file would have
    Code (Text):
    /root/.acme.sh/acme.sh --issue --dns dns_cf -d domain1.com -d '*.domain1.com' -k 2048 --useragent centminmod-centos7-acmesh-dns --renew-hook="/usr/bin/hitch-deploy-hook" --post-hook="service reload hitch"
    

    then in /etc/hitch/hitch.conf this would change paths from certbot location to acme.sh location

    from
    Code (Text):
    pem-file = "/etc/letsencrypt/live/site.com/hitch-bundle.pem"
    

    to
    Code (Text):
    pem-file = "/root/.acme.sh/domain1.com/hitch-bundle.pem"
    

    Create the acme.sh equivalent hitch-bundle.pem at /root/.acme.sh/domain1.com/hitch-bundle.pem
    Code (Text):
    cd /root/.acme.sh/domain1.com
    cat domain1.com.key fullchain.cer > hitch-bundle.pem
    

    Then change hitch-deploy-hook file's paths/names for fullchain.pem to fullchain.cer and privkey.pem to domain1.com.key

    from
    Code (Text):
    #!/bin/bash
    # Full path to pre-generated Diffie Hellman Parameters file
    dhparams=/etc/hitch/dhparams.pem
    
    if [[ "${RENEWED_LINEAGE}" == "" ]]; then
       echo "Error: missing RENEWED_LINEAGE env variable." >&2
       exit 1
    fi
    
    umask 077
    cat ${RENEWED_LINEAGE}/privkey.pem \
    ${RENEWED_LINEAGE}/fullchain.pem \
    ${dhparams} > ${RENEWED_LINEAGE}/hitch-bundle.pem
    

    to
    Code (Text):
    #!/bin/bash
    # Full path to pre-generated Diffie Hellman Parameters file
    dhparams=/etc/hitch/dhparams.pem
    
    if [[ "${RENEWED_LINEAGE}" == "" ]]; then
       echo "Error: missing RENEWED_LINEAGE env variable." >&2
       exit 1
    fi
    
    umask 077
    cat ${RENEWED_LINEAGE}/domain1.com.key \
    ${RENEWED_LINEAGE}/fullchain.cer \
    ${dhparams} > ${RENEWED_LINEAGE}/hitch-bundle.pem
    

    Haven't done this before with Hitch so not 100% sure, you'd need to figure it out the rest
     
    Last edited: Jul 3, 2020
  11. Atrix

    Atrix Member

    33
    12
    8
    Oct 7, 2018
    Ratings:
    +33
    Local Time:
    2:40 AM
    1.15.3
    MariaDB 10.1.36
    Cool thanks eva2000, the ability to do all this has only been around fully since about 2018 I think so it's pretty new. Integrating with the centminmod addon is definitely more ideal I'll update the guide when I get chances to test it.

    I also noticed I need to build hitch against openssl 1.1.1 or newer to be able to use tls 1.3 which I really want to do it's a major upgrade. I will post that too when I get time to do it and put in the guide.

    The reason I was never too fond of wp super cache was simply I couldn't get any nginx rules setup that would reliably let it fully store html files of a page, it used the other caching method which is slower as it actually runs PHP, and it seemed the amount of caching it did was less, to fix this there's settings and rules for apache but not nginx. WP Super Cache still serviced my needs for some sites though and it's already setup. Wp Fastest Cache would work with some custom nginx rules I found and modified and cache a result as html but it didn't work well for all WP sites. Varnish serves fully cached pages and also lets me write VCL rules so it's just what I need for a big woo site. I still think the work you do is great tho just explaining my thought process. You've definitely made my life much easier.
     
  12. ahmed

    ahmed Active Member

    338
    48
    28
    Feb 21, 2017
    Ratings:
    +60
    Local Time:
    10:40 AM
    Thank you both for valuable inputs

    I guess this is a must just in case you will have to install multiple domains later
     
  13. ahmed

    ahmed Active Member

    338
    48
    28
    Feb 21, 2017
    Ratings:
    +60
    Local Time:
    10:40 AM
    I can not move forward
    maybe google distro is having photon issues, but how to bypass that bug-I know I can not upgrade phyton

    Code:
     More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/developm
    ent/release-process/#python-2-support
    ERROR: Could not find a version that satisfies the requirement python-urllib3 (from versions: none)
    ERROR: No matching distribution found for python-urllib3
    [root@instance-2 varnish]# sudo pip install python
    DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Pytho
    n as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021.
     More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/developm
    ent/release-process/#python-2-support
    Requirement already satisfied: python in /usr/lib64/python2.7/lib-dynload (2.7.5)
    [root@instance-2 varnish]# sudo pip uninstall python
    DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Pytho
    n as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021.
     More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/developm
    ent/release-process/#python-2-support
    Found existing installation: Python 2.7.5
    ERROR: Cannot uninstall 'Python'. It is a distutils installed project and thus we cannot accurately
     determine which files belong to it which would lead to only a partial uninstall.
    [root@instance-2 varnish]# sudo yum  uninstall python
    Loaded plugins: fastestmirror, versionlock
    No such command: uninstall. Please use /bin/yum --help
    [root@instance-2 varnish]# sudo pip uninstall python
    DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Pytho
    n as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021.
     More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/developm
    ent/release-process/#python-2-support
    Found existing installation: Python 2.7.5
    ERROR: Cannot uninstall 'Python'. It is a distutils installed project and thus we cannot accurately
     determine which files belong to it which would lead to only a partial uninstall.
    [root@instance-2 varnish]# sudo pip update python
    ERROR: unknown command "update"
    [root@instance-2 varnish]# sudo pip upgrade python
    ERROR: unknown command "upgrade"
    [root@instance-2 varnish]# sudo pip innstall python -upgrade
    ERROR: unknown command "innstall" - maybe you meant "install"
    [root@instance-2 varnish]# sudo pip innstall python --upgrade
    ERROR: unknown command "innstall" - maybe you meant "install"
    [root@instance-2 varnish]# sudo pip install python --upgrade
    DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Pytho
    n as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021.
     More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/developm
    ent/release-process/#python-2-support
    Requirement already up-to-date: python in /usr/lib64/python2.7/lib-dynload (2.7.5)
    [root@instance-2 varnish]#
     
  14. eva2000

    eva2000 Administrator Staff Member

    44,780
    10,212
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,828
    Local Time:
    6:40 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    what are you trying to do ? install certbot ? you don't need it if you have acme.sh installed.

    If you're trying to upgrade CentOS 7's Python 2.7 to a newer version you can only do side by side installs of newer Python 2.7 versions. You can install new Python 3.4, 3.5, or 3.6 besides CentOS 7's Python 2.7 using via Centmin Mod 123.09beta01's addons/python34_install.sh, addons/python35_install.sh, addons/python36_install.sh and then setup Python 3.4+ binary virtual environment for install just your specific app's Python related apps within that virtual environment without polluting the CentOS 7's system environment as outlined at Sysadmin - CentOS 7 Python 2.7 will reach the end of its life on January 1st, 2020.

    If you need Python 3.7, then you need to source compile it on CentOS 7 and then use a Python virtual environment for Python 3.7 apps as outlined here.
     
  15. ahmed

    ahmed Active Member

    338
    48
    28
    Feb 21, 2017
    Ratings:
    +60
    Local Time:
    10:40 AM
    I give up with the phyton thing on Atrix guide (my bad)

    I'm now on acmh guide

    for this step

    /root/.acme.sh/acme.sh

    I need to create new file? with the acme.sh alone or get the whole repo acmesh-official/acme.sh
     
  16. Atrix

    Atrix Member

    33
    12
    8
    Oct 7, 2018
    Ratings:
    +33
    Local Time:
    2:40 AM
    1.15.3
    MariaDB 10.1.36
    I didn't have those errors so I can't really say where it went wrong for you. I didn't use the centminmod addon i just used yum to install it. I don't have time to do more work on it atm to integrate it cuz I'm doubled up on projects but I'd suggest either trying what eva suggested or uninstalling the addon and trying what I did. Doing both ways sounds like it won't work.
     
  17. ahmed

    ahmed Active Member

    338
    48
    28
    Feb 21, 2017
    Ratings:
    +60
    Local Time:
    10:40 AM
    so I will go with your code, let's state the facts


    it worked for u and u are on centmin

    now I installed a new centos7 server and the code worked

    and then installed centmin beta 73 and 74 and then the code did not work

    so it has something to do with centmin

    The problem is in this

    Error: Package: python-requests-2.6.0-9.el7_8.noarch (updates) Requires: python-urllib3 >= 1.10.2-1


    although it is there and on the 1.25.9

    but it does not see it

    proof
    Code:
    pip uninstall urllib3
    gives this

    Code:
    Found existing installation: urllib3 1.25.9
    Uninstalling urllib3-1.25.9:
      Would remove:
        /usr/lib/python2.7/site-packages/urllib3-1.25.9.dist-info/*
        /usr/lib/python2.7/site-packages/urllib3/*
    I hope Eva can help on that
     
  18. eva2000

    eva2000 Administrator Staff Member

    44,780
    10,212
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,828
    Local Time:
    6:40 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    for that just use pip uninstall for urllib3 and then do yum install and then reinstall
    urllib3 via pip
     
  19. ahmed

    ahmed Active Member

    338
    48
    28
    Feb 21, 2017
    Ratings:
    +60
    Local Time:
    10:40 AM
    Code:
    [18:48][root@test aa]# yum install urllib3
    Loaded plugins: fastestmirror, versionlock
    Loading mirror speeds from cached hostfile
     * base: mirrors.codec-cluster.org
     * epel: d2lzkl7pfhq30w.cloudfront.net
     * extras: sjc.edge.kernel.org
     * updates: mirrors.vcea.wsu.edu
    No package urllib3 available.
    Error: Nothing to do
    that what i get with yum
     
  20. eva2000

    eva2000 Administrator Staff Member

    44,780
    10,212
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,828
    Local Time:
    6:40 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    yum package is = python-urllib3 which is yum excluded by default so need to disable exclusion
    Code (Text):
    pip uninstall urllib3
    yum install python-urllib3 --disableexcludes=main