Woocommerce using Varnish, Hitch SSL, Cloudflare, Letsencrypt, NGINX with sockets

Forewarning to those embarking from here: I compiled this from memory and a history dump on the server. It may be missing essential lines, if you have any problems let me know and I'll look through my history file to see if I missed anything. 2nd disclaimer, it comes with no warranty, even implied, and I might not have time to help you much if it doesn't work. 3rd disclaimer, some of this might change, the site isn't production yet.

All that said I'm happy with it so far, quite happy in fact!

A lot of the ideas for this is taken from this blog entry and adapted for centminmod, I even quote some parts of it so credit first:
Faster Web Server Stack Powered by Unix Sockets and PROXY protocol - GetPageSpeed

## BUT WHY? ##

We're going to slightly change nginx to work via sockets for these benefits:
Sockets (UDS) benefits include:
* Bypassing network stack’s bottleneck, thus twice as fast with huge workloads
* Security: UNIX domain sockets are subject to file system permissions, while TCP sockets are not. As a result, it is much easier to regulate which users have access to a UNIX domain socket than it is for a TCP socket
* Unix sockets allow inter-process communication (IPC) between processes on the same machine.
* The other major modern goodie aside from UDS is the PROXY protocol which allows forwarding the client's real IP address seamlessly.

So UDS is exactly designed for what we’re after: running Varnish, NGINX and Hitch and having them talk to each other, on the same machine!

Why we're using Hitch SSL:
* NGINX is not capable of forwarding PROXY protocol via http proxy module. It means that server { proxy_pass .... } TLS termination to a Varnish which listens on a PROXY protocol, will not work.
* However, NGINX is capable of forwarding PROXY protocol via stream module
* However, nginx SSL stream + Varnish listening on PROXY protocol won’t support HTTP/2 because nginx SSL stream does not know how to negotiate ALPN proto.

So now you know why we have to use Hitch in order to leverage both UDS and PROXY protocol. Simply because it can do both. In fact, you can think of Hitch as an NGINX with “stream module” which doesn’t have the second issue, and HTTP/2 works fine with it.

Do not use this setup if you're going for a multiserver setup it's basically optimized for a single server setup.

Also adding letsencrypt setup in a way that will be auto renewable with a script and be able to authenticate via your domain even if your site isn't live or accessible.
Also adding cloudflare configurations.

I saw this guide and wanted to try out centminmod, which I love dearly, with the power of sockets. So here's the guide.

Final setup request will look like this:
Browser -> Cloudflare 443 SSL -> Hitch 443 SSL -> (PROXY protocol) -> Varnish socket -> (regular HTTP, sorry) -> NGINX socket

## BUT WHY VARNISH? ##
It's a fast and scriptable and trusted caching system for full page caching that will give you more robust results than anything else I have seen a wordpress plugin achieve. I actually found it easier to configure varnish than some plugins to actually work how they should too and I know the results work without bugs. I've used things like wp fastest cache before or the one that comes with centminmod too which is great if you don't need something more powerful.

I had troubles with wp fastest cache interacting weird with some modules before like headway but varnish well, it just works once you have it setup in my experience. You might have to edit some rules though to suit your particular needs. It's also tested to be one of the fastest in the industry. I'd only do this if you feel you need it though and the default centminmod one doesn't work for you. For me it comes default that it doesn't full page cache, just a partial cache and I tried making it work with full page cache but didn't succeed. If I did succeed though I'd still not be able to program conf files to do exactly what I want so this is the path for me. Things like caching some AJAX requests but not other ones are valid concerns for me.

## INITIAL SITE SETUP ##

If you haven't already, create a wordpress site using centminmod's menu option

Code:

22) Add Wordpress Nginx vhost + Cache Plugin

IMPORTANT: Any time you see site.com in the rest of the guide replace it with the domain you made in this step!

---

## VARNISH INSTALL ##

Add repo for Varnish 6.3
file: /etc/yum.repos.d/varnishcache_varnish63.repo

Code:

[varnishcache_varnish63]
name=varnishcache_varnish63
baseurl=https://packagecloud.io/varnishcache/varnish63/el/7/$basearch
repo_gpgcheck=1
gpgcheck=0
enabled=1
gpgkey=https://packagecloud.io/varnishcache/varnish63/gpgkey
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
metadata_expire=300

[varnishcache_varnish63-source]
name=varnishcache_varnish63-source
baseurl=https://packagecloud.io/varnishcache/varnish63/el/7/SRPMS
repo_gpgcheck=1
gpgcheck=0
enabled=1
gpgkey=https://packagecloud.io/varnishcache/varnish63/gpgkey
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
metadata_expire=300

Uninstall any prior versions of varnish found with:

Code:

yum list installed | grep varnish

Use yum remove ____ to uninstall them.

Install Varnish 6.3

Code:

yum install varnish.x86_64

Verify

Code:

varnishd -V

Output:

Code:

varnishd (varnish-6.3.2 revision 6870fd661a2b42c2e8adad838b5d92a71f27dccd)
Copyright (c) 2006 Verdens Gang AS
Copyright (c) 2006-2019 Varnish Software AS

Download this config bundle of vcl files I found, merged with wordpress vcls, and customized and modified for out setup:
varnish

extract the varnish folder in it so you end up with /etc/varnish/default.vcl

generate the secret file

Code:

dd if=/dev/random of=/etc/varnish/secret count=1

## HITCH AND LETSENCRYPT BOT INSTALL ##
Download & install hitch and certbot

Code:

yum install hitch
yum install certbot python2-certbot-dns-cloudflare
semanage permissive -a varnishd_t
semanage permissive -a httpd_t
openssl dhparam -rand - 2048 | sudo tee /etc/hitch/dhparams.pem

## CLOUDFLARE API ##
Get your cloudflare API key from your cloudflare account, they have instructions here:
Managing API Tokens and Keys
vi ~/.cloudflare.ini

Code:

# Cloudflare API credentials used by Certbot
dns_cloudflare_email = name@site.com
dns_cloudflare_api_key = APIKEYGOESHERE

Code:

chmod 0600 ~/.cloudflare.ini

Use cloudflare DNS to get your valid SSL cert even though your site isn't online or accessible and make it auto renew for you:

create this file named
hitch-deploy-hook

Code:

#!/bin/bash
# Full path to pre-generated Diffie Hellman Parameters file
dhparams=/etc/hitch/dhparams.pem

if [[ "${RENEWED_LINEAGE}" == "" ]]; then
    echo "Error: missing RENEWED_LINEAGE env variable." >&2
    exit 1
fi

umask 077
cat ${RENEWED_LINEAGE}/privkey.pem \
${RENEWED_LINEAGE}/fullchain.pem \
${dhparams} > ${RENEWED_LINEAGE}/hitch-bundle.pem

Code:

cp hitch-deploy-hook /usr/bin
chmod a+x /usr/bin/hitch-deploy-hook

create a file ~/cert-letsencrypt.sh and put this in it

Code:

certbot certonly --dns-cloudflare --dns-cloudflare-credentials ~/.cloudflare.ini \
  -d site.com -d *.site.com \
  --email name@site.com --non-interactive --agree-tos \
  --deploy-hook="/usr/bin/hitch-deploy-hook" \
  --post-hook="service restart hitch"

run this file

Code:

bash ~/cert-letsencrypt.sh

Edit hitch config:
vi /etc/hitch/hitch.conf

Code:

#Run 'man hitch.conf' for a description of all options.

tls-protos = TLSv1.0 TLSv1.1 TLSv1.2

pem-file = "/etc/letsencrypt/live/site.com/hitch-bundle.pem"

frontend = {
        host = "*"
        port = "443"
}

backend = "/var/run/varnish/varnish.sock"
workers = 4 # number of CPU cores

daemon = on

# We strongly recommend you create a separate non-privileged hitch
# user and group
user = "hitch"
group = "hitch"

# Enable to let clients negotiate HTTP/2 with ALPN:
alpn-protos = "h2, http/1.1"

# Varnish is our backend and it listens over PROXY
write-proxy-v2 = on             # Write PROXY header

modify group permissions

Code:

 usermod -a -G varnish hitch
 service restart hitch

Edit varnish startup
We need to tell Varnish to listen on a socket (Hitch SSL will use it to talk to Varnish). That’s all where it will listen in our setup, being the caching layer of our app. However, we also want to keep it on a private HTTP port (6081) – this is going to be used only by external cache purging apps which can be easily configured with TCP details.

Code:

systemctl edit varnish

Paste this:

Code:

[Service]
ExecStart=
ExecStart=/usr/sbin/varnishd -f /etc/varnish/default.vcl -s malloc,256m \
    -a /var/run/varnish/varnish.sock,PROXY,user=varnish,group=hitch,mode=660 \
    -a 127.0.0.1:6081 \
    -p feature=+http2

Take care of socket permissions and files for startup and running

Code:

#Since we’re going to store socket files in a dedicated directory, /var/run/varnish, we will “tell” SELinux what we’re going to use the directory for:
semanage fcontext -a -t varnishd_var_run_t "/var/run/varnish(/.*)?"

#And we also make sure that our sockets directory is created after reboot:
cat << _EOF_ >> /etc/tmpfiles.d/varnish.conf
  d /run/varnish 755 varnish varnish
_EOF_

#And we also create it right away via:
systemd-tmpfiles --create varnish.conf

#Make sure to setup correct SELinux label to our directory:
semanage fcontext -a -t httpd_var_run_t "/var/run/nginx(/.*)?"

#And ensure that the directory is created at boot time:
cat << _EOF_ >> /etc/tmpfiles.d/nginx.conf
  d /run/nginx 750 nginx varnish
_EOF_

#And we also create it right away via:
systemd-tmpfiles --create nginx.conf

## NGINX CONFIGS ##
Nginx configuration for our hitch ssl, varnish setup
Add this to line 59 of /usr/local/nginx/conf/nginx.conf

Code:

real_ip_header X-Forwarded-For;
set_real_ip_from unix:;
real_ip_recursive on;

rename /usr/local/nginx/conf.d/site.com.ssl.conf to /usr/local/nginx/conf.d/site.com.ssl.conf-disabled
This is because nginx will only be serving http requests because that is all varnish understands. Hitch does ALL the SSL work.

I didn't modify a ton but for easier instructions here's the full config file for ssl
vi /usr/local/nginx/conf.d/site.com.conf

Code:

# Centmin Mod Getting Started Guide
# must read http://centminmod.com/getstarted.html

# redirect from non-www to www
# uncomment, save file and restart Nginx to enable
# if unsure use return 302 before using return 301
#server {
#            listen   80;
#            server_name site.com;
#            return 301 $scheme://www.site.com$request_uri;
#       }
server {
   listen   80;
   #listen [::]:80 default_server;
   server_name site.com www.site.com;
   return 302 https://$server_name$request_uri;
}
server {
    listen unix:/var/run/nginx.sock;
    server_name www.site.com;
    return 301 https://$server_name$request_uri;
}

server {
  listen unix:/var/run/nginx.sock;
  server_name site.com www.site.com;

# ngx_pagespeed & ngx_pagespeed handler
#include /usr/local/nginx/conf/pagespeed.conf;
#include /usr/local/nginx/conf/pagespeedhandler.conf;
#include /usr/local/nginx/conf/pagespeedstatslog.conf;

  #add_header X-Frame-Options SAMEORIGIN;
  add_header X-Xss-Protection "1; mode=block" always;
  add_header X-Content-Type-Options "nosniff" always;
  #add_header Referrer-Policy "strict-origin-when-cross-origin";
  #add_header Feature-Policy "accelerometer 'none'; camera 'none'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; payment 'none'; usb 'none'";

  # limit_conn limit_per_ip 16;
  # ssi  on;

  access_log /home/nginx/domains/site.com/log/access.log combined buffer=256k flush=5m;
  error_log /home/nginx/domains/site.com/log/error.log;

  include /usr/local/nginx/conf/autoprotect/site.com/autoprotect-site.com.conf;
  root /home/nginx/domains/site.com/public;
  # uncomment cloudflare.conf include if using cloudflare for
  # server and/or vhost site
  #include /usr/local/nginx/conf/cloudflare.conf;
  include /usr/local/nginx/conf/503include-main.conf;

  #include /usr/local/nginx/conf/wpincludes/site.com/wpcacheenabler_site.com.conf;
  include /usr/local/nginx/conf/wpincludes/site.com/wpsupercache_site.com.conf;
  # https://community.centminmod.com/posts/18828/
  #include /usr/local/nginx/conf/wpincludes/site.com/rediscache_site.com.conf;

  location / {
  include /usr/local/nginx/conf/503include-only.conf;
 

  # Enables directory listings when index file not found
  #autoindex  on;

  # for wordpress super cache plugin
  try_files /wp-content/cache/supercache/$http_host/$cache_uri/index.html $uri $uri/ /index.php?q=$uri&$args;

  # for wp cache enabler plugin
  #try_files $cache_enabler_uri $uri $uri/ $custom_subdir/index.php?$args;

  # Wordpress Permalinks
  #try_files $uri $uri/ /index.php?q=$uri&$args;

  # Nginx level redis Wordpress
  # https://community.centminmod.com/posts/18828/
  #try_files $uri $uri/ /index.php?$args;

  }

location ~* /(wp-login\.php) {
    limit_req zone=xwplogin burst=1 nodelay;
    #limit_conn xwpconlimit 30;
    #auth_basic "Private";
    #auth_basic_user_file /home/nginx/domains/site.com/htpasswd_wplogin;
    include /usr/local/nginx/conf/php-wpsc.conf;
    # https://community.centminmod.com/posts/18828/
    #include /usr/local/nginx/conf/php-rediscache.conf;
}

location ~* /(xmlrpc\.php) {
    limit_req zone=xwprpc burst=45 nodelay;
    #limit_conn xwpconlimit 30;
    include /usr/local/nginx/conf/php-wpsc.conf;
    # https://community.centminmod.com/posts/18828/
    #include /usr/local/nginx/conf/php-rediscache.conf;
}

location ~* /wp-admin/(load-scripts\.php) {
    limit_req zone=xwprpc burst=5 nodelay;
    #limit_conn xwpconlimit 30;
    include /usr/local/nginx/conf/php-wpsc.conf;
    # https://community.centminmod.com/posts/18828/
    #include /usr/local/nginx/conf/php-rediscache.conf;
}

location ~* /wp-admin/(load-styles\.php) {
    limit_req zone=xwprpc burst=5 nodelay;
    #limit_conn xwpconlimit 30;
    include /usr/local/nginx/conf/php-wpsc.conf;
    # https://community.centminmod.com/posts/18828/
    #include /usr/local/nginx/conf/php-rediscache.conf;
}
location = /favicon.ico {
  log_not_found off;
}

  include /usr/local/nginx/conf/wpincludes/site.com/wpsecure_site.com.conf;
  include /usr/local/nginx/conf/php-wpsc.conf;
 
  # https://community.centminmod.com/posts/18828/
  #include /usr/local/nginx/conf/php-rediscache.conf;
  include /usr/local/nginx/conf/pre-staticfiles-local-site.com.conf;
  include /usr/local/nginx/conf/pre-staticfiles-global.conf;
  include /usr/local/nginx/conf/staticfiles.conf;
  include /usr/local/nginx/conf/drop.conf;
  #include /usr/local/nginx/conf/errorpage.conf;
  include /usr/local/nginx/conf/vts_server.conf;
}

create this file to restart your services, service nginx restart doesn't work becuase it doesn't remove the socket file for some reason
I haven't found time for a full solution to that yet but stopping the service, removing the socket file works
and this is a convenient way to restart your stack anyway
on my todo list before I get to production level though

create file ~/restartservices.sh

Code:

service nginx stop
rm /var/run/nginx.sock
service nginx start
service php-fpm restart
service varnish restart
service hitch restart

Ok! Server setup is done now for the wordpress side.
## WORDPRESS ##
We need a few lines in your wp-config.php file since the requests are http and we're serving https wordpress needs to be forced or you'll get infinite loopbacks as wordpress tries to redirect to http but nginx redirects to https.
We also set some headers so it's cloudflare friendly

Code:

//define( 'WP_DEBUG', false );
//Cloudflare Flexible SSL redirect loop fix
if ( isset( $_SERVER['HTTP_X_FORWARDED_PROTO'] ) && $_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https') $_SERVER['HTTPS']='on';

define( 'JETPACK_SIGNATURE__HTTPS_PORT', 0 );

Next disable any other caching modules.

Then install these by going to /home/nginx/site.com/public and running it
This plugin has been good to me so far no issues purgin after I setup the plugin properly, image here:

Code:

wp plugin install vcaching
wp plugin activate vcaching

Plugin settings screenshot HERE:
File sharing and storage made simple

You can copy the logged in cookie string from /etc/varnish/default.vcl

I also suggest making your own logged in cookie string and replacing it in both locations for security.

The plugin uses sessions so it can introduce session locking to wordpress which doesn't have it by default but since so many modules use sessions anyway it's not so much a downside as something to acknowledge.
It's a feature rich module the best I could find and it's free!

Test it out by hitting purge ALL varnish cache at the top if it works then it's doing purge requests right.

type ifconfig and get your interface address so we can run a test that bypasses cloudflare and we get accurate headers

Look for something like this in the output

Code:

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 111.111.111.111

111.111.111.111 will be different for you, use that IP

on your work machine, not the server, we are going to bypass cloudflare to get accurate headers
you'll have to google how to change the hosts file for your system, in windows it is
C:\Windows\System32\drivers\etc\hosts and add this line

Code:

111.111.111.111 site.com

now you will go directly to the server, you can comment it out later to test with cloudflare


Load up a browser that isn't loaded up with any logins to the site or clear all cookies for the site you are testing.

Use the network tab and look for response headers

You'll get a reply like this after reloading the page a few times if everything is working:

Code:

accept-ranges: bytes
age: 13
content-encoding: br
content-length: 13934
content-type: text/html; charset=UTF-8
date: Thu, 02 Jul 2020 02:15:36 GMT
link: <https://site.com.com/wp-json/>; rel="https://api.w.org/"
link: <https://site.com/>; rel=shortlink
server: nginx centminmod
status: 200
vary: Accept-Encoding
via: 1.1 varnish (Varnish/6.3)
x-content-type-options: nosniff
x-powered-by: centminmod
x-varnish: 65751 786439
x-vc-cache: HIT
x-vc-cacheable: YES:Is cacheable, ttl: 1209600.000
x-vc-debug: true
x-vc-enabled: true
x-vc-hash: /?cc=debug#site.com
x-vc-ttl: 1209600
x-xss-protection: 1; mode=block

If it's not working then x-vc-cacheable might hint why.


Notable headers to watch for:

Code:

via: 1.1 varnish (Varnish/6.3)
x-varnish: 164036
age: 0
X-VC-Cache: MISS

Age 0 and X-VC-Cache: MISS means it wasn't cached if it's >0 and HIT it's cached.

some other tools:

Code:

varnishlog -c

or you can run

Code:

varnishtop

Final steps when it's all working and before you go production:
in /etc/varnish/lib/static.vcl
comment out this line 14:

Code:

       #set req.http.X-VC-Debug = "true";

This will prevent everyone and their dog from knowing what your cache server is doing, you can still get it by adding the word debug somewhere in the url.

Last is set you SSL/TLS to FULL on cloudflare. Since we have a letsencrypt cert you might even be able to use FUll (strict) but I haven't tested it yet.

Annnd that's it, enjoy


## TIPS AND RESULTS ##

You can use the wordpress vcaching plugin stats page to make sure it's actually caching pages the Cache Hits counter should keep going up.

My initial testing was pretty positive for the entire stack the best I was able to get was about 85ms TTFB on a page for a served php file that says hello world. For wordpress it'll increase but a fully cached page shouldn't take a ton longer I have a 200ms TTFB for the home page on it now and I currently have a lot of things turned off like opcode caching etc so I can develop quickly.

You'll probably want to fiddle with it through cloudflare too, it will add time to your TTFB scores but there's a few workarounds like setting a page rule to do site.com/* cache everything, though this could have other effects so use at your discretion. I use cloudflare for security mostly, never had a hacked site since I started using it and it saves you on bandwidth and speeds up your site in other ways.

## THANKS ##

Hope this helps anyone wanting to build woocommerce sites with a more robust caching engine.

If you're building a big site too I can also suggest using the Scalability Pro plugin to make it work with up to a million products, even more. It's what made us able to consider woo.

 

1) I actually explain that in the guide. It should end up a faster stack because hitch is light weight and tested to be faster for SSL handling and we're using sockets and nginx doesn't support all the stuff we have to use to do sockets properly.

2) Fair enough I just don't have a server I'm comfortable to put file downloads on atm since they don't belong to me but my clients.

If you intended on having a varnish server separate from your nginx server or use separate servers somehow else in the stack. Since we're using unix sockets it has to all be done on a linux file level. Meaning it's designed to have the entire stack run on one server. You can add as many domains to that server as you want though

 

Good luck hopefully I didn't miss any commands if you find something weird let me know. I didn't compare it directly with any other setups so far. I don't have time to setup multiple environments and run tests unfortunately

What I knew going in is that sockets should have more throughput than TCP stack and that hitch was tested to be faster than nginx for ssl requests and supports more technology with it and that I was settled on using varnish because I've used it before and been very happy with it.

 

I didn't have it on mine I think that's optional with centminmod, I tried it before. You can either try to get the one you have working or try installing the way I did as I remember it ran the steps I did for me. Not sure if there's any diff versions. For the dependency issue try just doing a yum search for the package it claims to need and then installing it.

 

Cool thanks eva2000, the ability to do all this has only been around fully since about 2018 I think so it's pretty new. Integrating with the centminmod addon is definitely more ideal I'll update the guide when I get chances to test it.

I also noticed I need to build hitch against openssl 1.1.1 or newer to be able to use tls 1.3 which I really want to do it's a major upgrade. I will post that too when I get time to do it and put in the guide.

The reason I was never too fond of wp super cache was simply I couldn't get any nginx rules setup that would reliably let it fully store html files of a page, it used the other caching method which is slower as it actually runs PHP, and it seemed the amount of caching it did was less, to fix this there's settings and rules for apache but not nginx. WP Super Cache still serviced my needs for some sites though and it's already setup. Wp Fastest Cache would work with some custom nginx rules I found and modified and cache a result as html but it didn't work well for all WP sites. Varnish serves fully cached pages and also lets me write VCL rules so it's just what I need for a big woo site. I still think the work you do is great tho just explaining my thought process. You've definitely made my life much easier.

 

I didn't have those errors so I can't really say where it went wrong for you. I didn't use the centminmod addon i just used yum to install it. I don't have time to do more work on it atm to integrate it cuz I'm doubled up on projects but I'd suggest either trying what eva suggested or uninstalling the addon and trying what I did. Doing both ways sounds like it won't work.