Want to subscribe to topics you're interested in?
Become a Member

Install Have it working, if i have install Varnish Cache in my VPS...

Discussion in 'Install & Upgrades or Pre-Install Questions' started by computer19852007, Jun 23, 2018.

  1. computer19852007

    computer19852007 Member

    82
    8
    8
    Jun 25, 2014
    Ratings:
    +8
    Local Time:
    9:15 AM
    Centmin Mod 1.2.3-eva2000.06
    MariaDB 5.5.34
    I have install Varnish Cache in my VPS, and i use cloudflare.com, Have it working?

    Thanks you
     
  2. eva2000

    eva2000 Administrator Staff Member

    36,864
    8,069
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,427
    Local Time:
    12:15 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    ??? installing varnish cache by itself won't do anything unless you configure it to listen in front of nginx - meaning take over non-HTTPS port 80 and change nginx to listen to a different port for non-HTTPS i.e. port 81 but Varnish doesn't support HTTP/2 HTTPS natively so you need to use nginx to terminate HTTP/2 HTTPS connections in front of Varnish cache for HTTPS port 443

    So end up with configuration like this where Varnish Cache is sandwiched between Nginx HTTP/2 HTTPS and non-HTTPS Nginx backend origin i.e. on port 81. I assume you want HTTP/2 HTTPS default site so it would end up looking like the below.

    Visitor > Nginx HTTP/2 HTTPS Proxy Port 443 > Varnish Cache HTTP/2 Enabled port 6081 > Nginx non-HTTPS backend port 81

    Nginx realip module setup outlined at here will take care of Cloudflare translation of visitor ips via Cloudflare reverse proxy and pass to Nginx HTTP/2 HTTPS port 443 termination.

    Centmin Mod is provide as is, so short of scripted related bugs or issues, any further optimisation to the web stack components - nginx, php-fpm, mariadb mysql, csf firewall etc or web app specific configurations are left to the Centmin Mod user to deal with. So I do not provide any free support for such. So actual configuration of Varnish and Nginx for such is left up to you do to.

    I did cover it briefly for Magento 2 + Varnish Cache on Centmin Mod Nginx here centminmod/centminmod-magento2 under Magento 2 Varnish Cache Config & Benchmarks section but that is specific and basic setup for Magento 2. Varnish Cache needs specific VCL rules configured for each web application/script you have on your server so you need to understand how to do that for each web app meaning you need to understand each web app's cookie usage and authentication/login vs guest visitor structure so you can configure via web app specific VCL rules what to cache and what not to cache for each web app. Otherwise, Varnish Cache is useless or may not cache anything or may cache sensitive private logged in user data and serve that private info to all guest visitors.

    However, Centmin Mod users are free to help each other out and ask questions or give answers on this community forum. My hopes are that this community forum evolves so that more veteran long time Centmin Mod users help new Centmin Mod users out :)
     
  3. computer19852007

    computer19852007 Member

    82
    8
    8
    Jun 25, 2014
    Ratings:
    +8
    Local Time:
    9:15 AM
    Centmin Mod 1.2.3-eva2000.06
    MariaDB 5.5.34
  4. computer19852007

    computer19852007 Member

    82
    8
    8
    Jun 25, 2014
    Ratings:
    +8
    Local Time:
    9:15 AM
    Centmin Mod 1.2.3-eva2000.06
    MariaDB 5.5.34
    My config File
    1. default.vcl
    Code:
    # VCL version 5.0 is not supported so it should be 4.0 even though actually used Varnish version is 5
    vcl 4.0;
    
    import std;
    # The minimal Varnish version is 5.0
    # For SSL offloading, pass the following header in your proxy server or load balancer: 'X-Forwarded-Proto: https'
    
    backend default {
        .host = "localhost";
        .port = "8686";
        .first_byte_timeout = 600s;
        .connect_timeout = 5s;
        .between_bytes_timeout = 3s;
        .probe = {
            .url = "/health_check.php";
            .timeout = 3s;
            .interval = 5s;
            .window = 10;
            .threshold = 5;
       }
    }
    
    acl purge {
        "localhost";
    }
    
    sub vcl_recv {
        if (req.method == "PURGE") {
            if (client.ip !~ purge) {
                return (synth(405, "Method not allowed"));
            }
            # To use the X-Pool header for purging varnish during automated deployments, make sure the X-Pool header
            # has been added to the response in your backend server config. This is used, for example, by the
            # capistrano-magento2 gem for purging old content from varnish during it's deploy routine.
            if (!req.http.X-Magento-Tags-Pattern && !req.http.X-Pool) {
                return (synth(400, "X-Magento-Tags-Pattern or X-Pool header required"));
            }
            if (req.http.X-Magento-Tags-Pattern) {
              ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
            }
            if (req.http.X-Pool) {
              ban("obj.http.X-Pool ~ " + req.http.X-Pool);
            }
            return (synth(200, "Purged"));
        }
    
        if (req.method != "GET" &&
            req.method != "HEAD" &&
            req.method != "PUT" &&
            req.method != "POST" &&
            req.method != "TRACE" &&
            req.method != "OPTIONS" &&
            req.method != "DELETE") {
              /* Non-RFC2616 or CONNECT which is weird. */
              return (pipe);
        }
    
        if (req.url ~ "/?nocache=1") {
            return (pass);
        }
    
        # We only deal with GET and HEAD by default
        if (req.method != "GET" && req.method != "HEAD") {
            return (pass);
        }
    
        # Bypass shopping cart, checkout and search requests
        if (req.url ~ "/checkout" || req.url ~ "/catalogsearch") {
            return (pass);
        }
    
        # Bypass health check requests
        if (req.url ~ "/health_check.php") {
            return (pass);
        }
    
        # Set initial grace period usage status
        set req.http.grace = "none";
    
        # normalize url in case of leading HTTP scheme and domain
        set req.url = regsub(req.url, "^http[s]?://", "");
    
        # collect all cookies
        std.collect(req.http.Cookie);
    
        # Compression filter. See https://www.varnish-cache.org/trac/wiki/FAQ/Compression
        if (req.http.Accept-Encoding) {
            if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|flv)$") {
                # No point in compressing these
                unset req.http.Accept-Encoding;
            } elsif (req.http.Accept-Encoding ~ "gzip") {
                set req.http.Accept-Encoding = "gzip";
            } elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
                set req.http.Accept-Encoding = "deflate";
            } else {
                # unkown algorithm
                unset req.http.Accept-Encoding;
            }
        }
    
        # brotli
        if(req.http.Accept-Encoding ~ "br" && req.url !~ "\.(jpg|png|gif|gz|mp3|mov|avi|mpg|mp4|swf|wmf)$") {
            set req.http.X-brotli = "true";
        }
    
        # Remove Google gclid parameters to minimize the cache objects
        set req.url = regsuball(req.url,"\?gclid=[^&]+$",""); # strips when QS = "?gclid=AAA"
        set req.url = regsuball(req.url,"\?gclid=[^&]+&","?"); # strips when QS = "?gclid=AAA&foo=bar"
        set req.url = regsuball(req.url,"&gclid=[^&]+",""); # strips when QS = "?foo=bar&gclid=AAA" or QS = "?foo=bar&gclid=AAA&bar=baz"
    
        # Remove DoubleClick offensive cookies
        set req.http.Cookie = regsuball(req.http.Cookie, "__gads=[^;]+(; )?", "");
        # Remove the AddThis cookies
        set req.http.Cookie = regsuball(req.http.Cookie, "__atuv.=[^;]+(; )?", "");
    
        # Large static files are delivered directly to the end-user without
        # waiting for Varnish to fully read the file first.
        # Varnish 4 fully supports Streaming, so set do_stream in vcl_backend_response()
        # https://www.varnish-software.com/wiki/content/tutorials/varnish/sample_vclTemplate.html#turn-on-varnish-support-for-streaming
        if (req.url ~ "^[^?]*\.(7z|avi|bz2|flac|flv|gz|mka|mkv|mov|mp3|mp4|mpeg|mpg|ogg|ogm|opus|rar|tar|tgz|tbz|txz|wav|webm|xz|zip)(\?.*)?$") {
          unset req.http.Cookie;
          return (hash);
        }
    
        # Static files caching
        if (req.url ~ "^/(pub/)?(media|static)/") {
            # Static files should not be cached by default
            return (pass);
    
            # But if you use a few locales and don't use CDN you can enable caching static files by commenting previous line (#return (pass);) and uncommenting next 3 lines
            #unset req.http.Https;
            #unset req.http.X-Forwarded-Proto;
            #unset req.http.Cookie;
        }
    
        return (hash);
    }
    
    sub vcl_backend_fetch
    {
        if(bereq.http.X-brotli == "true") {
            set bereq.http.Accept-Encoding = "br";
            unset bereq.http.X-brotli;
        }
    }
    
    sub vcl_hash {
        if (req.http.cookie ~ "X-Magento-Vary=") {
            hash_data(regsub(req.http.cookie, "^.*?X-Magento-Vary=([^;]+);*.*$", "\1"));
        }
    
        # brotli
        if(req.http.X-brotli == "true") {
            hash_data("brotli");
        }
    
        # For multi site configurations to not cache each other's content
        if (req.http.host) {
            hash_data(req.http.host);
        } else {
            hash_data(server.ip);
        }
    
        # To make sure http users don't see ssl warning
        if (req.http.X-Forwarded-Proto) {
            hash_data(req.http.X-Forwarded-Proto);
        }
        
    }
    
    sub vcl_backend_response {
    
        set beresp.grace = 3d;
    
        if (beresp.http.content-type ~ "text") {
            set beresp.do_esi = true;
        }
    
        if (bereq.url ~ "\.js$" || beresp.http.content-type ~ "text") {
            set beresp.do_gzip = true;
        }
    
        if (bereq.url ~ "^[^?]*\.(7z|avi|bmp|bz2|css|csv|doc|docx|eot|flac|flv|gif|gz|ico|jpeg|jpg|js|less|mka|mkv|mov|mp3|mp4|mpeg|mpg|odt|otf|ogg|ogm|opus|pdf|png|ppt|pptx|rar|rtf|svg|svgz|swf|tar|tbz|tgz|ttf|txt|txz|wav|webm|webp|woff|woff2|xls|xlsx|xml|xz|zip)(\?.*)?$") {
          unset beresp.http.set-cookie;
        }
    
        if (beresp.http.X-Magento-Debug) {
            set beresp.http.X-Magento-Cache-Control = beresp.http.Cache-Control;
        }
    
        # cache only successfully responses and 404s
        if (beresp.status != 200 && beresp.status != 404) {
            set beresp.ttl = 0s;
            set beresp.uncacheable = true;
            return (deliver);
        } elsif (beresp.http.Cache-Control ~ "private") {
            set beresp.uncacheable = true;
            set beresp.ttl = 86400s;
            return (deliver);
        }
    
        # validate if we need to cache it and prevent from setting cookie
        # images, css and js are cacheable by default so we have to remove cookie also
        if (beresp.ttl > 0s && (bereq.method == "GET" || bereq.method == "HEAD")) {
            unset beresp.http.set-cookie;
        }
    
       # If page is not cacheable then bypass varnish for 2 minutes as Hit-For-Pass
       if (beresp.ttl <= 0s ||
           beresp.http.Surrogate-control ~ "no-store" ||
           (!beresp.http.Surrogate-Control &&
           beresp.http.Cache-Control ~ "no-cache|no-store") ||
           beresp.http.Vary == "*") {
            # Mark as Hit-For-Pass for the next 2 minutes
            set beresp.ttl = 120s;
            set beresp.uncacheable = true;
        }
    
        return (deliver);
    }
    
    sub vcl_deliver {
        if (resp.http.X-Magento-Debug) {
            if (resp.http.x-varnish ~ " ") {
                set resp.http.X-Magento-Cache-Debug = "HIT";
                set resp.http.Grace = req.http.grace;
            } else {
                set resp.http.X-Magento-Cache-Debug = "MISS";
            }
        } else {
            unset resp.http.Age;
        }
    
        # Not letting browser to cache non-static files.
        if (resp.http.Cache-Control !~ "private" && req.url !~ "^/(pub/)?(media|static)/") {
            set resp.http.Pragma = "no-cache";
            set resp.http.Expires = "-1";
            set resp.http.Cache-Control = "no-store, no-cache, must-revalidate, max-age=0";
        }
    
        unset resp.http.X-Magento-Debug;
        unset resp.http.X-Magento-Tags;
        unset resp.http.X-Powered-By;
        unset resp.http.Server;
        unset resp.http.X-Varnish;
        unset resp.http.Via;
        # https://github.com/magento/magento2/issues/8126
        #unset resp.http.Link;
    }
    
    sub vcl_hit {
        if (obj.ttl >= 0s) {
            # Hit within TTL period
            return (deliver);
        }
        if (std.healthy(req.backend_hint)) {
            if (obj.ttl + 300s > 0s) {
                # Hit after TTL expiration, but within grace period
                set req.http.grace = "normal (healthy server)";
                return (deliver);
            } else {
                # Hit after TTL and grace expiration
                return (miss);
            }
        } else {
            # server is not healthy, retrieve from cache
            set req.http.grace = "unlimited (unhealthy server)";
            return (deliver);
        }
    }
    
    2. nginx.conf

    Code:
    user              nginx nginx;
    worker_processes 1;
    worker_priority -10;
    
    worker_rlimit_nofile 520000;
    timer_resolution 100ms;
    
    pcre_jit on;
    include /usr/local/nginx/conf/dynamic-modules.conf;
    
    
    pid         logs/nginx.pid;
    
    events {
        worker_connections  50000;
        accept_mutex off;
        accept_mutex_delay 200ms;
        use epoll;
        #multi_accept on;
    }
    
    http {
    map $scheme $mag_hstsheader { https  "max-age=31556926; includeSubDomains"; }
     # map_hash_bucket_size 128;
     # map_hash_max_size 4096;
     server_names_hash_bucket_size 128;
     server_names_hash_max_size 2048;
     variables_hash_max_size 2048;
    
    limit_req_zone $binary_remote_addr zone=xwplogin:16m rate=40r/m;
    #limit_conn_zone $binary_remote_addr zone=xwpconlimit:16m;
    
    # sets Centmin Mod headers via headers more nginx module
    # https://github.com/openresty/headers-more-nginx-module
    # don't remove the first 2 lines as centmin mod checks to see if they're
    # missing and re-adds them anyway. Just uncomment the 3rd & 4th lines
    # which is used to override the Server header to what you want = nginx
    # and remove the X-Powered-By header + restart nginx service
    # do not disable headers more nginx module itself as it's required for
    # other centmin mod features like redis nginx level caching & letsencrypt
    # integration in vhosts created by addons/acmetool.sh
    more_set_headers "Server: centminmod";
    more_set_headers "X-Powered-By: centminmod";
    #more_set_headers "Server: nginx";
    #more_clear_headers "X-Powered-By";
    
    # uncomment cloudflare.conf include if using cloudflare for
    # server and/or vhost site + setup cron job for command
    # /usr/local/src/centminmod/tools/csfcf.sh auto
    # run the auto command once to populate cloudflare ips
    #include /usr/local/nginx/conf/cloudflare.conf;
    # uncomment incapsula.conf include if using incapsula for
    # server and/or vhost site + setup cron job for command
    # /usr/local/src/centminmod/tools/csfincapsula.sh auto
    # run the auto command once to popular incapsula ips
    #include /usr/local/nginx/conf/incapsula.conf;
    include /usr/local/nginx/conf/maintenance.conf;
    #include /usr/local/nginx/conf/vts_http.conf;
    include /usr/local/nginx/conf/geoip.conf;
    include /usr/local/nginx/conf/webp.conf;
    #include /usr/local/nginx/conf/pagespeedadmin.conf;
    include /usr/local/nginx/conf/fastcgi_param_https_map.conf;
    
    log_format  main  '$remote_addr - $remote_user [$time_local] $request '
                    '"$status" $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for" "$gzip_ratio"'
                    ' "$connection" "$connection_requests" "$request_time"';
    
    log_format  ddos-proxy '$remote_addr for $http_x_real_ip - $remote_user [$time_local] $request '
                    '"$status" $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for" "$gzip_ratio"'
                    ' "$connection" "$connection_requests" "$request_time"';
    
    log_format  main_ext '$remote_addr - $remote_user [$time_local] "$request" '
                             '$status $body_bytes_sent "$http_referer" '
                             '"$http_user_agent" "$http_x_forwarded_for" '
                             'rt=$request_time ua="$upstream_addr" '
                             'us="$upstream_status" ut="$upstream_response_time" '
                             'ul="$upstream_response_length" '
                             'cs=$upstream_cache_status' ;
    
    access_log  off;
    error_log   logs/error.log warn;
    
        index  index.php index.html index.htm;
        include       mime.types;
        default_type  application/octet-stream;
        charset utf-8;
    
            sendfile on;
            sendfile_max_chunk 512k;
            tcp_nopush  on;
            tcp_nodelay on;
            server_tokens off;
            server_name_in_redirect off;
            
            keepalive_timeout  5;
            keepalive_requests 500;
            lingering_time 20s;
            lingering_timeout 5s;
            keepalive_disable msie6;
    
        gzip on;
        gzip_vary   on;
        gzip_disable "MSIE [1-6]\.";
            gzip_static on;
            gzip_min_length   1400;
            gzip_buffers      1024 8k;
            gzip_http_version 1.0;
            gzip_comp_level 5;
            gzip_proxied    any;
            gzip_types text/plain text/css text/xml application/javascript application/x-javascript application/xml application/xml+rss application/ecmascript application/json image/svg+xml;
    
     client_body_buffer_size 256k;
     client_body_in_file_only off;
     client_body_timeout 60s;
     client_header_buffer_size 64k;
    ## how long a connection has to complete sending
    ## it's headers for request to be processed
     client_header_timeout  10s;
     client_max_body_size 1024m;
     connection_pool_size  512;
     directio  4m;
     directio_alignment 512;
     ignore_invalid_headers on;       
     large_client_header_buffers 8 64k;
     output_buffers   1 512k;
     postpone_output  1460;
     proxy_temp_path  /tmp/nginx_proxy/;
     request_pool_size  32k;
     reset_timedout_connection on;
     send_timeout     60s;
     types_hash_max_size 2048;
    
    # for nginx proxy backends to prevent redirects to backend port
    # port_in_redirect off;
    
    open_file_cache max=50000 inactive=60s;
    open_file_cache_valid 120s;
    open_file_cache_min_uses 2;
    open_file_cache_errors off;
    open_log_file_cache max=10000 inactive=30s min_uses=2;
    
    ## limit number of concurrency connections per ip to 16
    ## add to your server {} section the next line
    ## limit_conn limit_per_ip 16;
    ## uncomment below line allows 500K sessions
    # limit_conn_log_level error;
    #######################################
    # use limit_zone for Nginx <v1.1.7 and lower
    # limit_zone $binary_remote_addr zone=limit_per_ip:16m;
    #######################################
    # use limit_conn_zone for Nginx >v1.1.8 and higher
    # limit_conn_zone $binary_remote_addr zone=limit_per_ip:16m;
    #######################################
     include /usr/local/nginx/conf/proxycache_map.conf;
     include /usr/local/nginx/conf/proxycache-includes.conf;
     include /usr/local/nginx/conf/conf.d/*.conf;
    }
    
     
  5. computer19852007

    computer19852007 Member

    82
    8
    8
    Jun 25, 2014
    Ratings:
    +8
    Local Time:
    9:15 AM
    Centmin Mod 1.2.3-eva2000.06
    MariaDB 5.5.34
    And i unknow when use:
    or
    my site use https?
     
  6. eva2000

    eva2000 Administrator Staff Member

    36,864
    8,069
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,427
    Local Time:
    12:15 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    you need to create a separate non-https vhost listening on port 8686 with same nginx rules as your https vhost config just without https directives (so it's non-https) so varnish cache can read from that non-https backend origin on port 8686
     
..