Discover Centmin Mod today
Register Now

Featured Nginx How to use Brotli compression for Centmin Mod Nginx web servers

Discussion in 'Centmin Mod User Tutorials & Guides' started by eva2000, Mar 6, 2017.

  1. eva2000

    eva2000 Administrator Staff Member

    29,707
    6,708
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,012
    Local Time:
    1:21 PM
    Nginx 1.13.x
    MariaDB 5.5
    Brotli compression for web servers is an alternative to Gzip compression for static files. You can read about Brotli compression in web server context over on Cloudflare's blog and the current state of web browser adoption at http://caniuse.com/#search=brotli. Google Play Store on a daily basis saves 1.5 petabytes (1.5 million gigabytes) of bandwidth each day using Brotli !

    To properly utilise Brotli compression, your web server needs to be able to serve Brotli Content-Encoding headers to only web browsers that support it and your web site needs to serve the web assets, css and js etc over HTTPS with SSL certificate. May not work with sites behind Cloudflare WAF/Proxy.

    For Nginx there is an ngx_brotli module which can be compiled into Nginx to support such. Since October 2015, Centmin Mod latest 123.09beta01 branch's Nginx server has built in support for ngx_brotli module that can optionally be enabled or disabled via persistent config file /etc/centminmod/custom_config.inc set variables below:
    Code (Text):
    NGXDYNAMIC_BROTLI='y'
    NGINX_LIBBROTLI='y'
    

    Once variables set, you recompile Centmin Mod Nginx via centmin.sh menu option 4 and will have ngx_brotli support enabled. Centmin Mod Nginx dynamic modules are loaded from /usr/local/nginx/modules directory via an include file /usr/local/nginx/conf/dynamic-modules.conf in /usr/local/nginx/conf/nginx.conf.

    centmin.sh menu option 4
    Code (Text):
    --------------------------------------------------------
         Centmin Mod Menu 123.09beta01 centminmod.com
    --------------------------------------------------------
    1).  Centmin Install
    2).  Add Nginx vhost domain
    3).  NSD setup domain name DNS
    4).  Nginx Upgrade / Downgrade
    5).  PHP Upgrade / Downgrade
    6).  XCache Re-install
    7).  APC Cache Re-install
    8).  XCache Install
    9).  APC Cache Install
    10). Memcached Server Re-install
    11). MariaDB MySQL Upgrade & Management
    12). Zend OpCache Install/Re-install
    13). Install/Reinstall Redis PHP Extension
    14). SELinux disable
    15). Install/Reinstall ImagicK PHP Extension
    16). Change SSHD Port Number
    17). Multi-thread compression: pigz,pbzip2,lbzip2...
    18). Suhosin PHP Extension install
    19). Install FFMPEG and FFMPEG PHP Extension
    20). NSD Install/Re-Install
    21). Update - Nginx + PHP-FPM + Siege
    22). Add Wordpress Nginx vhost + Cache Plugin
    23). Update Centmin Mod Code Base
    24). Exit
    --------------------------------------------------------
    Enter option [ 1 - 24 ]
    

    You can check Centmin Mod Nginx has ngx_brotli enabled via command below to show Nginx's compiled and enabled modules
    Code (Text):
    nginx -V
    

    Look for the line --add-dynamic-module=../ngx_brotli
    In /usr/local/nginx/conf/nginx.conf contains include file to load Nginx dynamic modules
    Code (Text):
    include /usr/local/nginx/conf/dynamic-modules.conf;
    

    And also contains an include file with actual Nginx Brotli settings
    Code (Text):
    include /usr/local/nginx/conf/brotli_inc.conf;
    

    Contents of /usr/local/nginx/conf/dynamic-modules.conf will contain Nginx Brotli's dynamic and static modules
    Code (Text):
    load_module "modules/ngx_http_brotli_filter_module.so";
    load_module "modules/ngx_http_brotli_static_module.so";
    

    Contents of /usr/local/nginx/conf/brotli_inc.conf with ngx_brotli settings
    Code (Text):
    /usr/local/nginx/conf/brotli_inc.conf
    brotli on;
    brotli_static on;
    brotli_min_length 1000;
    brotli_buffers 32 8k;
    brotli_comp_level 5;
    brotli_types text/plain text/css text/xml application/javascript application/x-javascript application/xml application/xml+rss application/ecmascript application/json image/svg+xml;
    

    With Centmin Mod Nginx's Brotli module enabled, this will configure both Brotli on the fly compression as well as support Brotli static file serving if a *.br extension file is detected. To test, you can use curl command with appropate Accept-Encoding directives for gzip and br.

    Curl content encoding gzip,br check for Centmin Mod Nginx based server with ngx_brotli enabled. Check for Content-Encoding: br to confirm that Nginx is serving Brotli compressed version of the css or js file.
    Code (Text):
    curl -sI /dev/null -H"Accept-Encoding: gzip,br" https://domain.com/brotlitest2/bootstrap.min.css
    HTTP/1.1 200 OK
    Date: Sun, 05 Mar 2017 16:39:04 GMT
    Content-Type: text/css
    Content-Length: 16149
    Last-Modified: Mon, 25 Jul 2016 16:08:01 GMT
    Connection: keep-alive
    Vary: Accept-Encoding
    ETag: "57963961-3f15"
    Content-Encoding: br
    Server: nginx centminmod
    X-Powered-By: centminmod
    Expires: Tue, 04 Apr 2017 16:39:04 GMT
    Cache-Control: max-age=2592000
    Access-Control-Allow-Origin: *
    Cache-Control: public, must-revalidate, proxy-revalidate
    

    Curl content encoding gzip check for Centmin Mod Nginx based server with ngx_brotli enabled. Check for Content-Encoding: gzip to confirm that Nginx is serving Gzip compressed version of the css or js file.
    Code (Text):
    curl -sI /dev/null -H"Accept-Encoding: gzip" https://domain.com/brotlitest2/bootstrap.min.css
    HTTP/1.1 200 OK
    Date: Sun, 05 Mar 2017 16:40:19 GMT
    Content-Type: text/css
    Content-Length: 18322
    Last-Modified: Mon, 25 Jul 2016 16:08:01 GMT
    Connection: keep-alive
    Vary: Accept-Encoding
    ETag: "57963961-4792"
    Content-Encoding: gzip
    Server: nginx centminmod
    X-Powered-By: centminmod
    Expires: Tue, 04 Apr 2017 16:40:19 GMT
    Cache-Control: max-age=2592000
    Access-Control-Allow-Origin: *
    Cache-Control: public, must-revalidate, proxy-revalidate
    


    Other Notes


    • For Xenforo you may need to add to your library/config.php the setting
      Code (Text):
      $config['enableGzip'] = false;
     
    Last edited: Mar 23, 2017
    • Informative Informative x 4
  2. eva2000

    eva2000 Administrator Staff Member

    29,707
    6,708
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,012
    Local Time:
    1:21 PM
    Nginx 1.13.x
    MariaDB 5.5
    Also posted on Github a brotli.sh tool to generate static brotli (*.br) and gzip (*.gz) assets for css and js files you can play with at https://centminmod.com/brotlistatic

    • brotli.sh tool to auto compress css and js files by specifying path on CentOS based Centmin Mod LEMP web stack servers.
    • Appending clean flag on end of directory path will remove any *.br or *.gz compressed files. I usually place my tools at /root/tools but you can place brotli.sh whereever you want.
    • Appending display flag will find and display all *.br and *.gz compressed css and js files.
    Code (Text):
    Usage
    
    /root/tools/brotli.sh /path/to/parent/directory
    /root/tools/brotli.sh /path/to/parent/directory clean
    /root/tools/brotli.sh /path/to/parent/directory display
    


    gzip_static and brotli_static options in Nginx are enabled by default. So when Nginx sees a html, css, js file it wants to serve to visitors, it can either serve and compress on the fly at visitor request time the static file using gzip or brotli (if supported) or with gzip_static and brotli_static enabled look for *.gz or *.br precompressed versions of the static file and serve that instead. Precompressed static assets can be compressed using higher compression levels to further reduce the size of the static assets and improve page loading performance without the expensive cpu overhead of on the fly compression.
     
    Last edited: Mar 7, 2017
    • Informative Informative x 3
  3. eva2000

    eva2000 Administrator Staff Member

    29,707
    6,708
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,012
    Local Time:
    1:21 PM
    Nginx 1.13.x
    MariaDB 5.5
    You can also use your Chrome or web browser's developer tools network tab to see Content-Encoding of site's assets i.e. html file index for centminmod.com has Content-Encoding = br for brotli compression. Page is served over HTTP/2 based HTTPS (Protocol = h2).

    opera_devtools_br-01.png
     
    • Informative Informative x 2
  4. eva2000

    eva2000 Administrator Staff Member

    29,707
    6,708
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,012
    Local Time:
    1:21 PM
    Nginx 1.13.x
    MariaDB 5.5
    Don't have a HTTP/2 based HTTPS site yet for Brotli ? Centmin Mod Nginx has 2 guides for setting up free Letsencrypt SSL certificates and Centmin Mod Nginx HTTP/2 based HTTPS below:
     
    • Agree Agree x 2
    • Like Like x 1
  5. pamamolf

    pamamolf Well-Known Member

    2,666
    240
    63
    May 31, 2014
    Ratings:
    +424
    Local Time:
    6:21 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    So it can automaticaly check if user browser support brotli and serve a .br file and if not it will use a .gz file?

    It will auto fallback so no issues if the user uses an old browser ?

    Also how brotli handle css/js file changes ? It is autodetect them and recompress them?

    Also how it plays with Cloudflare enabled?
     
    Last edited: Mar 7, 2017
  6. SFLC

    SFLC Active Member

    224
    59
    28
    Dec 4, 2016
    The Canadas
    Ratings:
    +112
    Local Time:
    5:21 AM
    1
    10
    Works perfectly for me, I think you'd have to run with clean and then run it again on the path when you make changes as there's no way for it to detect changes automatically.

    Also, I'm using cloudflare and definitely no issues there, it's a lot faster and using this provides a noticeable improvement in my page load times.

    Can't get it to work with wordpress though but I think it's because I have it set to output HTTP/1.1 503 Service Temporarily Unavailable in the header as that site's not officially open to the public yet
     
    • Informative Informative x 1
  7. eva2000

    eva2000 Administrator Staff Member

    29,707
    6,708
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,012
    Local Time:
    1:21 PM
    Nginx 1.13.x
    MariaDB 5.5
    yes .br should only be served to browsers that support it. Though i have one issue related to Sucuri cloudproxy's caching of brotli files and serving to non supported browsers https://community.centminmod.com/threads/site-broken-in-internet-explorer-edge.9701/. So turned off brotli for forums when behind Sucuri cloudproxy.

    Yes for on the fly compression if no precompressed .br or .gz files are detected. Otherwise if precompressed .br or .gz files detected, they get served and can get out of date and need recompressing if you precompress files i.e. using brotli.sh from centminmod.com/brotlistatic

    Works for me on centminmod.com cloudflare mirror at centmin.sh though not sure why files are served gzip and not brotli there.
     
  8. eva2000

    eva2000 Administrator Staff Member

    29,707
    6,708
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,012
    Local Time:
    1:21 PM
    Nginx 1.13.x
    MariaDB 5.5
    If you use wordpress plugins that cache and/or pre-gzip compress files, then you'll already have pre-gzipped compressed versions that Centmin Mod Nginx serves via gzip_static directive. Not sure but IIRC pre-gzipped files take priority over on the fly brotli compression. But I believe if both pre-gzipped and pre-brotli compressed files exist, pre-brotli compressed files take priority if browser supports brotli.
    Code (Text):
    curl -sI /dev/null -H"Accept-Encoding: gzip,br" https://domain.com/brotlitest2/bootstrap.min.css
    HTTP/1.1 200 OK
    Date: Sun, 05 Mar 2017 16:39:04 GMT
    Content-Type: text/css
    Content-Length: 16149
    Last-Modified: Mon, 25 Jul 2016 16:08:01 GMT
    Connection: keep-alive
    Vary: Accept-Encoding
    ETag: "57963961-3f15"
    Content-Encoding: br
    Server: nginx centminmod
    X-Powered-By: centminmod
    Expires: Tue, 04 Apr 2017 16:39:04 GMT
    Cache-Control: max-age=2592000
    Access-Control-Allow-Origin: *
    Cache-Control: public, must-revalidate, proxy-revalidate
     
    • Like Like x 1
  9. BamaStangGuy

    BamaStangGuy Active Member

    470
    137
    43
    May 25, 2014
    Ratings:
    +180
    Local Time:
    10:21 PM
    I have this enabled on Christian Forums but not seeing any content being served via br. Do I need to flush CloudFlare cache or change anything else?
     
    • Like Like x 1
  10. RB1

    RB1 Active Member

    278
    72
    28
    Nov 11, 2016
    California
    Ratings:
    +119
    Local Time:
    8:21 PM
    Nginx 1.13.x
    MariaDB 10.1.x
    How do I force Brotli to start working :p

    I'm using a supported browser, Brotli is compiled in nginx, and all Brotli config files exist.
    Currently testing on a static website (not Wordpress) over HTTPS with all PageSpeed and Cloudflare cache cleared however it's still showing gzip headers and KeyCDN Brotli Test | Brotli compression check showing Brotli unsupported.

    Edit: Also restarted server.
     
  11. eva2000

    eva2000 Administrator Staff Member

    29,707
    6,708
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,012
    Local Time:
    1:21 PM
    Nginx 1.13.x
    MariaDB 5.5
    i see the same for my mirror under Cloudflare

    @RB1 and @BamaStangGuy so probably doesn't work with Cloudflare - might need to ask tech support at CF
     
    • Like Like x 2
  12. RB1

    RB1 Active Member

    278
    72
    28
    Nov 11, 2016
    California
    Ratings:
    +119
    Local Time:
    8:21 PM
    Nginx 1.13.x
    MariaDB 10.1.x
    Just opened a ticket with CF; I'll post here when they get back to me.
     
    • Like Like x 1
  13. BamaStangGuy

    BamaStangGuy Active Member

    470
    137
    43
    May 25, 2014
    Ratings:
    +180
    Local Time:
    10:21 PM
    I opened a ticket with them.
     
    • Like Like x 1
  14. RB1

    RB1 Active Member

    278
    72
    28
    Nov 11, 2016
    California
    Ratings:
    +119
    Local Time:
    8:21 PM
    Nginx 1.13.x
    MariaDB 10.1.x
    Cloudflare response:
    Not exactly the answer I was looking for because I'm still having the problem, but CF seems to have Brotli working... :D
     
  15. BamaStangGuy

    BamaStangGuy Active Member

    470
    137
    43
    May 25, 2014
    Ratings:
    +180
    Local Time:
    10:21 PM
  16. BamaStangGuy

    BamaStangGuy Active Member

    470
    137
    43
    May 25, 2014
    Ratings:
    +180
    Local Time:
    10:21 PM
    So since CloudFlare also can compress (and will if we don't do it), I wonder what the performance impact would be if we do not do any compression on our server and instead let CloudFlare cache our content and do the compression on their end?
     
  17. eva2000

    eva2000 Administrator Staff Member

    29,707
    6,708
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,012
    Local Time:
    1:21 PM
    Nginx 1.13.x
    MariaDB 5.5
    Guess for cloudflare you could turn off gzip and brotli on the backend then if you want
     
    • Like Like x 1
  18. BamaStangGuy

    BamaStangGuy Active Member

    470
    137
    43
    May 25, 2014
    Ratings:
    +180
    Local Time:
    10:21 PM
    All of our servers now disable gzip caching via nginx.conf. So the initial request from cloudflare will see be uncompressed but every request after that will see cloudflare compressed cache.
     
  19. RB1

    RB1 Active Member

    278
    72
    28
    Nov 11, 2016
    California
    Ratings:
    +119
    Local Time:
    8:21 PM
    Nginx 1.13.x
    MariaDB 10.1.x
    So is Brotli working with this configuration?
     
  20. BamaStangGuy

    BamaStangGuy Active Member

    470
    137
    43
    May 25, 2014
    Ratings:
    +180
    Local Time:
    10:21 PM
    No, I am not using any compression at all since CloudFlare will ignore the Brotli. Nginx serves uncompressed content to CloudFlare and then CloudFlare compresses and caches it. All future requests for the content are served by CloudFlare and never touch our server.