Featured Insight Guide How to boost Centmin Mod LEMP stack performance

Centmin Mod LEMP stack installations are already auto optimised and tuned out of the box based on detected server resources available (memory installed, disk space and speed, cpu model used) and based on the server environment detected (Xen, KVM, OpenVZ, Linux Kernel supported features). For example, if Linux kernel IPSET support is detected, Centmin Mod would enable CSF Firewall with IPSET support to be better handle larger numbers of IP address tracking/banning and Centmin Mod auto configures the CSF Firewall IP deny limits based on detected amount of memory installed on the server. However, that just means the software Centmin Mod installs and configures is optimised based on the server environment detected and not optimised for the web applications and scripts you intend to install on the server. Optimising for your specific web app/script usage requirements is left up to you to figure out yourself as Centmin Mod is provided as is.

However, below are some general configuration tips which will allow you to even further improve Centmin Mod LEMP stack server performance overall. Some of these tips can offer an extra 5-20% boost and generally aren't available on other control panels or LAMP/LEMP web stack configurations unless you know how to manually install/configure them. Whereas, most of these tips below have built in support on Centmin Mod LEMP stack and just need enabling as they are optional settings and not enabled by default. These configuration tips are made possible as Centmin Mod source compiles Nginx and PHP-FPM installs instead of using distro provided YUM/RPM packages. Meaning Nginx and PHP-FPM source compile times take longer to install Nginx and PHP-FPM, but the resulting binaries performs better than binaries provided by YUM/RPM packages.

For instance, the recent Kernel Page Table Isolation (KPTI) related security updates for Meltdown & Spectre vulnerabilities have proven to impact negatively Nginx and Apache performance. But some of the below configuration tips for Cloudflare Zlib Performance patch usage in Nginx Zlib integration and selecting the right GCC/Clang compiler + cpu pairing can in fact regain the lost performance and further improve Nginx performance. Another example is KPTI patched OpenSSL 1.1 ECDSA performance before/after Cloudflare patch.

Centmin Mod LEMP Stack Configuration Tips to boost performance

1. Use or switch to Centmin Mod 123.09beta01 124.00stable or 130.00beta01
2. Use PHP 7.x with Profile Guide Optimizations (PGO)
3. Optimising Nginx Compilation with the right GCC/Clang compiler
4. Use Brotli compression with HTTPS based web sites (page speed vs scalability)
5. Use Google ngx_pagespeed Nginx module
6. Leverage Nginx Thread Pooling
7. Enable other Cloudflare related patches
8. HTTP/2 HTTPS and Server Push
9. HTTPS TLS 1.3 Support
10. Leverage Linux 5.1+ Kernels and Nginx io_uring patch for better Async I/O

Use or switch to Centmin Mod 124.00stable or 130.00beta01

If you have yet to install Centmin Mod, you can follow the quick install guide here. Ideally, you want to be using CentOS 7.x 64bit. Minimum and recommended memory and disk requirements are outlined on official Install page.

Update: on May 8, 2022, Centmin Mod 124.00stable and 130.00beta01 were released. The below recommendations are generally still applicable.

Pretty much all the following suggestions below are only supported and/or fully tested in Centmin Mod 123.09beta01 or newer branch and not in stable Centmin Mod 123.08stable. So upgrading Centmin Mod Code to Latest Version is the very first step if you are not on Centmin Mod 123.09beta01 or higher. Getting Started Guide step 19 outlines also how to keep Centmin Mod code updated or how to switch version branches or you can run cmupdate command that was recently added.

Centmin Mod LEMP stack's script code is constantly updated for improvements, bug fixes and security fixes so keeping the Centmin Mod code up to date is important. With Centmin Mod 1.2.3-eva2000.08) (123.08stable) and higher releases, a newly added centmin.sh menu option 23 allows much easier code updates and version branch swicthing via Git backed environment you can setup. For full details read the following links:

• New centmin.sh menu option 23 updating
• Centmin Mod Upgrades
• Cronjob Auto Updating Centmin Mod Code Guide
• How to upgrade Centmin Mod + backing up customisations

Upgrading Centmin Mod involves 2 parts.

1. Upgrading the actual Centmin Mod code outlined at Upgrade Centmin Mod. This is heart of Centmin Mod where the code is the engine that runs centmin.sh shell based menu and all the automation you're accustomed to. You can easily update within a Centmin Mod version branch or switch version branches via centmin.sh menu option 23 outlined here.
2. Upgrade software that Centmin Mod installed or manages. For this part following outline at How to upgrade Centmin Mod software installed on your server.

So essentially, you can upgrade from one version branch to another i.e. 123.08stable to 123.09beta01 or higher in place, but not everything is upgraded as some things like server initial environment setup isn't changed i.e. how swap, tmp setup and allocation are created etc. The main parts from part 2 above are what in place upgrades do i.e. Nginx and PHP-FPM compilation and config/settings parameters and MariaDB version from 5.5 to 10.0.x. If you want the full environment changed including tmp and swap setup to .08's configuration, then you would need a fresh OS install and fresh .08 initial install. You can think of it like upgrading Windows 7 to Windows 8. An in place upgrade will upgrade code but won't change your computer environment from when you installed Windows 7 i.e. disk configuration and partition sizes won't change from when you initially installed Windows 7. Only way to change that would be fresh Windows 8 install.

Use PHP 7.x with Profile Guide Optimizations (PGO)

Supported branch: Centmin Mod 123.09beta01 and higher only and if more than 2 cpu cores detected only.

PHP 7 with Profile Guided Optimization (PGO) can boost PHP performance by up to +17% - particularly for Wordpress, MediaWiki and Drupal like PHP web applications and similar PHP execution patterns. This isn't enabled by default on PHP 7 installs as it can increase PHP compilation/install time by 40-100% depending on the server hardware and resources you have. You can enable PHP with Profile Guided Optimizations according to steps outlined at Added Profile Guided Optimizations To Boost PHP 7 Performance. That is you need to recompile/compile latest PHP 7 version via centmin.sh menu option 5 with PHP_PGO='y' set in persistent config file /etc/centminmod/custom_config.inc before you actually run centmin.sh menu option 5.

You can see PGO vs non-PGO general benchmarks posted at PHP - PHP 7.2.0 GA Stable vs PHP 7.1.12 vs 7.0.26 vs 5.6.32 Benchmarks.



November 26, 2018 added even more dramatic example of PGO boosted performance by PGO retraining a php benchmark script detailed_benchmark.php with PHP 7.3 resulting in ~25% better PHP performance at PHP - PHP 7.3 vs 7.2 vs 7.1 vs 7.0 (PHP-FPM) Benchmarks

November 2020 added PHP 8.0 PGO and PHP 8.0 JIT benchmarks

Optimising Nginx Compilation with the right GCC/Clang compiler

Supported branch: Centmin Mod 123.09beta01 and higher only

99.9999% of all LEMP/LAMP stacks or control panels default to installing Nginx on CentOS/Redhat or Debian/Ubuntu etc with their OS's respective package management repositories. For CentOS/Redhat that is installing via YUM/RPMs or you can install Nginx via source compilation which is what Centmin Mod LEMP stack does.

When you install Nginx via YUM/RPMs, you are installing a prebuilt Nginx package which is compiled using the CentOS native GCC compiler by default. For CentOS 6.x, GCC 4.4.7 is used and for CentOS 7.x, GCC 4.8.5 is used. Without going into the technical side of things, lets just say for newer cpu/processor models, you can get a potential boost in performance by pairing your Centmin Mod compiles/installs with a new compiler version or alternate compiler like Clang instead of GCC. On older cpu models, you may not get a boost in performance and depending on cpu model and compiler pairing, you can also have performance regressions and thus lower performance. So this is an exercise of trial and error testing to find the right compiler for your specific cpu on your server.

Currently, Centmin Mod 123.09beta01 has native support for compiling Nginx with the following compilers. Instructions to switch and use these compilers for Nginx are outlined here.

• GCC native version
• GCC 5.3.1
• GCC 6.3.1
• GCC 7.3.1
• GCC 8.0/8.1/8.2
• GCC 9
• GCC 10 <- current default as at November 6, 2021
• GCC 11
• Clang 3.4.2
• Clang 4.0.1
• Clang 5.0.1
• Clang 6.0.0
• Clang 7.0

Centmin Mod Nginx performance compared with different GCC/Clang compilers



Centmin Mod Nginx & custom Caddy web server compiled and benchmarked with different GCC/Clang compilers here. Specific post results here.



Updated: February 2, 2019 GCC 8.2.1 vs GCC 7.3.1 vs GCC 4.8.5 vs Clang 5.0.1 Nginx compiled benchmarks

Use Brotli compression with HTTPS based web sites

Supported branch: Centmin Mod 123.09beta01 and higher only

Update: February 2, 2019 GCC 8.2.1 vs GCC 7.3.1 vs GCC 4.8.5 vs Clang 5.0.1 Nginx compiled benchmarks again highlight Nginx Brotli vs Gzip performance. Updated February 21, 2019 - the assumption was flawed that Intel Skylake cpus accelerated brotli compression when it fact it was an incorrect ngx_brotli setting set in Centmin Mod Nginx default settings as explained here.

Update: December 17, 2018, seems Nginx Brotli compression over HTTP/2 HTTPS has much better performance compared to standard Nginx gzip/zlib compression using the newer Intel Xeon Gold Scalable Skylake cpus from DigitalOcean Xeon Gold 6140 KVM VPS droplet benchmarks. Not sure if it's same with all classes of Skylake cpus - desktop vs server class. So picking the right cpu for your server for Nginx Brotli compression maybe important Updated February 21, 2019 - the assumption was flawed that Intel Skylake cpus accelerated brotli compression when it fact it was an incorrect ngx_brotli setting set in Centmin Mod Nginx default settings as explained here.

Update: November 4, 2018, brotli compression has been updated so it only uses brotli compression for static files greater than 65536 bytes as files below this are actually slower than gzip according to Cloudflare tests.

Centmin Mod Nginx Brotli is used for page load speed - better compression means smaller compressed file sizes. But it comes at expense of scalability in terms of handling larger concurrency traffic loads according to benchmarks here. However, this only applies to Nginx Brotli implementation. If you use Cloudflare in front of your server, using Cloudflare's Brotli implementation is faster than Cloudflare zlib/gzip in terms of page load speed + scales just as well so Cloudflare Brotli is a good to have enabled. So for Centmin Mod Nginx you may want to disable ngx_brotli (the default state already) if you have alot of concurrent user traffic and you're hitting server resource limits. Though for page load speed ngx_brotli is better so if you have the server resources to handle the concurrent traffic load and not running against server resource limits, ngx_brotli is still viable option to enable.

Traditionally, for web servers serving static files like html, css, js etc can be either served uncompressed or compressed. When compressed they normally use Gzip based compression. However, Centmin Mod Nginx also supports optional Google Brotli compression which can potentially compress files better than Gzip resulting in smaller compressed static files served to visitors. Google uses Brotli compression extensively on their web properties to save bandwidth and speed up page loads by reducing static file assets' average sizes.

Centmin Mod Nginx server can enable ngx_brotli Nginx module by following instructions outlined at Nginx - How to use Brotli compression for Centmin Mod Nginx web servers. Note, Brotli is only used for HTTPS web sites. For non-HTTPS, Nginx will fall back to normal Gzip compression.

Use Google ngx_pagespeed Nginx module

Supported branch: Centmin Mod 123.09beta01 and higher only. Update: with Cloudflare's service and feature improvements, these days generally don't recommend Google ngx_pagespeed Nginx module anymore. A optimally configured Cloudflare CDN proxy setup in front of your Centmin Mod Nginx server will be the best way to improvement performance for all visitors from all geographical locations.

Google ngx_pagspeed Nginx module is same as Google mod_pagespeed Apache module. As such the Google Pagespeed documentation for both mod_pagespeed and ngx_pagespeed are located at the same location here. It allows you to automate the Google Pagespeed optimisation recommendations to a certain extent on the fly without any modifications to your web apps/scripts generally.

Centmin Mod Nginx has supported ngx_pagspeed since June 2013 and supported Nginx PageSpeed module as a dynamic module since February 2016. The benefits that Nginx PageSpeed provide of improved page load speed are real Nginx PageSpeed - Benefits of ngx_pagespeed

Note, ngx_pagespeed isn't set and forget. Depending on your web site style/scripts etc, there maybe alot of troubleshooting to disable or tweak certain ngx_pagespeed settings and filters to allow it to work on your web site. Also you won't be able to use reverse proxy type of caching out of the box with ngx_pagespeed i.e. fastcgi_cache, proxy_cache, varnish cache as there are more extensive configuration options for some of these outlined for downstream caching. You'd be responsible for learning how to do that yourself as Centmin Mod is provided as is. As such, in some cases it maybe a choice between reverse proxy type of cache acceleration vs using ngx_pagespeed as they may not work well together.

You can read up on how to enable Google ngx_pagespeed module via below links:

• Nginx - Optimizing Your Website with the Google PageSpeed Dynamic Module for NGINX Plus
• Nginx PageSpeed - CentminMod.com LEMP Nginx web stack for CentOS

Leverage Nginx Thread Pooling

Centmin Mod Nginx supports Nginx thread pooling as outlined at Boosting NGINX Performance 9x with Thread Pools via --with-threads configuration parameter. While support is there, you will need to configure it yourself as instructed from official Nginx blog link above to benefit certain workloads.

Enable other Cloudflare related patches

---

Supported branch: Centmin Mod 123.09beta01 and higher only

Since Centmin Mod source compiles Nginx and PHP-FPM, it allows Centmin Mod to support on the fly patching of Nginx and OpenSSL source code to implement performance improvements. Such improvements predominantly come from Cloudflare's open source provided patches to improve Nginx and OpenSSL performance.

These patches are disabled by default as development updates on patch side as well as on Nginx/OpenSSL side can break or cause issues with functionality or prevent Nginx from compiling. So Centmin Mod allows you to enable or disable each of these patches and decide for yourself if it's worth using.

• Cloudflare ECDSA performance patch for up to 40+ % improvement for HTTPS using ECC 256bit ECDSA SSL certificates. No longer required as 123.09beta01 and higher default to OpenSSL 1.1.1 which have native ECDSA performance patch added.
• Cloudflare HTTP/2 HTTPS HPACK Full Encoding patch when using Nginx with OpenSSL. Note one user has reported issues with static file loading issues on Microsoft Edge browsers only - so test on all web browsers.
• Cloudflare HTTPS Dynamic TLS Record Size patch
• Cloudflare zlib performance fork patch for Nginx zlib below benchmark numbers show 22-29% better performance for gzip compressed HTTP requests !
• Cloudflare zlib performance fork library usage for PHP-FPM - just added as at Jan 24th, 2018

Cloudflare Zlib Performance Numbers

As part of this How to boost Centmin Mod LEMP stack performance guide, I did some benchmark comparisons for each of the supported Centmin Mod Nginx compiler build options for GCC 4.8.5, 5.3.1, 6.3.1, 7.2.1, 8.0 and Clang 3.4.2, 4.0.1, 5.0.1, 6.0.0 with the standard default Nginx Zlib library versus Cloudflare's Zlib performance fork library. I did load testing benchmarks using my own custom forked version of wrk load testing tool testing gzip compressed requests so as to properly test each Zlib library https://github.com/centminmod/wrk/tree/centminmod.

As you can see, Cloudflare's Zlib performance fork library has around 22-29% better performance than the default Centmin Mod Nginx Zlib library performance which is compiled against Zlib v1.2.11 on CentOS 7.4.

You can view the raw wrk benchmark numbers here.

HTTP/2 HTTPS & Server Push

Centmin Mod Nginx web server supports Nginx HTTPS and uses HTTP/2 based HTTPS as opposed to slower and older HTTP/1.1 based HTTPS. This post doesn't have that much room to go into details so just leave you with these 2 links to HTTP/2 Revisited Youtube video and Centmin Mod Nginx 1.13.9+ soon to be released HTTP/2 Server Push. There's also soon to be announced OpenSSL 1.1.1 with TLS 1.3 support that you should watch out for.

1. SSL - WebPerf - PageSpeed - HTTP/2 HTTPS Revisited
2. Nginx - Nginx HTTP/2 Server Push Finally Supported
3. OpenSSL - OpenSSL 1.1.1 First Alpha Pre-Release 1 Out

HTTPS TLS 1.3 Support

From Nginx 1.15.3+ mainline and newer, Nginx supports newest TLS 1.3 protocol for HTTPS secure connections when either compiled with OpenSSL 1.1.1 or BoringSSL crypto libraries. Not all LEMP stacks use OpenSSL 1.1.1, so not all support HTTPS with TLS 1.3. However, Centmin Mod 123.09beta01 and newer support both OpenSSL 1.1.1 or BoringSSL based Nginx builds. Details and links outlined at Centmin Mod Nginx HTTP/2 HTTPS TLS 1.3 Support.

If you use Cloudflare in front of a Centmin Mod Nginx HTTP/2 HTTPS TLS 1.3 supported server, you get added performance benefit now that Cloudflare official supports TLS 1.3 on backend origin server connections which can speak TLS 1.3 (like Centmin Mod Nginx can). Full details at SSL - Cloudflare - Cloudflare Enables HTTPS TLS 1.3 Backend Origin Communication

Leverage Linux 5.1+ Kernels and Nginx io_uring patch for better Async I/O

Centmin Mod 123.09beta01 added NGINX_IOURING_PATCH variable support which can be enabled if centmin.sh menu option 4 Nginx builds detect that Nginx 1.17.0+ + Linux 5.1+ Kernels are in use and when NGINX_IOURING_PATCH='y' is enabled via persistent config file /etc/centminmod/custom_config.inc setting (disabled by default). This allows Centmin Mod Nginx 1.17.0+ server binaries to leverage and use Linux 5.1+ Kernel's io_ring support for more efficient buffered AIO with less system calls - better Nginx disk I/O performance. More reading on io_uring below:

• Ringing in a new asynchronous I/O API [LWN.net]
• Linux 5.1 out with Io_uring IO interface, persistent memory, new patching improvements and more! | Packt Hub
• Native Linux asynchronous file operation, io_uring experience - Develop Paper
• older benchmarks done by io_uring author https://lore.kernel.org/linux-block/20190116175003.17880-1-axboe@kernel.dk/

 

Subscribe For Updates

This thread will be updated over time, so you may want to subscribe and watch this thread for update notifications. And if you like this guide or find Centmin Mod useful, please consider supporting development via a donation

 

Short Bookmark Url

To make accessing and remembering the link to this thread easier, I created 2 short url links below. Bookmark

• https://centminmod.com/perf/
• https://centminmod.com/performance/

 

Centmin mod Performance Boost Configuration Tip Updates

To update your existing Centmin Mod 123.09beta01 installs, just update Centmin Mod code via SSH command = cmupdate and then recompile Nginx via centmin.sh menu option 4.

February 24, 2018

• Nginx compiles by default now switch to better performance configurations by default. Switching from Clang 3.4.2 compiler to GCC 7.2.1 compiler and switching from default LibreSSL 2.6.4+ to OpenSSL 1.1.0g (or 1.0.2n if Lua Nginx modules optionally enabled) [details].
• Cloudflare zlib library usage for Nginx zlib is enabled by default now [details]

May 19th, 2018

• Added notes regarding Nginx Brotli usage for page load speed vs scalability.

June 11th, 2018

• When Cloudflare zlib library is used raise nginx default gzip compression level from 5 to 9

September 11th, 2018

• Centmin Mod 123.09beta01 and newer support both OpenSSL 1.1.1 or BoringSSL based Nginx builds for TLS 1.3 protocol support. Details and links outlined at Centmin Mod Nginx HTTP/2 HTTPS TLS 1.3 Support

February 3rd, 2019

• Add links to Beta Branch - Testing GCC 8 & Clang 5/6/7 compiler for Centmin Mod Nginx & PHP-FPM for newer GCC benchmarks and brotli vs gzip benchmarks.

May 6th, 2019

• Add link to Cloudflare TLS 1.3 support for backend origin connections https://community.centminmod.com/th...s-tls-1-3-backend-origin-communication.16795/

August 6th, 2019

• Add leverage Linux 5.1+ Kernels and Nginx io_uring patch for better Async I/O info (https://community.centminmod.com/th...patch-variable-support-in-123-09beta01.18075/)

November 6th, 2021

• Add GCC 10 support for Nginx and FFMPEG compilations by default