Nginx Nginx Static HTML Benchmarks - Centmin Mod vs Easyengine vs Webinoly vs VestaCP vs OneInStack

Now that I have setup my ssdnode's 4 CPU KVM VPS with Ubuntu 18.04 LTS and LXD containers, I can spin up multiple LXD guest containers for CentOS 7, Ubuntu 16.04 LTS and Ubuntu 18.04 LTS to do more testing on various OSes for Centmin Mod continued performance and scalability optimisation testing and development. Update: Nginx HTTP/2 HTTPS benchmark comparison is also underway. PHP comparison benchmarks are posted here.

Nginx Benchmarks Background

• I have limited experience with Easyengine for Ubuntu and first time installing Webinoly - which seems to be a fork based on Easyengine with more updates. Being unfamiliar with the 2, I only setup a -php basic site for them and today I am just testing the respective LEMP stack's Nginx static HTML performance using my forked version of wrk, wrk-cmm.
• The test index.html page is taken from Centmin Mod default nginx generated index.html place holder page.
• Centmin Mod LEMP stack using 123.09beta01 beta Nginx 1.15.0 on CentOS 7.5 64bit (default gzip compression = 5) and retested at gzip compression level = 6 and retested at gzip compression level = 9
• Easyengine 3.8.1 using Nginx 1.14.0 on Ubuntu 16.04 LTS
• Webinoly 1.4.3 using Nginx 1.14.0 on Ubuntu 18.04 LTS
• Updated: VestaCP 0.9.8-21 using Nginx 1.15.0 on Ubuntu 16.04 LTS
• Updated: OneInStack Nginx 1.14.0 & OneInStack OpenResty Nginx on Ubuntu 16.04 LTS

LXD container OS listing where centos75-2 was used for Centmin Mod

Code (Text):

lxc list
+-------------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+
|             NAME              |  STATE  |         IPV4         |                     IPV6                      |    TYPE    | SNAPSHOTS |
+-------------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| centos75                      | STOPPED |                      |                                               | PERSISTENT | 1         |
+-------------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| centos75-2                    | RUNNING | 10.71.164.50 (eth0)  | fd42:769c:ebd9:a0f7:216:3eff:fe2e:a205 (eth0) | PERSISTENT | 1         |
+-------------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| easyengine-ubuntu1604         | RUNNING | 10.71.164.21 (eth0)  | fd42:769c:ebd9:a0f7:216:3eff:fe44:b2ee (eth0) | PERSISTENT | 0         |
+-------------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| oneinstack-ubuntu16-nginx     | RUNNING | 10.71.164.124 (eth0) | fd42:769c:ebd9:a0f7:216:3eff:fe4a:4878 (eth0) | PERSISTENT | 2         |
+-------------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| oneinstack-ubuntu16-openresty | RUNNING | 10.71.164.25 (eth0)  | fd42:769c:ebd9:a0f7:216:3eff:fe1f:b702 (eth0) | PERSISTENT | 1         |
+-------------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| oneinstack-ubuntu16-tengine   | RUNNING | 10.71.164.68 (eth0)  | fd42:769c:ebd9:a0f7:216:3eff:fe8c:4513 (eth0) | PERSISTENT | 0         |
+-------------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| vestacp-ubuntu1604            | RUNNING | 10.71.164.97 (eth0)  | fd42:769c:ebd9:a0f7:216:3eff:fef2:b4ae (eth0) | PERSISTENT | 1         |
+-------------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| webinoly                      | RUNNING | 10.71.164.45 (eth0)  | fd42:769c:ebd9:a0f7:216:3eff:fec7:d433 (eth0) | PERSISTENT | 0         |
+-------------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+

ssdnodes 4cpu KVM VPS is Intel Xeon Scalable Skylake processor with 16GB memory and 80GB SSD disk

---

Code (Text):

lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    1
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 94
Model name:            Intel Core Processor (Skylake, IBRS)
Stepping:              3
CPU MHz:               2099.998
BogoMIPS:              4199.99
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              4096K
NUMA node0 CPU(s):     0-3
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 ibpb ibrs

Code (Text):

uname -r
4.15.0-22-generic

Code (Text):

free -m
              total        used        free      shared  buff/cache   available
Mem:          16040        2837        3671          74        9531       12799
Swap:             0           0           0

Code (Text):

df -hT
Filesystem                  Type      Size  Used Avail Use% Mounted on
udev                        devtmpfs  7.9G     0  7.9G   0% /dev
tmpfs                       tmpfs     1.6G  968K  1.6G   1% /run
/dev/mapper/ubuntu--vg-root ext4       79G   38G   37G  51% /
tmpfs                       tmpfs     7.9G     0  7.9G   0% /dev/shm
tmpfs                       tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs                       tmpfs     7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sda1                   ext2      472M  128M  320M  29% /boot
/dev/loop0                  squashfs   87M   87M     0 100% /snap/core/4650
/dev/loop1                  squashfs   55M   55M     0 100% /snap/lxd/7412
tmpfs                       tmpfs     1.6G     0  1.6G   0% /run/user/0

 

Centmin Mod 123.09beta01 LEMP Stack

LEMP stack install

Code (Text):

yum -y update; curl -O https://centminmod.com/betainstaller.sh && chmod 0700 betainstaller.sh && bash betainstaller.sh

http2.domain.com nginx vhost creation

Code (Text):

nv -d http2.domain.com -s y -u ftpusername

OS

Code (Text):

cat /etc/redhat-release 
CentOS Linux release 7.5.1804 (Core) 

Nginx version

 

Easyengine 3.8.1 LEMP Stack

LEMP stack install

Code (Text):

wget -qO ee rt.cx/ee && sudo bash ee

http2.domain.com nginx vhost creation

Code (Text):

sudo ee site create http2.domain.com --php

OS

Code (Text):

cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.4 LTS"

Nginx version

 

Webinoly 1.4.3 LEMP Stack

LEMP stack install

Code (Text):

wget -qO weby qrok.es/wy && sudo bash weby 3

http2.domain.com nginx vhost creation

Code (Text):

sudo site http2.domain.com -php

OS

Code (Text):

cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04 LTS"

Nginx version

 

wrk-cmm HTTP Benchmarks

Nginx static HTML benchmarks are done using my forked version of wrk, wrk-cmm. Each test configuration was run 2x times. Raw numbers are further below while summary chart is directly below:

wrk-cmm load tests were done at 4 user concurrency levels - 10 users, 100 users, 500 users and 1000 users for 10 second duration using following test parameters

Code (Text):

wrk-cmm -c10 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
wrk-cmm -c100 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
wrk-cmm -c500 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
wrk-cmm -c1000 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/

• at 10 user concurrency, Centmin Mod Nginx is 16.7% faster than Easyengine Nginx and 8.76% faster than Webinoly Nginx and 18.39% faster than VestaCP Nginx and Webinoly is 7.3% faster than Easyengine Nginx and 8.86% faster than VestaCP Nginx
• at 100 user concurrency, Centmin Mod Nginx is 55.77% faster than Easyengine Nginx and 32.04% faster than Webinoly Nginx and 56.9% faster than VestaCP Nginx and Webinoly is 17.8% faster than Easyengine Nginx and 18.86% faster than VestaCP Nginx
• at 500 user concurrency, Centmin Mod Nginx is 39.73% faster than Easyengine Nginx and 33.45% faster than Webinoly Nginx and 41.3% faster than VestaCP Nginx and Webinoly is 4.7% faster than Easyengine Nginx and 5.9% faster than VestaCP Nginx
• at 1000 user concurrency, Centmin Mod Nginx is 43.70% faster than Easyengine Nginx and 33.08% faster than Webinoly Nginx and 39.06% faster than VestaCP Nginx and Webinoly is 7.97% faster than Easyengine Nginx and 4.49% faster than VestaCP Nginx

Update: added VestaCP Nginx benchmarks too



Updated and retested Centmin Mod Nginx at gzip compression level 6 (level 5 is out of box default for Centmin Mod Nginx)



Updated Centmin Mod Nginx gzip level 9 compression tests



Updated OneInStack Nginx 1.14.0 & OneInStack OpenResty Nginx on Ubuntu 16.04 LTS



Previous charts used Google Sheets and was limited in formatting options, so downloaded to local Excel to better format the charts



More condensed summary so far by only focusing on wrk-cmm 500 and 1000 user load tests for Centmin Mod vs Easyengine vs Webinoly vs VestaCP VS OneInStack



CentOS wrk-cmm install steps

Code (Text):

cd /svr-setup; rm -rf wrk-cmm; git clone -b centminmod https://github.com/centminmod/wrk wrk-cmm; cd wrk-cmm; make -j$(nproc); \cp -af wrk /usr/local/bin/wrk-cmm

Code (Text):

wrk-cmm --version
wrk 4.1.0-22-g3d33cee [epoll] Copyright (C) 2012 Will Glozer
Usage: wrk <options> <url>               
  Options:                               
    -c, --connections <N>  Connections to keep open
    -d, --duration    <T>  Duration of test
    -t, --threads     <N>  Number of threads to use
                                         
    -b, --bind-ip     <S>  Source IP (or CIDR mask)
                                         
    -s, --script      <S>  Load Lua script file
    -H, --header      <H>  Add header to request
        --latency          Print latency statistics
        --breakout         Print breakout statistics
        --timeout     <T>  Socket/request timeout
    -v, --version          Print version details
                                         
  Numeric arguments may include a SI unit (1k, 1M, 1G)
  Time arguments may include a time unit (2s, 2m, 2h

Ubuntu wrk-cmm install steps

Code (Text):

sudo apt -y install gcc make luajit; sudo mkdir -p /svr-setup; cd /svr-setup; rm -rf wrk-cmm; git clone -b centminmod https://github.com/centminmod/wrk wrk-cmm; cd wrk-cmm; make -j$(nproc); \cp -af wrk /usr/local/bin/wrk-cmm

Code (Text):

wrk-cmm --version
wrk 4.1.0-22-g3d33cee [epoll] Copyright (C) 2012 Will Glozer
Usage: wrk <options> <url>               
  Options:                               
    -c, --connections <N>  Connections to keep open
    -d, --duration    <T>  Duration of test
    -t, --threads     <N>  Number of threads to use
                                         
    -b, --bind-ip     <S>  Source IP (or CIDR mask)
                                         
    -s, --script      <S>  Load Lua script file
    -H, --header      <H>  Add header to request
        --latency          Print latency statistics
        --breakout         Print breakout statistics
        --timeout     <T>  Socket/request timeout
    -v, --version          Print version details
                                         
  Numeric arguments may include a SI unit (1k, 1M, 1G)
  Time arguments may include a time unit (2s, 2m, 2h)

 

Centmin Mod wrk-cmm HTTP benchmark results

Code (Text):

curl -I http://http2.domain.com/index.html
HTTP/1.1 200 OK
Date: Sun, 10 Jun 2018 05:55:43 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 6457
Last-Modified: Fri, 08 Jun 2018 03:54:47 GMT
Connection: keep-alive
Vary: Accept-Encoding
ETag: "5b19fe07-1939"
Server: nginx centminmod
X-Powered-By: centminmod
Accept-Ranges: bytes

10 users

Code (Text):

wrk-cmm -c10 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/ 
Running 10s test @ http://http2.domain.com/
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   669.24us  816.21us  19.81ms   96.12%
    Req/Sec     8.62k     2.18k   13.52k    67.00%
  171505 requests in 10.01s, 396.30MB read
Requests/sec:  17139.55
Transfer/sec:     39.61MB

wrk-cmm -c10 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   776.66us    1.33ms  28.33ms   96.13%
    Req/Sec     8.71k     2.17k   16.69k    66.67%
  174188 requests in 10.10s, 402.50MB read
Requests/sec:  17246.92
Transfer/sec:     39.85MB

100 users

Code (Text):

wrk-cmm -c100 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency     5.10ms    3.48ms 125.86ms   79.00%
   Req/Sec    10.30k     1.66k   14.82k    79.50%
  204914 requests in 10.02s, 473.50MB read
Requests/sec:  20458.64
Transfer/sec:     47.27MB

wrk-cmm -c100 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency     4.66ms    3.04ms 104.04ms   74.28%
   Req/Sec    11.19k     1.40k   14.29k    65.00%
  222658 requests in 10.01s, 514.51MB read
Requests/sec:  22238.10
Transfer/sec:     51.39MB

500 users

Code (Text):

wrk-cmm -c500 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency    31.35ms   63.54ms   1.53s    98.77%
   Req/Sec     9.83k     1.92k   12.69k    77.50%
  195669 requests in 10.05s, 452.14MB read
Requests/sec:  19476.06
Transfer/sec:     45.00MB

wrk-cmm -c500 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency    24.06ms    8.34ms 370.34ms   87.40%
   Req/Sec    10.50k   748.56    12.07k    75.00%
  209011 requests in 10.04s, 482.97MB read
Requests/sec:  20827.22
Transfer/sec:     48.13MB

1000 users

Code (Text):

wrk-cmm -c1000 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency    49.59ms   15.42ms 414.43ms   82.73%
   Req/Sec    10.12k     0.89k   11.47k    83.50%
  201388 requests in 10.08s, 465.36MB read
Requests/sec:  19983.09
Transfer/sec:     46.18MB

wrk-cmm -c1000 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency    51.32ms   15.67ms 141.33ms   73.88%
   Req/Sec     9.76k   849.34    10.75k    86.50%
  194337 requests in 10.05s, 449.06MB read
Requests/sec:  19335.37
Transfer/sec:     44.68MB

 

Easyengine wrk-cmm HTTP benchmark results

Code (Text):

curl -I http://http2.domain.com/index.html
HTTP/1.1 200 OK
Server: nginx
Date: Sun, 10 Jun 2018 06:28:17 GMT
Content-Type: text/html
Content-Length: 6456
Last-Modified: Sun, 10 Jun 2018 06:28:01 GMT
Connection: keep-alive
Vary: Accept-Encoding
ETag: "5b1cc4f1-1938"
X-Powered-By: EasyEngine 3.8.1
Accept-Ranges: bytes

10 users

Code (Text):

wrk-cmm -c10 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency   676.50us  302.85us  11.22ms   87.41%
   Req/Sec     7.50k   679.18     9.28k    77.00%
  149227 requests in 10.01s, 344.11MB read
Requests/sec:  14911.69
Transfer/sec:     34.39MB

wrk-cmm -c10 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency   728.62us  644.63us  20.52ms   97.70%
   Req/Sec     7.32k     0.95k    9.51k    84.00%
  145638 requests in 10.01s, 335.83MB read
Requests/sec:  14553.09
Transfer/sec:     33.56MB

100 users

Code (Text):

wrk-cmm -c100 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency     7.74ms    4.92ms  36.53ms   72.90%
   Req/Sec     6.90k     1.33k   10.23k    76.00%
  137387 requests in 10.00s, 316.81MB read
Requests/sec:  13732.31
Transfer/sec:     31.67MB

wrk-cmm -c100 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency     7.49ms    3.65ms  71.40ms   82.82%
   Req/Sec     6.89k     0.96k    9.09k    79.50%
  137231 requests in 10.03s, 316.45MB read
Requests/sec:  13678.04
Transfer/sec:     31.54MB

500 users

Code (Text):

wrk-cmm -c500 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency    35.39ms   23.32ms  93.32ms   58.93%
   Req/Sec     7.14k   837.47     9.45k    70.00%
  142017 requests in 10.02s, 327.48MB read
Requests/sec:  14166.64
Transfer/sec:     32.67MB

wrk-cmm -c500 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency    34.45ms   16.26ms 101.83ms   73.28%
   Req/Sec     7.40k     0.86k    9.36k    66.50%
  147240 requests in 10.03s, 339.53MB read
Requests/sec:  14677.58
Transfer/sec:     33.85MB

1000 users

Code (Text):

wrk-cmm -c1000 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency    70.95ms   53.17ms 184.46ms   42.64%
   Req/Sec     7.13k     0.91k    9.42k    67.50%
  141891 requests in 10.07s, 327.19MB read
Requests/sec:  14092.73
Transfer/sec:     32.50MB

wrk-cmm -c1000 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency    76.78ms   48.99ms 267.05ms   70.72%
   Req/Sec     6.71k     1.78k   10.74k    65.50%
  133533 requests in 10.06s, 307.92MB read
Requests/sec:  13270.07
Transfer/sec:     30.60MB

 

Webinoly wrk-cmm HTTP benchmark results

Code (Text):

curl -I http://http2.domain.com/index.html
HTTP/1.1 200 OK
Server: nginx
Date: Sun, 10 Jun 2018 06:11:46 GMT
Content-Type: text/html
Content-Length: 6457
Last-Modified: Sun, 10 Jun 2018 05:51:21 GMT
Connection: keep-alive
Vary: Accept-Encoding
ETag: "5b1cbc59-1939"
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Cache-Control: public, no-cache
Referrer-Policy: unsafe-url
Accept-Ranges: bytes

10 users

Code (Text):

wrk-cmm -c10 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency   681.05us  663.29us  24.68ms   97.67%
   Req/Sec     7.86k     1.03k    9.81k    73.76%
  157880 requests in 10.10s, 382.88MB read
Requests/sec:  15631.99
Transfer/sec:     37.91MB

wrk-cmm -c10 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency   642.87us  427.48us  17.36ms   96.37%
   Req/Sec     8.02k     0.90k   11.45k    84.16%
  161274 requests in 10.09s, 391.11MB read
Requests/sec:  15983.42
Transfer/sec:     38.76MB

100 users

Code (Text):

wrk-cmm -c100 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency     6.40ms    2.50ms  33.02ms   77.69%
   Req/Sec     7.91k   730.23    10.43k    76.50%
  157482 requests in 10.02s, 381.92MB read
Requests/sec:  15716.90
Transfer/sec:     38.12MB

wrk-cmm -c100 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency     6.09ms    2.65ms  33.96ms   80.28%
   Req/Sec     8.37k     1.07k   13.35k    85.00%
  166667 requests in 10.03s, 404.19MB read
Requests/sec:  16619.67
Transfer/sec:     40.31MB

500 users

Code (Text):

wrk-cmm -c500 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency    44.22ms   44.18ms 172.71ms   77.55%
   Req/Sec     7.18k     1.58k   10.51k    68.00%
  142874 requests in 10.03s, 346.49MB read
Requests/sec:  14244.53
Transfer/sec:     34.55MB

wrk-cmm -c500 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency    39.06ms   37.55ms 160.80ms   77.20%
   Req/Sec     8.04k     3.85k   13.49k    46.00%
  160009 requests in 10.03s, 388.05MB read
Requests/sec:  15957.01
Transfer/sec:     38.70MB

1000 users

Code (Text):

wrk-cmm -c1000 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency    72.69ms   59.78ms 335.04ms   75.95%
   Req/Sec     7.70k     3.57k   13.22k    54.50%
  153183 requests in 10.07s, 371.49MB read
Requests/sec:  15217.59
Transfer/sec:     36.91MB

wrk-cmm -c1000 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency    78.93ms   69.85ms 466.64ms   62.99%
   Req/Sec     7.24k     4.00k   13.69k    56.00%
  144082 requests in 10.06s, 349.42MB read
Requests/sec:  14326.37
Transfer/sec:     34.74MB

 

not sure if serverpilot supports installs on LXD containers with ssdnodes ? Using LXD containers so the exact same underlying hardware is used simultaneously for all lemp stacks for better apples to apples comparisons. Will see.. now that I have LXD container support setup, I can simultaneously test various OSes/lemp stacks

but serverpilot might need direct SSH connection ? LXD container setup I have doesn't have direct LXD container level SSH connection from outside world yet as you need iptables/UFW firewall redirect rules from host to LXD container and if multiple web facing LXD containers are used need a load balancer in between I think ? Still new to LXD heh.

edit: woohoo, managed to setup Ubuntu 18.04 LTS LXD host's UFW firewall redirect rules for custom sshd port for a fresh Ubuntu 18.04 LTS LXD guest container so outside world can connect to LXD guest container via ssh But serverpilot's initial connection via SSH is assuming default sshd port 22 but LXD containers need custom sshd port as UFW firewall needs prerouting rules from public ip + sshd port to the LXD container as you can't redirect to sshd port 22 if you have multiple LXD guest containers - each needs it's own sshd custom listening port.

Code (Text):

lxc list
+------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+
|          NAME          |  STATE  |         IPV4         |                     IPV6                      |    TYPE    | SNAPSHOTS |
+------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| centos75               | RUNNING | 10.71.164.168 (eth0) | fd42:769c:ebd9:a0f7:216:3eff:fefd:23a2 (eth0) | PERSISTENT | 2         |
+------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| centos75-2             | RUNNING | 10.71.164.50 (eth0)  | fd42:769c:ebd9:a0f7:216:3eff:fe2e:a205 (eth0) | PERSISTENT | 1         |
+------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| centos75-3             | RUNNING | 10.71.164.174 (eth0) | fd42:769c:ebd9:a0f7:216:3eff:fe50:5edb (eth0) | PERSISTENT | 0         |
+------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| easyengine-ubuntu1604  | RUNNING | 10.71.164.21 (eth0)  | fd42:769c:ebd9:a0f7:216:3eff:fe44:b2ee (eth0) | PERSISTENT | 0         |
+------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| easyengine-ubuntu1804  | RUNNING | 10.71.164.63 (eth0)  | fd42:769c:ebd9:a0f7:216:3eff:fe12:53aa (eth0) | PERSISTENT | 0         |
+------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| serverpilot-ubuntu1804 | RUNNING | 10.71.164.228 (eth0) | fd42:769c:ebd9:a0f7:216:3eff:fe4a:8b8c (eth0) | PERSISTENT | 0         |
+------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| ubuntu18               | RUNNING | 10.71.164.145 (eth0) | fd42:769c:ebd9:a0f7:216:3eff:fe42:4ca8 (eth0) | PERSISTENT | 1         |
+------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| webinoly               | RUNNING | 10.71.164.45 (eth0)  | fd42:769c:ebd9:a0f7:216:3eff:fec7:d433 (eth0) | PERSISTENT | 0         |
+------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+

 

VestaCP 0.9.8-21 LEMP Stack

New LEMP stack addition for VestaCP 0.9.8-21 on Ubuntu 16.04 LXD container

LEMP stack install

Code (Text):

curl -O http://vestacp.com/pub/vst-install.sh
bash vst-install.sh --nginx yes --phpfpm yes --apache no --named yes --remi yes --vsftpd yes --proftpd no --iptables yes --fail2ban yes --quota no --exim yes --dovecot yes --spamassassin yes --clamav yes --softaculous yes --mysql yes --postgresql no --hostname vestacp-ubuntu1604 --email $email --password $pass -f

http2.domain.com nginx vhost creation via control panel gui

OS

Code (Text):

cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.4 LTS"

Nginx version

VestaCP wrk-cmm HTTP benchmark results

Code (Text):

curl -I http://http2.domain.com/index.html
HTTP/1.1 200 OK
Server: nginx
Date: Mon, 11 Jun 2018 00:04:24 GMT
Content-Type: text/html
Content-Length: 6456
Last-Modified: Sun, 10 Jun 2018 23:56:56 GMT
Connection: keep-alive
Keep-Alive: timeout=60
ETag: "5b1dbac8-1938"
Accept-Ranges: bytes

10 users

Code (Text):

wrk-cmm -c10 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   710.37us  462.95us  13.27ms   95.53%
    Req/Sec     7.30k   827.24     9.27k    78.50%
  145249 requests in 10.00s, 329.78MB read
Requests/sec:  14523.06
Transfer/sec:     32.97MB

wrk-cmm -c10 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   712.13us  543.23us  21.33ms   96.73%
    Req/Sec     7.33k     0.96k   13.51k    85.07%
  146677 requests in 10.10s, 333.02MB read
Requests/sec:  14522.25
Transfer/sec:     32.97MB

100 users

Code (Text):

wrk-cmm -c100 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency     7.38ms    2.31ms  51.58ms   81.71%
   Req/Sec     6.85k   500.64     8.07k    81.50%
  136368 requests in 10.02s, 309.61MB read
Requests/sec:  13607.66
Transfer/sec:     30.90MB

wrk-cmm -c100 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency     7.32ms    1.93ms  25.96ms   74.15%
   Req/Sec     6.86k   408.43     8.01k    72.00%
  136537 requests in 10.04s, 310.00MB read
Requests/sec:  13601.60
Transfer/sec:     30.88MB

500 users

Code (Text):

wrk-cmm -c500 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency    35.40ms   21.29ms 127.29ms   70.07%
   Req/Sec     7.26k     1.04k    9.80k    62.00%
  144452 requests in 10.07s, 327.98MB read
Requests/sec:  14344.26
Transfer/sec:     32.57MB

wrk-cmm -c500 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency    35.30ms   17.07ms 146.00ms   66.09%
   Req/Sec     7.18k   837.54     8.77k    70.00%
  142929 requests in 10.08s, 324.51MB read
Requests/sec:  14176.04
Transfer/sec:     32.19MB

1000 users

Code (Text):

wrk-cmm -c1000 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency    75.14ms   58.59ms 283.79ms   66.14%
   Req/Sec     7.20k     1.67k   11.67k    63.32%
  142431 requests in 10.10s, 323.39MB read
Requests/sec:  14106.63
Transfer/sec:     32.03MB

wrk-cmm -c1000 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency    69.30ms   22.12ms 177.87ms   74.79%
   Req/Sec     7.21k     1.01k    9.27k    63.13%
  142124 requests in 10.03s, 322.69MB read
Requests/sec:  14167.59
Transfer/sec:     32.17MB

 

Centmin Mod LEMP wrk-cmm benchmarks revisited

Looking at raw wrk-cmm results I see bandwidth read on Centmin Mod Nginx tests being much higher than other LEMP stacks and remembered that Centmin Mod Nginx defaults to gzip compression level 5 by default in nginx.conf instead of gzip level 6 favouring scalability performance over page size. So re-testing wrk-cmm changing gzip_comp_level from 5 to 6

from

Code (Text):

gzip on;
gzip_vary   on;
gzip_disable "MSIE [1-6]\.";
gzip_static on;
gzip_min_length   1400;
gzip_buffers      1024 8k;
gzip_http_version 1.0;
gzip_comp_level 5;
gzip_proxied    any;
gzip_types text/plain text/css text/xml application/javascript application/x-javascript application/xml application/xml+rss application/ecmascript application/json image/svg+xml;

change to

Code (Text):

gzip on;
gzip_vary   on;
gzip_disable "MSIE [1-6]\.";
gzip_static on;
gzip_min_length   1400;
gzip_buffers      1024 8k;
gzip_http_version 1.0;
gzip_comp_level 6;
gzip_proxied    any;
gzip_types text/plain text/css text/xml application/javascript application/x-javascript application/xml application/xml+rss application/ecmascript application/json image/svg+xml;

gzip compression level 6 results do show lower numbers than the default gzip compression level 5 but still faster than other LEMP stacks' nginx servers.

Code (Text):

wrk-cmm -c10 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   700.18us  800.73us  28.68ms   96.84%
    Req/Sec     7.85k     1.80k   12.26k    66.00%
  156357 requests in 10.01s, 361.00MB read
Requests/sec:  15624.80
Transfer/sec:     36.08MB

wrk-cmm -c10 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   671.41us  640.87us  20.48ms   94.18%
    Req/Sec     8.17k     2.01k   13.01k    67.66%
  163247 requests in 10.10s, 376.91MB read
Requests/sec:  16162.93
Transfer/sec:     37.32MB

Code (Text):

wrk-cmm -c100 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     5.04ms    3.07ms  50.74ms   69.49%
    Req/Sec    10.29k     1.48k   14.72k    74.00%
  204648 requests in 10.01s, 472.50MB read
Requests/sec:  20451.21
Transfer/sec:     47.22MB

wrk-cmm -c100 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     5.37ms    3.66ms  73.29ms   72.43%
    Req/Sec     9.80k     1.65k   12.87k    65.00%
  195113 requests in 10.02s, 450.48MB read
Requests/sec:  19475.41
Transfer/sec:     44.97MB

Code (Text):

wrk-cmm -c500 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    27.47ms   28.21ms 933.45ms   97.80%
    Req/Sec     9.74k   560.51    11.53k    78.50%
  193746 requests in 10.03s, 447.33MB read
Requests/sec:  19313.56
Transfer/sec:     44.59MB

wrk-cmm -c500 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    26.39ms   11.03ms 434.33ms   90.55%
    Req/Sec     9.65k   427.54    10.66k    68.50%
  192149 requests in 10.04s, 443.64MB read
Requests/sec:  19147.21
Transfer/sec:     44.21MB

Code (Text):

wrk-cmm -c1000 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    50.36ms   12.67ms 115.02ms   78.98%
    Req/Sec     9.93k   660.59    11.27k    70.50%
  197698 requests in 10.08s, 456.45MB read
Requests/sec:  19616.53
Transfer/sec:     45.29MB

wrk-cmm -c1000 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    53.42ms   18.55ms 931.71ms   85.47%
    Req/Sec     9.41k   662.32    10.82k    63.00%
  187284 requests in 10.08s, 432.41MB read
Requests/sec:  18579.28
Transfer/sec:     42.90MB

 

Centmin Mod LEMP wrk-cmm benchmarks revisited gzip level 9

Change Centmin Mod Nginx gzip compression level from out of box defaul level 5 in nginx.conf to gzip level 9. Seems using Cloudflare gzip compression library with Nginx compiled binaries has different profiles for gzip compression levels. Might need to change Centmin Mod Nginx default out of the box gzip compression level from 5 to 9 Updated Centmin Mod 123.09beta01 when Cloudflare zlib library is used raise nginx default gzip compression level from 5 to 9

from

Code (Text):

gzip on;
gzip_vary   on;
gzip_disable "MSIE [1-6]\.";
gzip_static on;
gzip_min_length   1400;
gzip_buffers      1024 8k;
gzip_http_version 1.0;
gzip_comp_level 5;
gzip_proxied    any;
gzip_types text/plain text/css text/xml application/javascript application/x-javascript application/xml application/xml+rss application/ecmascript application/json image/svg+xml;

change to

Code (Text):

gzip on;
gzip_vary   on;
gzip_disable "MSIE [1-6]\.";
gzip_static on;
gzip_min_length   1400;
gzip_buffers      1024 8k;
gzip_http_version 1.0;
gzip_comp_level 9;
gzip_proxied    any;
gzip_types text/plain text/css text/xml application/javascript application/x-javascript application/xml application/xml+rss application/ecmascript application/json image/svg+xml;

gzip compression level 9 results show lower numbers than the default gzip compression level 5 but still faster than other LEMP stacks' nginx servers.

10 users

Code (Text):

wrk-cmm -c10 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   677.58us  622.17us  22.63ms   94.35%
    Req/Sec     7.88k     1.78k   12.66k    66.17%
  157659 requests in 10.10s, 363.71MB read
Requests/sec:  15610.82
Transfer/sec:     36.01MB

wrk-cmm -c10 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   671.06us  615.46us  17.04ms   93.00%
    Req/Sec     8.14k     1.92k   14.27k    71.64%
  162733 requests in 10.10s, 375.41MB read
Requests/sec:  16111.98
Transfer/sec:     37.17MB

100 users

Code (Text):

wrk-cmm -c100 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     5.60ms    3.32ms  63.35ms   66.38%
    Req/Sec     9.13k     1.25k   12.21k    75.00%
  181683 requests in 10.01s, 419.13MB read
Requests/sec:  18158.53
Transfer/sec:     41.89MB

wrk-cmm -c100 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     5.61ms    3.05ms  54.88ms   76.58%
    Req/Sec     9.22k     1.19k   13.30k    73.50%
  183494 requests in 10.00s, 423.31MB read
Requests/sec:  18344.99
Transfer/sec:     42.32MB

500 users

Code (Text):

wrk-cmm -c500 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    27.95ms    6.73ms 122.95ms   73.42%
    Req/Sec     8.98k   553.58     9.91k    79.50%
  178562 requests in 10.04s, 411.93MB read
Requests/sec:  17781.81
Transfer/sec:     41.02MB

wrk-cmm -c500 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    28.45ms   10.25ms 128.14ms   86.89%
    Req/Sec     8.88k   549.88     9.72k    78.50%
  176694 requests in 10.03s, 407.62MB read
Requests/sec:  17614.12
Transfer/sec:     40.63MB

1000 users

Code (Text):

wrk-cmm -c1000 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    61.58ms   80.66ms   1.97s    98.78%
    Req/Sec     9.09k   623.62    11.62k    77.50%
  181032 requests in 10.07s, 417.63MB read
  Socket errors: connect 0, read 0, write 0, timeout 7
Requests/sec:  17971.09
Transfer/sec:     41.46MB

wrk-cmm -c1000 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    60.49ms   54.15ms   1.90s    95.24%
    Req/Sec     8.73k   578.20    10.56k    74.00%
  173716 requests in 10.09s, 400.75MB read
  Socket errors: connect 0, read 0, write 0, timeout 14
Requests/sec:  17209.13
Transfer/sec:     39.70MB

 

Check Nginx Server Gzip Compression defaults

Let's check to see what are the gzip compression defaults for each respective LEMP stacks' nginx servers. Interesting that none of other LEMP stacks except Centmin Mod enables gzip_static directive which has a huge boost in performance if you pre-compress/gzip static files as seen in benchmarks here.

I also installed my custom curltest script which reports both the uncompressed and gzip or brotli compressed static file sizes from curl header checks.

Centmin Mod 123.09beta01

Centmin Mod 123.09beta01 CentOS 7.5 64bit Nginx = gzip compression level 5 (Updated as at June 11, 2018, Centmin Mod 123.09beta01 when Cloudflare zlib library is used raise nginx default gzip compression level from 5 to 9 )

Code (Text):

grep gzip /usr/local/nginx/conf/nginx.conf
        gzip on;
        gzip_vary   on;
        gzip_disable "MSIE [1-6]\.";
        gzip_static on;
        gzip_min_length   1400;
        gzip_buffers      1024 8k;
        gzip_http_version 1.0;
        gzip_comp_level 5;
        gzip_proxied    any;
        gzip_types text/plain text/css text/xml application/javascript application/x-javascript application/xml application/xml+rss application/ecmascript application/json image/svg+xml;

curltest gzip size check at previous default compression level 5

Code (Text):

curltest gzip http://http2.domain.com/index.html
URI: http://http2.domain.com/index.html (gzip)
Uncompressed size : 6.30 KiB
Compressed size   : 2.04 KiB

curltest gzip size check at new default compression level 9 - strangely the same ?

Code (Text):

curltest gzip http://http2.domain.com/index.html
URI: http://http2.domain.com/index.html (gzip)
Uncompressed size : 6.30 KiB
Compressed size   : 2.04 KiB

Easyengine 3.8.1

Easyengine 3.8.1 Ubuntu 16.04 LTS Nginx = gzip compression level 6

Code (Text):

grep gzip /etc/nginx/nginx.conf
        gzip on;
        gzip_disable "msie6";
        gzip_vary on;
        gzip_proxied any;
        gzip_comp_level 6;
        gzip_buffers 16 8k;
        gzip_http_version 1.1;
        gzip_types

curltest gzip size check at previous default compression level

Code (Text):

curltest gzip http://http2.domain.com/index.html
URI: http://http2.domain.com/index.html (gzip)
Uncompressed size : 6.30 KiB
Compressed size   : 2.05 KiB

Webinoly 1.4.3

Webinoly 1.4.3 Ubuntu 18.04 LTS Nginx = gzip compression level 6

Code (Text):

grep gzip /etc/nginx/nginx.conf
        gzip on;
        gzip_disable "msie6";
        gzip_vary on;
        gzip_proxied any;
        gzip_comp_level 6;
        gzip_buffers 16 8k;
        gzip_http_version 1.1;
        gzip_types application/atom+xml application/javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component text/xml text/javascript;

curltest gzip size check at previous default compression level

Code (Text):

curltest gzip http://http2.domain.com/index.html
URI: http://http2.domain.com/index.html (gzip)
Uncompressed size : 6.30 KiB
Compressed size   : 2.05 KiB

VestaCP 0.9.8-21

VestaCP 0.9.8-21 Ubuntu 16.04 LTS Nginx = gzip compression level 9

Code (Text):

grep gzip /etc/nginx/nginx.conf
    gzip                on;
    gzip_comp_level     9;
    gzip_min_length     512;
    gzip_buffers        8 64k;
    gzip_types          text/plain text/css text/javascript text/js text/xml application/json application/javascript application/x-javascript application/xml application/xml+rss application/x-font-ttf image/svg+xml font/opentype;
    gzip_proxied        any;
    gzip_disable        "MSIE [1-6]\.";

curltest gzip size check at previous default compression level

Code (Text):

curltest gzip http://http2.domain.com/index.html
URI: http://http2.domain.com/index.html (gzip)
Uncompressed size : 6.30 KiB
Compressed size   : 2.05 KiB

Next up, I'll be investigating Nginx memory and cpu usage under wrk-cmm load tests for various LEMP stacks

 

Yeah all that 123.09beta01 is waiting on is writing up and updating all official documentation on centminmod.com site when I have more time.

 

Yeah heard of those as well

 

OneInStack v1.7 LEMP Stack

OneInStack has options to choose standard Nginx source install, OpenResty Nginx or Tengine Nginx forks. I only managed to install std Nginx and OpenResty Nginx as Tengine Nginx failed to install. Ubuntu 16.04 LTS LXD container images were used.

Code (Text):

lxc list
+-------------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+
|             NAME              |  STATE  |         IPV4         |                     IPV6                      |    TYPE    | SNAPSHOTS |
+-------------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| centos75                      | STOPPED |                      |                                               | PERSISTENT | 1         |
+-------------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| centos75-2                    | RUNNING | 10.71.164.50 (eth0)  | fd42:769c:ebd9:a0f7:216:3eff:fe2e:a205 (eth0) | PERSISTENT | 1         |
+-------------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| easyengine-ubuntu1604         | RUNNING | 10.71.164.21 (eth0)  | fd42:769c:ebd9:a0f7:216:3eff:fe44:b2ee (eth0) | PERSISTENT | 0         |
+-------------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| oneinstack-ubuntu16-nginx     | RUNNING | 10.71.164.124 (eth0) | fd42:769c:ebd9:a0f7:216:3eff:fe4a:4878 (eth0) | PERSISTENT | 2         |
+-------------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| oneinstack-ubuntu16-openresty | RUNNING | 10.71.164.25 (eth0)  | fd42:769c:ebd9:a0f7:216:3eff:fe1f:b702 (eth0) | PERSISTENT | 1         |
+-------------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| oneinstack-ubuntu16-tengine   | RUNNING | 10.71.164.68 (eth0)  | fd42:769c:ebd9:a0f7:216:3eff:fe8c:4513 (eth0) | PERSISTENT | 0         |
+-------------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| vestacp-ubuntu1604            | RUNNING | 10.71.164.97 (eth0)  | fd42:769c:ebd9:a0f7:216:3eff:fef2:b4ae (eth0) | PERSISTENT | 1         |
+-------------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| webinoly                      | RUNNING | 10.71.164.45 (eth0)  | fd42:769c:ebd9:a0f7:216:3eff:fec7:d433 (eth0) | PERSISTENT | 0         |
+-------------------------------+---------+----------------------+-----------------------------------------------+------------+-----------+

Nginx LEMP stack install

Code (Text):

wget http://mirrors.linuxeye.com/oneinstack-full.tar.gz && tar xzf oneinstack-full.tar.gz && ./oneinstack/install.sh --nginx_option 1 --php_option 7 --phpcache_option 1 --php_extensions imagick --phpmyadmin  --db_option 5 --dbinstallmethod 1 --dbrootpwd oneinstack --pureftpd  --redis  --memcached  --iptables  --ssh_port 1029

OpenResty Nginx LEMP stack install

Code (Text):

wget http://mirrors.linuxeye.com/oneinstack-full.tar.gz && tar xzf oneinstack-full.tar.gz && ./oneinstack/install.sh --nginx_option 3 --php_option 7 --phpcache_option 1 --php_extensions imagick --phpmyadmin  --db_option 5 --dbinstallmethod 1 --dbrootpwd oneinstack --pureftpd  --redis  --memcached  --iptables  --ssh_port 1031

http2.domain.com nginx vhost creation via interactive vhost.sh script

Code (Text):

cd ~/oneinstack
./vhost.sh

OS

Code (Text):

cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.4 LTS"

OneInStack source compiles Nginx/PHP-FPM and MySQL can either be source compiled or direct binary install but doesn't use package manager to install. But OneInStack doesn't properly configure the source compiled binaries' environment PATH so simple commands like checking Nginx, PHP and MySQL versions fail and need to be directly called from the full path of the binary.

Code (Text):

echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin

Code (Text):

nginx -V
The program 'nginx' can be found in the following packages:
 * nginx-core
 * nginx-extras
 * nginx-full
 * nginx-light
Try: apt install <selected package>

Std Nginx version install has very minimal nginx modules enabled

OpenResty Nginx install had more modules and OpenResty 1.13.6.2 is based on Nginx 1.13.6 so around 6+ versions behind 1.13.12/1.14.0/1.15.0

Tengine failed to install

Code (Text):

installing libcrypto.a
installing libssl.a
cp libcrypto.pc /root/oneinstack/src/tengine-2.2.2/../openssl-1.0.2o/.openssl/lib/pkgconfig
chmod 644 /root/oneinstack/src/tengine-2.2.2/../openssl-1.0.2o/.openssl/lib/pkgconfig/libcrypto.pc
cp libssl.pc /root/oneinstack/src/tengine-2.2.2/../openssl-1.0.2o/.openssl/lib/pkgconfig
chmod 644 /root/oneinstack/src/tengine-2.2.2/../openssl-1.0.2o/.openssl/lib/pkgconfig/libssl.pc
cp openssl.pc /root/oneinstack/src/tengine-2.2.2/../openssl-1.0.2o/.openssl/lib/pkgconfig
chmod 644 /root/oneinstack/src/tengine-2.2.2/../openssl-1.0.2o/.openssl/lib/pkgconfig/openssl.pc
make[2]: Leaving directory '/root/oneinstack/src/openssl-1.0.2o'
objs/Makefile:1411: recipe for target '../openssl-1.0.2o/.openssl/include/openssl/ssl.h' failed
make[1]: *** [../openssl-1.0.2o/.openssl/include/openssl/ssl.h] Error 2
make[1]: Leaving directory '/root/oneinstack/src/tengine-2.2.2'
Makefile:8: recipe for target 'build' failed
make: *** [build] Error 2
Tengine install failed, Please Contact the author!
Killed
root@oneinstack-ubuntu16-tengine:~# /bin/cp: cannot stat '../init.d/nginx.service': No such file or directory
sed: can't read /lib/systemd/system/nginx.service: No such file or directory
Failed to execute operation: No such file or directory
mv: cannot stat '/usr/local/tengine/conf/nginx.conf': No such file or directory
/bin/cp: cannot stat '../config/nginx.conf': No such file or directory
grep: /usr/local/tengine/conf/nginx.conf: No such file or directory
sed: can't read /usr/local/tengine/conf/nginx.conf: No such file or directory
include/tengine.sh: line 63: /usr/local/tengine/conf/proxy.conf: No such file or directory
sed: can't read /usr/local/tengine/conf/nginx.conf: No such file or directory
sed: can't read /usr/local/tengine/conf/nginx.conf: No such file or directory
sed: can't read /usr/local/tengine/conf/nginx.conf: No such file or directory
grep: /usr/local/tengine/conf/nginx.conf: No such file or directory
sed: can't read /usr/local/tengine/conf/nginx.conf: No such file or directory
sed: can't read /usr/local/tengine/conf/nginx.conf: No such file or directory
~/oneinstack/src ~/oneinstack ~
Failed to start nginx.service: Unit nginx.service not found.

Due to number of entries to compare, had to change to a bar chart



Previous charts used Google Sheets and was limited in formatting options, so downloaded to local Excel to better format the charts

OneInStack Nginx wrk-cmm results

default gzip level 6

Code (Text):

grep -rn gzip /usr/local/nginx/
/usr/local/nginx/conf/nginx.conf_bk:33:    #gzip  on;
/usr/local/nginx/conf/nginx.conf.default:33:    #gzip  on;
/usr/local/nginx/conf/nginx.conf:38:  gzip on;
/usr/local/nginx/conf/nginx.conf:39:  gzip_buffers 16 8k;
/usr/local/nginx/conf/nginx.conf:40:  gzip_comp_level 6;
/usr/local/nginx/conf/nginx.conf:41:  gzip_http_version 1.1;
/usr/local/nginx/conf/nginx.conf:42:  gzip_min_length 256;
/usr/local/nginx/conf/nginx.conf:43:  gzip_proxied any;
/usr/local/nginx/conf/nginx.conf:44:  gzip_vary on;
/usr/local/nginx/conf/nginx.conf:45:  gzip_types
/usr/local/nginx/conf/nginx.conf:52:  gzip_disable "MSIE [1-6]\.(?!.*SV1)";

Code (Text):

curl -I http://http2.domain.com/index.html
HTTP/1.1 200 OK
Server: nginx
Date: Mon, 11 Jun 2018 21:27:38 GMT
Content-Type: text/html
Content-Length: 6456
Last-Modified: Mon, 11 Jun 2018 21:27:15 GMT
Connection: keep-alive
Vary: Accept-Encoding
ETag: "5b1ee933-1938"
Accept-Ranges: bytes

10 users

Code (Text):

wrk-cmm -c10 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency   748.29us    0.90ms  22.17ms   97.63%
   Req/Sec     7.45k     1.03k    9.37k    78.00%
  148297 requests in 10.00s, 337.44MB read
Requests/sec:  14826.92
Transfer/sec:     33.74MB

wrk-cmm -c10 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency   652.94us  472.53us  17.55ms   96.96%
   Req/Sec     7.93k     0.87k   13.99k    85.07%
  158600 requests in 10.10s, 360.88MB read
Requests/sec:  15703.36
Transfer/sec:     35.73MB

100 users

Code (Text):

wrk-cmm -c100 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency     6.57ms    2.06ms  25.39ms   74.19%
   Req/Sec     7.65k   450.92     9.05k    73.00%
  152184 requests in 10.02s, 346.28MB read
Requests/sec:  15184.00
Transfer/sec:     34.55MB

wrk-cmm -c100 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency     6.46ms    1.96ms  45.59ms   79.86%
   Req/Sec     7.79k   670.19    10.44k    84.50%
  155162 requests in 10.02s, 353.06MB read
Requests/sec:  15487.64
Transfer/sec:     35.24MB

500 users

Code (Text):

wrk-cmm -c500 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency    33.71ms   14.37ms  79.24ms   66.65%
   Req/Sec     7.44k   637.25     8.80k    66.50%
  148109 requests in 10.09s, 337.01MB read
Requests/sec:  14684.28
Transfer/sec:     33.41MB

wrk-cmm -c500 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency    35.47ms   23.71ms 104.68ms   67.11%
   Req/Sec     7.47k     1.27k   10.43k    62.50%
  148623 requests in 10.06s, 338.18MB read
Requests/sec:  14775.35
Transfer/sec:     33.62MB

1000 users

Code (Text):

wrk-cmm -c1000 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency    69.20ms   29.69ms 260.90ms   76.29%
   Req/Sec     7.33k     1.04k    9.93k    72.73%
  144474 requests in 10.01s, 328.74MB read
Requests/sec:  14437.16
Transfer/sec:     32.85MB

wrk-cmm -c1000 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency    69.12ms   54.16ms 188.40ms   46.43%
   Req/Sec     7.49k   582.23     8.96k    63.64%
  147491 requests in 10.01s, 335.61MB read
Requests/sec:  14736.01
Transfer/sec:     33.53MB

OneInStack OpenResty Nginx wrk-cmm results

default gzip level 6

Code (Text):

grep -rn gzip /usr/local/openresty/nginx
/usr/local/openresty/nginx/conf/nginx.conf_bk:33:    #gzip  on;
/usr/local/openresty/nginx/conf/nginx.conf.default:33:    #gzip  on;
/usr/local/openresty/nginx/conf/nginx.conf:38:  gzip on;
/usr/local/openresty/nginx/conf/nginx.conf:39:  gzip_buffers 16 8k;
/usr/local/openresty/nginx/conf/nginx.conf:40:  gzip_comp_level 6;
/usr/local/openresty/nginx/conf/nginx.conf:41:  gzip_http_version 1.1;
/usr/local/openresty/nginx/conf/nginx.conf:42:  gzip_min_length 256;
/usr/local/openresty/nginx/conf/nginx.conf:43:  gzip_proxied any;
/usr/local/openresty/nginx/conf/nginx.conf:44:  gzip_vary on;
/usr/local/openresty/nginx/conf/nginx.conf:45:  gzip_types
/usr/local/openresty/nginx/conf/nginx.conf:52:  gzip_disable "MSIE [1-6]\.(?!.*SV1)";

Code (Text):

curl -I http://http2.domain.com/index.html
HTTP/1.1 200 OK
Server: openresty
Date: Mon, 11 Jun 2018 22:29:36 GMT
Content-Type: text/html
Content-Length: 6456
Last-Modified: Mon, 11 Jun 2018 22:28:32 GMT
Connection: keep-alive
Vary: Accept-Encoding
ETag: "5b1ef790-1938"
Accept-Ranges: bytes

10 users

Code (Text):

wrk-cmm -c10 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency   691.51us  655.96us  28.64ms   98.30%
   Req/Sec     7.61k   837.48     8.88k    68.50%
  151541 requests in 10.00s, 345.40MB read
Requests/sec:  15150.82
Transfer/sec:     34.53MB

wrk-cmm -c10 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency   703.18us  573.20us  14.74ms   97.35%
   Req/Sec     7.52k   697.87     9.24k    71.50%
  149685 requests in 10.00s, 341.17MB read
Requests/sec:  14967.97
Transfer/sec:     34.12MB

100 users

Code (Text):

wrk-cmm -c100 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency     6.42ms    2.75ms  52.11ms   81.05%
   Req/Sec     7.91k   616.68     9.65k    80.00%
  157473 requests in 10.04s, 358.92MB read
Requests/sec:  15686.12
Transfer/sec:     35.75MB

wrk-cmm -c100 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency     6.79ms    3.68ms  46.66ms   88.34%
   Req/Sec     7.75k     1.27k   10.52k    82.50%
  154171 requests in 10.02s, 351.39MB read
Requests/sec:  15391.27
Transfer/sec:     35.08MB

500 users

Code (Text):

wrk-cmm -c500 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency    42.27ms   41.94ms 165.91ms   75.28%
   Req/Sec     7.72k   842.48     9.81k    68.50%
  153688 requests in 10.07s, 350.29MB read
Requests/sec:  15259.57
Transfer/sec:     34.78MB

wrk-cmm -c500 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency    37.12ms   17.53ms 116.51ms   75.03%
   Req/Sec     6.86k     1.41k    9.20k    72.50%
  136460 requests in 10.08s, 311.03MB read
Requests/sec:  13544.16
Transfer/sec:     30.87MB

1000 users

Code (Text):

wrk-cmm -c1000 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency    83.52ms   79.08ms 420.66ms   75.16%
   Req/Sec     7.48k     1.76k   11.37k    69.85%
  148076 requests in 10.10s, 337.50MB read
Requests/sec:  14667.30
Transfer/sec:     33.43MB

wrk-cmm -c1000 -d10s -H "Accept-Encoding: gzip" http://http2.domain.com/
Running 10s test @ http://http2.domain.com/
  2 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency    73.11ms   41.81ms 308.92ms   67.35%
   Req/Sec     7.07k     1.46k    9.94k    73.23%
  139339 requests in 10.02s, 317.59MB read
Requests/sec:  13906.47
Transfer/sec:     31.70MB

 

More condensed summary so far by only focusing on wrk-cmm 500 and 1000 user load tests for Centmin Mod vs Easyengine vs Webinoly vs VestaCP VS OneInStack