Sysadmin Discuss Nginx & PHP-FPM log rotation compression using zstd vs gzip

This thread is dedicated to dicussing folks switching their Centmin Mod Nginx & PHP-FPM log rotation routines from using gzip for compression to using Facebook's zstd compression for smaller compressed log sizes as discussed here.

To switch from gzip to zstd compression for logrotate, you first need to have zstd installed. Centmin Mod 123.09beta01 and higher users can install zstd via centmin.sh menu option 17 to install all multi-threaded compression tools.

Code (Text):

--------------------------------------------------------
    Centmin Mod Menu 123.09beta01 centminmod.com
--------------------------------------------------------
1).  Centmin Install
2).  Add Nginx vhost domain
3).  NSD setup domain name DNS
4).  Nginx Upgrade / Downgrade
5).  PHP Upgrade / Downgrade
6).  XCache Re-install
7).  APC Cache Re-install
8).  XCache Install
9).  APC Cache Install
10). Memcached Server Re-install
11). MariaDB MySQL Upgrade & Management
12). Zend OpCache Install/Re-install
13). Install/Reinstall Redis PHP Extension
14). SELinux disable
15). Install/Reinstall ImagicK PHP Extension
16). Change SSHD Port Number
17). Multi-thread compression: pigz,pbzip2,lbzip2...
18). Suhosin PHP Extension install
19). Install FFMPEG and FFMPEG PHP Extension
20). NSD Install/Re-Install
21). Update - Nginx + PHP-FPM + Siege
22). Add Wordpress Nginx vhost + Cache Plugin
23). Update Centmin Mod Code Base
24). Exit
--------------------------------------------------------
Enter option [ 1 - 24 ] 17

Code (Text):

zstd -V
*** zstd command line interface 64-bits v1.3.8, by Yann Collet ***

---

Modify Log Rotation To Use Zstd Compression

Then you need to edit your logrotate profiles to add the settings right after the delaycompress setting

Code (Text):

compresscmd /usr/local/bin/zstd
uncompresscmd /usr/local/bin/unzstd
compressoptions -9 --long -T0
compressext .zst

• Compression level 9 is used for zstd which produces smaller files than gzip level 6 default and level 9 max. But if you want even smaller compressed logs at equivalent gzip compression speed, you can change level 9 (-9) to level 12 (-12).
• Also zstd use --long for long range mode for better compression ratios as outlined here. If you have more than 2GB of free usable memory you can also further reduce compressed log file size by change --long to --long=31 to allocate 2GB window for zstd compression to work with. By default --long uses 128MB window size.
• -T0 tells zstd to use multi-threaded compression value equal to number of cpu cores available.

For example my nginx logrotation profile /etc/logrotate.d/nginx becomes

Code (Text):

/var/log/nginx/*.log /usr/local/nginx/logs/*.log /home/nginx/domains/*/log/*.log {
        daily
        dateext
        missingok
        rotate 10
        maxsize 500M
        compress
        delaycompress
        compresscmd /usr/local/bin/zstd
        uncompresscmd /usr/local/bin/unzstd
        compressoptions -9 --long -T0
        compressext .zst
        notifempty
        postrotate
        /bin/kill -SIGUSR1 $(cat /usr/local/nginx/logs/nginx.pid 2>/dev/null) 2>/dev/null || true
        endscript
}

Then you can test logrotate via debug mode to see what it would do for nginx logrotate profile - it's only debug test where no actual logrotation is done yet.

Code (Text):

logrotate -df /etc/logrotate.d/nginx

starting few lines will pick up the compression program to use which is zstd at level 9 compression with .zst extension for compressed log files

Code (Text):

logrotate -df /etc/logrotate.d/nginx
reading config file /etc/logrotate.d/nginx
compress_prog is now /usr/local/bin/zstd
uncompress_prog is now /usr/local/bin/unzstd
compress_options is now  -9 --long -T0
compress_ext is now .zst
Allocating hash table for state file, size 15360 B

Handling 1 logs

rotating pattern: /var/log/nginx/*.log /usr/local/nginx/logs/*.log /home/nginx/domains/*/log/*.log  forced from command line (10 rotations)
empty log files are not rotated, log files >= 524288000 are rotated earlier, old logs are removed
considering log /var/log/nginx/cfcomp-access.log
  log needs rotating
considering log /var/log/nginx/localhost.access.log
  log needs rotating
considering log /var/log/nginx/localhost.error.log
  log needs rotating
considering log /usr/local/nginx/logs/access.log
  log does not need rotating (log is empty)considering log /usr/local/nginx/logs/error.log
  log does not need rotating (log is empty)considering log /home/nginx/domains/cpdomain.com/log/access.log
  log does not need rotating (log is empty)considering log /home/nginx/domains/cpdomain.com/log/error.log
  log does not need rotating (log is empty)considering log /home/nginx/domains/demodomain.com/log/access.log
  log does not need rotating (log is empty)considering log /home/nginx/domains/demodomain.com/log/error.log
  log does not need rotating (log is empty)considering log /home/nginx/domains/domain9.com/log/access.log
  log does not need rotating (log is empty)considering log /home/nginx/domains/domain9.com/log/error.log
  log does not need rotating (log is empty)considering log

Now if you want to really test and force rotate the logs run without -d debug flag

Code (Text):

logrotate -fv /etc/logrotate.d/nginx

Checking /var/log/nginx for Nginx main hostname logs, you'll see most recent rotation with .zst extension for zstd compressed files

Code (Text):

ls -lah /var/log/nginx
total 1.8M
drwxr-xr-x.  2 root  root 4.0K Jan  1 07:09 .
drwxr-xr-x. 16 root  root 4.0K Jan  1 03:29 ..
-rw-r--r--   1 nginx root 179K Jan  1 07:09 cfcomp-access.log
-rw-r--r--   1 nginx root  74K Dec 23 03:31 cfcomp-access.log-20181223.gz
-rw-r--r--   1 nginx root 135K Dec 24 03:36 cfcomp-access.log-20181224.gz
-rw-r--r--   1 nginx root  75K Dec 25 03:47 cfcomp-access.log-20181225.gz
-rw-r--r--   1 nginx root  85K Dec 26 03:20 cfcomp-access.log-20181226.gz
-rw-r--r--   1 nginx root  59K Dec 27 03:40 cfcomp-access.log-20181227.gz
-rw-r--r--   1 nginx root  74K Dec 28 03:18 cfcomp-access.log-20181228.gz
-rw-r--r--   1 nginx root  60K Dec 29 03:31 cfcomp-access.log-20181229.gz
-rw-r--r--   1 nginx root  76K Dec 30 03:42 cfcomp-access.log-20181230.gz
-rw-r--r--   1 nginx root  83K Dec 31 03:48 cfcomp-access.log-20181231.gz
-rw-r--r--   1 nginx root  72K Jan  1 03:29 cfcomp-access.log-20190101.zst
-rw-rw----   1 nginx root 180K Jan  1 07:09 localhost.access.log
-rw-rw----   1 nginx root  47K Dec 23 03:31 localhost.access.log-20181223.gz
-rw-rw----   1 nginx root  83K Dec 24 03:36 localhost.access.log-20181224.gz
-rw-rw----   1 nginx root  47K Dec 25 03:47 localhost.access.log-20181225.gz
-rw-rw----   1 nginx root  54K Dec 26 03:20 localhost.access.log-20181226.gz
-rw-rw----   1 nginx root  38K Dec 27 03:40 localhost.access.log-20181227.gz
-rw-rw----   1 nginx root  47K Dec 28 03:18 localhost.access.log-20181228.gz
-rw-rw----   1 nginx root  39K Dec 29 03:31 localhost.access.log-20181229.gz
-rw-rw----   1 nginx root  47K Dec 30 03:42 localhost.access.log-20181230.gz
-rw-rw----   1 nginx root  52K Dec 31 03:48 localhost.access.log-20181231.gz
-rw-rw----   1 nginx root  46K Jan  1 03:29 localhost.access.log-20190101.zst
-rw-rw----   1 nginx root  962 Jan  1 07:08 localhost.error.log
-rw-rw----   1 nginx root 1.5K Dec 23 02:27 localhost.error.log-20181223.gz
-rw-rw----   1 nginx root 5.8K Dec 24 03:34 localhost.error.log-20181224.gz
-rw-rw----   1 nginx root 1.8K Dec 25 01:33 localhost.error.log-20181225.gz
-rw-rw----   1 nginx root 1.2K Dec 26 02:21 localhost.error.log-20181226.gz
-rw-rw----   1 nginx root 1.6K Dec 26 22:20 localhost.error.log-20181227.gz
-rw-rw----   1 nginx root 1.3K Dec 28 02:34 localhost.error.log-20181228.gz
-rw-rw----   1 nginx root 1.8K Dec 28 23:33 localhost.error.log-20181229.gz
-rw-rw----   1 nginx root 1.8K Dec 30 02:52 localhost.error.log-20181230.gz
-rw-rw----   1 nginx root 1.7K Dec 31 01:53 localhost.error.log-20181231.gz
-rw-rw----   1 nginx root 2.5K Jan  1 03:19 localhost.error.log-20190101.zst

same with domain9.com's logs

Code (Text):

ls -lah /home/nginx/domains/domain9.com/log
total 160K
drwxr-s--- 2 nginx nginx 4.0K Jan  1 07:09 .
drwxr-s--- 6 nginx nginx 4.0K May 29  2017 ..
-rw-r--r-- 1 nginx nginx  60K Jan  1 06:41 access.log
-rw-r--r-- 1 nginx nginx 6.5K Dec 23 03:31 access.log-20181223.gz
-rw-r--r-- 1 nginx nginx 2.3K Dec 24 03:36 access.log-20181224.gz
-rw-r--r-- 1 nginx nginx 2.3K Dec 25 03:47 access.log-20181225.gz
-rw-r--r-- 1 nginx nginx 1.5K Dec 26 03:16 access.log-20181226.gz
-rw-r--r-- 1 nginx nginx 3.7K Dec 27 03:40 access.log-20181227.gz
-rw-r--r-- 1 nginx nginx 3.5K Dec 28 03:18 access.log-20181228.gz
-rw-r--r-- 1 nginx nginx 3.8K Dec 29 02:46 access.log-20181229.gz
-rw-r--r-- 1 nginx nginx 1.6K Dec 30 03:42 access.log-20181230.gz
-rw-r--r-- 1 nginx nginx  990 Dec 31 02:11 access.log-20181231.gz
-rw-r--r-- 1 nginx nginx 4.9K Jan  1 03:29 access.log-20190101.zst
-rw-r--r-- 1 nginx nginx  673 Jan  1 04:39 error.log
-rw-r--r-- 1 nginx nginx  258 Dec 21 21:52 error.log-20181222.gz
-rw-r--r-- 1 nginx nginx  797 Dec 22 13:59 error.log-20181223.gz
-rw-r--r-- 1 nginx nginx 1000 Dec 23 21:24 error.log-20181224.gz
-rw-r--r-- 1 nginx nginx  304 Dec 24 12:25 error.log-20181225.gz
-rw-r--r-- 1 nginx nginx  434 Dec 27 02:56 error.log-20181227.gz
-rw-r--r-- 1 nginx nginx  382 Dec 28 02:22 error.log-20181228.gz
-rw-r--r-- 1 nginx nginx  382 Dec 28 21:06 error.log-20181229.gz
-rw-r--r-- 1 nginx nginx  376 Dec 29 12:38 error.log-20181230.gz
-rw-r--r-- 1 nginx nginx  416 Dec 30 22:58 error.log-20181231.gz
-rw-r--r-- 1 nginx nginx  637 Dec 31 15:00 error.log-20190101.zst

For PHP-FPM similar steps, change the logrotation profile /etc/logrotate.d/php-fpm

from

Code (Text):

/var/log/php-fpm/*.log {
        daily
        dateext
        missingok
        rotate 10
        maxsize 500M
        compress
        delaycompress
        notifempty
        postrotate
        /bin/kill -SIGUSR1 $(cat /var/run/php-fpm/php-fpm.pid 2>/dev/null) 2>/dev/null || true
        endscript           
}

to

Code (Text):

/var/log/php-fpm/*.log {
        daily
        dateext
        missingok
        rotate 10
        maxsize 500M
        compress
        delaycompress
        compresscmd /usr/local/bin/zstd
        uncompresscmd /usr/local/bin/unzstd
        compressoptions -9 --long -T0
        compressext .zst
        notifempty
        postrotate
        /bin/kill -SIGUSR1 $(cat /var/run/php-fpm/php-fpm.pid 2>/dev/null) 2>/dev/null || true
        endscript           
}

Test debug

Code (Text):

logrotate -df /etc/logrotate.d/php-fpm

For actual logrotation

Code (Text):

logrotate -fv /etc/logrotate.d/php-fpm

For inspecting compressed .gz or .zst logs, see instructions for using zcat/zgrep, pzcat/pzgrep and zstdcat/zstdgrep commands here.

 

Actually zstd uses 1 cpu thread by default so -T0 does number of physical cpu cores available

Code (Text):

 -T#    : spawns # compression threads (default: 1, 0==# cores) 

from help

Code (Text):

zstd --help
*** zstd command line interface 64-bits v1.3.8, by Yann Collet ***
Usage : 
      zstd [args] [FILE(s)] [-o file] 

FILE    : a filename 
          with no FILE, or when FILE is - , read standard input
Arguments : 
 -#     : # compression level (1-19, default: 3) 
 -d     : decompression 
 -D file: use `file` as Dictionary 
 -o file: result stored into `file` (only if 1 input file) 
 -f     : overwrite output without prompting and (de)compress links 
--rm    : remove source file(s) after successful de/compression 
 -k     : preserve source file(s) (default) 
 -h/-H  : display help/long help and exit 

Advanced arguments : 
 -V     : display Version number and exit 
 -v     : verbose mode; specify multiple times to increase verbosity
 -q     : suppress warnings; specify twice to suppress errors too
 -c     : force write to standard output, even if it is the console
 -l     : print information about zstd compressed files 
--ultra : enable levels beyond 19, up to 22 (requires more memory)
--long[=#]: enable long distance matching with given window log (default: 27)
--fast[=#]: switch to ultra fast compression level (default: 1)
--adapt : dynamically adapt compression level to I/O conditions 
 -T#    : spawns # compression threads (default: 1, 0==# cores) 
 -B#    : select size of each job (default: 0==automatic) 
 --rsyncable : compress using a rsync-friendly method (-B sets block size) 
--no-dictID : don't write dictID into header (dictionary compression)
--[no-]check : integrity check (default: enabled) 
 -r     : operate recursively on directories 
--format=zstd : compress files to the .zst format (default) 
--test  : test compressed file integrity 
--[no-]sparse : sparse mode (default: enabled on file, disabled on stdout)
 -M#    : Set a memory usage limit for decompression 
--no-progress : do not display the progress bar 
--      : All arguments after "--" are treated as files 

Dictionary builder : 
--train ## : create a dictionary from a training set of files 
--train-cover[=k=#,d=#,steps=#,split=#] : use the cover algorithm with optional args
--train-fastcover[=k=#,d=#,f=#,steps=#,split=#,accel=#] : use the fast cover algorithm with optional args
--train-legacy[=s=#] : use the legacy algorithm with selectivity (default: 9)
 -o file : `file` is dictionary name (default: dictionary) 
--maxdict=# : limit dictionary to specified size (default: 112640) 
--dictID=# : force dictionary ID to specified value (default: random)

Benchmark arguments : 
 -b#    : benchmark file(s), using # compression level (default: 3) 
 -e#    : test all compression levels from -bX to # (default: 1)
 -i#    : minimum evaluation time in seconds (default: 3s) 
 -B#    : cut file into independent blocks of size # (default: no block)
--priority=rt : set process priority to real-time