Join the community today
Register Now

Sysadmin Discuss Nginx & PHP-FPM log rotation compression using zstd vs gzip

Discussion in 'System Administration' started by eva2000, Jan 4, 2019.

  1. eva2000

    eva2000 Administrator Staff Member

    41,262
    9,259
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,202
    Local Time:
    3:39 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    This thread is dedicated to dicussing folks switching their Centmin Mod Nginx & PHP-FPM log rotation routines from using gzip for compression to using Facebook's zstd compression for smaller compressed log sizes as discussed here.

    To switch from gzip to zstd compression for logrotate, you first need to have zstd installed. Centmin Mod 123.09beta01 and higher users can install zstd via centmin.sh menu option 17 to install all multi-threaded compression tools.
    Code (Text):
    --------------------------------------------------------
        Centmin Mod Menu 123.09beta01 centminmod.com
    --------------------------------------------------------
    1).  Centmin Install
    2).  Add Nginx vhost domain
    3).  NSD setup domain name DNS
    4).  Nginx Upgrade / Downgrade
    5).  PHP Upgrade / Downgrade
    6).  XCache Re-install
    7).  APC Cache Re-install
    8).  XCache Install
    9).  APC Cache Install
    10). Memcached Server Re-install
    11). MariaDB MySQL Upgrade & Management
    12). Zend OpCache Install/Re-install
    13). Install/Reinstall Redis PHP Extension
    14). SELinux disable
    15). Install/Reinstall ImagicK PHP Extension
    16). Change SSHD Port Number
    17). Multi-thread compression: pigz,pbzip2,lbzip2...
    18). Suhosin PHP Extension install
    19). Install FFMPEG and FFMPEG PHP Extension
    20). NSD Install/Re-Install
    21). Update - Nginx + PHP-FPM + Siege
    22). Add Wordpress Nginx vhost + Cache Plugin
    23). Update Centmin Mod Code Base
    24). Exit
    --------------------------------------------------------
    Enter option [ 1 - 24 ] 17
    

    Code (Text):
    zstd -V
    *** zstd command line interface 64-bits v1.3.8, by Yann Collet ***
    


    Modify Log Rotation To Use Zstd Compression



    Then you need to edit your logrotate profiles to add the settings right after the delaycompress setting
    Code (Text):
    compresscmd /usr/local/bin/zstd
    uncompresscmd /usr/local/bin/unzstd
    compressoptions -9 --long -T0
    compressext .zst
    

    • Compression level 9 is used for zstd which produces smaller files than gzip level 6 default and level 9 max. But if you want even smaller compressed logs at equivalent gzip compression speed, you can change level 9 (-9) to level 12 (-12).
    • Also zstd use --long for long range mode for better compression ratios as outlined here. If you have more than 2GB of free usable memory you can also further reduce compressed log file size by change --long to --long=31 to allocate 2GB window for zstd compression to work with. By default --long uses 128MB window size.
    • -T0 tells zstd to use multi-threaded compression value equal to number of cpu cores available.
    For example my nginx logrotation profile /etc/logrotate.d/nginx becomes
    Code (Text):
    /var/log/nginx/*.log /usr/local/nginx/logs/*.log /home/nginx/domains/*/log/*.log {
            daily
            dateext
            missingok
            rotate 10
            maxsize 500M
            compress
            delaycompress
            compresscmd /usr/local/bin/zstd
            uncompresscmd /usr/local/bin/unzstd
            compressoptions -9 --long -T0
            compressext .zst
            notifempty
            postrotate
            /bin/kill -SIGUSR1 $(cat /usr/local/nginx/logs/nginx.pid 2>/dev/null) 2>/dev/null || true
            endscript
    }
    

    Then you can test logrotate via debug mode to see what it would do for nginx logrotate profile - it's only debug test where no actual logrotation is done yet.
    Code (Text):
    logrotate -df /etc/logrotate.d/nginx
    

    starting few lines will pick up the compression program to use which is zstd at level 9 compression with .zst extension for compressed log files
    Code (Text):
    logrotate -df /etc/logrotate.d/nginx
    reading config file /etc/logrotate.d/nginx
    compress_prog is now /usr/local/bin/zstd
    uncompress_prog is now /usr/local/bin/unzstd
    compress_options is now  -9 --long -T0
    compress_ext is now .zst
    Allocating hash table for state file, size 15360 B
    
    Handling 1 logs
    
    rotating pattern: /var/log/nginx/*.log /usr/local/nginx/logs/*.log /home/nginx/domains/*/log/*.log  forced from command line (10 rotations)
    empty log files are not rotated, log files >= 524288000 are rotated earlier, old logs are removed
    considering log /var/log/nginx/cfcomp-access.log
      log needs rotating
    considering log /var/log/nginx/localhost.access.log
      log needs rotating
    considering log /var/log/nginx/localhost.error.log
      log needs rotating
    considering log /usr/local/nginx/logs/access.log
      log does not need rotating (log is empty)considering log /usr/local/nginx/logs/error.log
      log does not need rotating (log is empty)considering log /home/nginx/domains/cpdomain.com/log/access.log
      log does not need rotating (log is empty)considering log /home/nginx/domains/cpdomain.com/log/error.log
      log does not need rotating (log is empty)considering log /home/nginx/domains/demodomain.com/log/access.log
      log does not need rotating (log is empty)considering log /home/nginx/domains/demodomain.com/log/error.log
      log does not need rotating (log is empty)considering log /home/nginx/domains/domain9.com/log/access.log
      log does not need rotating (log is empty)considering log /home/nginx/domains/domain9.com/log/error.log
      log does not need rotating (log is empty)considering log
    

    Now if you want to really test and force rotate the logs run without -d debug flag
    Code (Text):
    logrotate -fv /etc/logrotate.d/nginx
    

    Checking /var/log/nginx for Nginx main hostname logs, you'll see most recent rotation with .zst extension for zstd compressed files
    Code (Text):
    ls -lah /var/log/nginx
    total 1.8M
    drwxr-xr-x.  2 root  root 4.0K Jan  1 07:09 .
    drwxr-xr-x. 16 root  root 4.0K Jan  1 03:29 ..
    -rw-r--r--   1 nginx root 179K Jan  1 07:09 cfcomp-access.log
    -rw-r--r--   1 nginx root  74K Dec 23 03:31 cfcomp-access.log-20181223.gz
    -rw-r--r--   1 nginx root 135K Dec 24 03:36 cfcomp-access.log-20181224.gz
    -rw-r--r--   1 nginx root  75K Dec 25 03:47 cfcomp-access.log-20181225.gz
    -rw-r--r--   1 nginx root  85K Dec 26 03:20 cfcomp-access.log-20181226.gz
    -rw-r--r--   1 nginx root  59K Dec 27 03:40 cfcomp-access.log-20181227.gz
    -rw-r--r--   1 nginx root  74K Dec 28 03:18 cfcomp-access.log-20181228.gz
    -rw-r--r--   1 nginx root  60K Dec 29 03:31 cfcomp-access.log-20181229.gz
    -rw-r--r--   1 nginx root  76K Dec 30 03:42 cfcomp-access.log-20181230.gz
    -rw-r--r--   1 nginx root  83K Dec 31 03:48 cfcomp-access.log-20181231.gz
    -rw-r--r--   1 nginx root  72K Jan  1 03:29 cfcomp-access.log-20190101.zst
    -rw-rw----   1 nginx root 180K Jan  1 07:09 localhost.access.log
    -rw-rw----   1 nginx root  47K Dec 23 03:31 localhost.access.log-20181223.gz
    -rw-rw----   1 nginx root  83K Dec 24 03:36 localhost.access.log-20181224.gz
    -rw-rw----   1 nginx root  47K Dec 25 03:47 localhost.access.log-20181225.gz
    -rw-rw----   1 nginx root  54K Dec 26 03:20 localhost.access.log-20181226.gz
    -rw-rw----   1 nginx root  38K Dec 27 03:40 localhost.access.log-20181227.gz
    -rw-rw----   1 nginx root  47K Dec 28 03:18 localhost.access.log-20181228.gz
    -rw-rw----   1 nginx root  39K Dec 29 03:31 localhost.access.log-20181229.gz
    -rw-rw----   1 nginx root  47K Dec 30 03:42 localhost.access.log-20181230.gz
    -rw-rw----   1 nginx root  52K Dec 31 03:48 localhost.access.log-20181231.gz
    -rw-rw----   1 nginx root  46K Jan  1 03:29 localhost.access.log-20190101.zst
    -rw-rw----   1 nginx root  962 Jan  1 07:08 localhost.error.log
    -rw-rw----   1 nginx root 1.5K Dec 23 02:27 localhost.error.log-20181223.gz
    -rw-rw----   1 nginx root 5.8K Dec 24 03:34 localhost.error.log-20181224.gz
    -rw-rw----   1 nginx root 1.8K Dec 25 01:33 localhost.error.log-20181225.gz
    -rw-rw----   1 nginx root 1.2K Dec 26 02:21 localhost.error.log-20181226.gz
    -rw-rw----   1 nginx root 1.6K Dec 26 22:20 localhost.error.log-20181227.gz
    -rw-rw----   1 nginx root 1.3K Dec 28 02:34 localhost.error.log-20181228.gz
    -rw-rw----   1 nginx root 1.8K Dec 28 23:33 localhost.error.log-20181229.gz
    -rw-rw----   1 nginx root 1.8K Dec 30 02:52 localhost.error.log-20181230.gz
    -rw-rw----   1 nginx root 1.7K Dec 31 01:53 localhost.error.log-20181231.gz
    -rw-rw----   1 nginx root 2.5K Jan  1 03:19 localhost.error.log-20190101.zst
    

    same with domain9.com's logs
    Code (Text):
    ls -lah /home/nginx/domains/domain9.com/log
    total 160K
    drwxr-s--- 2 nginx nginx 4.0K Jan  1 07:09 .
    drwxr-s--- 6 nginx nginx 4.0K May 29  2017 ..
    -rw-r--r-- 1 nginx nginx  60K Jan  1 06:41 access.log
    -rw-r--r-- 1 nginx nginx 6.5K Dec 23 03:31 access.log-20181223.gz
    -rw-r--r-- 1 nginx nginx 2.3K Dec 24 03:36 access.log-20181224.gz
    -rw-r--r-- 1 nginx nginx 2.3K Dec 25 03:47 access.log-20181225.gz
    -rw-r--r-- 1 nginx nginx 1.5K Dec 26 03:16 access.log-20181226.gz
    -rw-r--r-- 1 nginx nginx 3.7K Dec 27 03:40 access.log-20181227.gz
    -rw-r--r-- 1 nginx nginx 3.5K Dec 28 03:18 access.log-20181228.gz
    -rw-r--r-- 1 nginx nginx 3.8K Dec 29 02:46 access.log-20181229.gz
    -rw-r--r-- 1 nginx nginx 1.6K Dec 30 03:42 access.log-20181230.gz
    -rw-r--r-- 1 nginx nginx  990 Dec 31 02:11 access.log-20181231.gz
    -rw-r--r-- 1 nginx nginx 4.9K Jan  1 03:29 access.log-20190101.zst
    -rw-r--r-- 1 nginx nginx  673 Jan  1 04:39 error.log
    -rw-r--r-- 1 nginx nginx  258 Dec 21 21:52 error.log-20181222.gz
    -rw-r--r-- 1 nginx nginx  797 Dec 22 13:59 error.log-20181223.gz
    -rw-r--r-- 1 nginx nginx 1000 Dec 23 21:24 error.log-20181224.gz
    -rw-r--r-- 1 nginx nginx  304 Dec 24 12:25 error.log-20181225.gz
    -rw-r--r-- 1 nginx nginx  434 Dec 27 02:56 error.log-20181227.gz
    -rw-r--r-- 1 nginx nginx  382 Dec 28 02:22 error.log-20181228.gz
    -rw-r--r-- 1 nginx nginx  382 Dec 28 21:06 error.log-20181229.gz
    -rw-r--r-- 1 nginx nginx  376 Dec 29 12:38 error.log-20181230.gz
    -rw-r--r-- 1 nginx nginx  416 Dec 30 22:58 error.log-20181231.gz
    -rw-r--r-- 1 nginx nginx  637 Dec 31 15:00 error.log-20190101.zst
    


    For PHP-FPM similar steps, change the logrotation profile /etc/logrotate.d/php-fpm

    from
    Code (Text):
    /var/log/php-fpm/*.log {
            daily
            dateext
            missingok
            rotate 10
            maxsize 500M
            compress
            delaycompress
            notifempty
            postrotate
            /bin/kill -SIGUSR1 $(cat /var/run/php-fpm/php-fpm.pid 2>/dev/null) 2>/dev/null || true
            endscript           
    }
    

    to
    Code (Text):
    /var/log/php-fpm/*.log {
            daily
            dateext
            missingok
            rotate 10
            maxsize 500M
            compress
            delaycompress
            compresscmd /usr/local/bin/zstd
            uncompresscmd /usr/local/bin/unzstd
            compressoptions -9 --long -T0
            compressext .zst
            notifempty
            postrotate
            /bin/kill -SIGUSR1 $(cat /var/run/php-fpm/php-fpm.pid 2>/dev/null) 2>/dev/null || true
            endscript           
    }
    

    Test debug
    Code (Text):
    logrotate -df /etc/logrotate.d/php-fpm
    

    For actual logrotation
    Code (Text):
    logrotate -fv /etc/logrotate.d/php-fpm
    


    For inspecting compressed .gz or .zst logs, see instructions for using zcat/zgrep, pzcat/pzgrep and zstdcat/zstdgrep commands here.
     
  2. pamamolf

    pamamolf Premium Member Premium Member

    3,430
    324
    83
    May 31, 2014
    Ratings:
    +622
    Local Time:
    8:39 AM
    Nginx-1.17.x
    MariaDB 10.3.x
    If i am not wrong as default it uses all cpu cores/threads....
    Isn't better to not give all cores/threads to it?
     
    Last edited: Apr 16, 2019
  3. eva2000

    eva2000 Administrator Staff Member

    41,262
    9,259
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,202
    Local Time:
    3:39 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Actually zstd uses 1 cpu thread by default so -T0 does number of physical cpu cores available
    Code (Text):
     -T#    : spawns # compression threads (default: 1, 0==# cores) 

    from help
    Code (Text):
    zstd --help
    *** zstd command line interface 64-bits v1.3.8, by Yann Collet ***
    Usage : 
          zstd [args] [FILE(s)] [-o file] 
    
    FILE    : a filename 
              with no FILE, or when FILE is - , read standard input
    Arguments : 
     -#     : # compression level (1-19, default: 3) 
     -d     : decompression 
     -D file: use `file` as Dictionary 
     -o file: result stored into `file` (only if 1 input file) 
     -f     : overwrite output without prompting and (de)compress links 
    --rm    : remove source file(s) after successful de/compression 
     -k     : preserve source file(s) (default) 
     -h/-H  : display help/long help and exit 
    
    Advanced arguments : 
     -V     : display Version number and exit 
     -v     : verbose mode; specify multiple times to increase verbosity
     -q     : suppress warnings; specify twice to suppress errors too
     -c     : force write to standard output, even if it is the console
     -l     : print information about zstd compressed files 
    --ultra : enable levels beyond 19, up to 22 (requires more memory)
    --long[=#]: enable long distance matching with given window log (default: 27)
    --fast[=#]: switch to ultra fast compression level (default: 1)
    --adapt : dynamically adapt compression level to I/O conditions 
     -T#    : spawns # compression threads (default: 1, 0==# cores) 
     -B#    : select size of each job (default: 0==automatic) 
     --rsyncable : compress using a rsync-friendly method (-B sets block size) 
    --no-dictID : don't write dictID into header (dictionary compression)
    --[no-]check : integrity check (default: enabled) 
     -r     : operate recursively on directories 
    --format=zstd : compress files to the .zst format (default) 
    --test  : test compressed file integrity 
    --[no-]sparse : sparse mode (default: enabled on file, disabled on stdout)
     -M#    : Set a memory usage limit for decompression 
    --no-progress : do not display the progress bar 
    --      : All arguments after "--" are treated as files 
    
    Dictionary builder : 
    --train ## : create a dictionary from a training set of files 
    --train-cover[=k=#,d=#,steps=#,split=#] : use the cover algorithm with optional args
    --train-fastcover[=k=#,d=#,f=#,steps=#,split=#,accel=#] : use the fast cover algorithm with optional args
    --train-legacy[=s=#] : use the legacy algorithm with selectivity (default: 9)
     -o file : `file` is dictionary name (default: dictionary) 
    --maxdict=# : limit dictionary to specified size (default: 112640) 
    --dictID=# : force dictionary ID to specified value (default: random)
    
    Benchmark arguments : 
     -b#    : benchmark file(s), using # compression level (default: 3) 
     -e#    : test all compression levels from -bX to # (default: 1)
     -i#    : minimum evaluation time in seconds (default: 3s) 
     -B#    : cut file into independent blocks of size # (default: no block)
    --priority=rt : set process priority to real-time