Featured Multi-threaded compression tools (pigz, pbzip2, lbzip2, plzip, p7zip)

So anyone actually use centmin.sh menu option 17 to install and use the included multi-threaded compression tools (pigz, pbzip2, lbzip2, plzip, p7zip) ? Heaps of benefits particularly for using pigz as opposed to gzip http://vbtechsupport.com/2094/

---

Code:

pigz -h
Usage: pigz [options] [files ...]
  will compress files in place, adding the suffix '.gz'.  If no files are
  specified, stdin will be compressed to stdout.  pigz does what gzip does,
  but spreads the work over multiple processors and cores when compressing.

Options:
  -0 to -9, -11        Compression level (11 is much slower, a few % better)
  --fast, --best       Compression levels 1 and 9 respectively
  -b, --blocksize mmm  Set compression block size to mmmK (default 128K)
  -c, --stdout         Write all processed output to stdout (won't delete)
  -d, --decompress     Decompress the compressed input
  -f, --force          Force overwrite, compress .gz, links, and to terminal
  -F  --first          Do iterations first, before block split for -11
  -h, --help           Display a help screen and quit
  -i, --independent    Compress blocks independently for damage recovery
  -I, --iterations n   Number of iterations for -11 optimization
  -k, --keep           Do not delete original file after processing
  -K, --zip            Compress to PKWare zip (.zip) single entry format
  -l, --list           List the contents of the compressed input
  -L, --license        Display the pigz license and quit
  -M, --maxsplits n    Maximum number of split blocks for -11
  -n, --no-name        Do not store or restore file name in/from header
  -N, --name           Store/restore file name and mod time in/from header
  -O  --oneblock       Do not split into smaller blocks for -11
  -p, --processes n    Allow up to n compression threads (default is the
                       number of online processors, or 8 if unknown)
  -q, --quiet          Print no messages, even on error
  -r, --recursive      Process the contents of all subdirectories
  -R, --rsyncable      Input-determined block locations for rsync
  -S, --suffix .sss    Use suffix .sss instead of .gz (for compression)
  -t, --test           Test the integrity of the compressed input
  -T, --no-time        Do not store or restore mod time in/from header
  -v, --verbose        Provide more verbose output
  -V  --version        Show the version of pigz
  -z, --zlib           Compress to zlib (.zz) instead of gzip format
  --                   All arguments after "--" are treated as files

 

or part 2 http://vbtechsupport.com/1614/

 

yeah was going to say lbzip2 level 1 is faster and compresses more that pbzip2

 

Yeah my semi-abandoned mysqlmybackup.sh script used pigz http://vbtechsupport.com/974/ (script just got too big so broke it down into).

 

For those not reading my blog at gziptest.sh part 2: multi-threaded compression benchmarks | vbtechsupport.com (you should read the LZMA (7Z) compression limitations note )

for MySQL sql dump file



For directory tar'd compressed file

 

Yeah lbzip2 level 1 for higher compression ratio at the best speed but read gziptest.sh part 2: multi-threaded compression benchmarks | vbtechsupport.com and the notes for memory usage which increase alot. If you don't have enough memory pbzip2 level should be good alternative to pigz. Pigz level 4 is what I use for fast compression times.

 

TBH, probably a waste of cpu time if you want fast backups. Probably get better compression ratio and speed with plzip level 3 or 4.

Try it just prefix the command with time to compare times and file sizes

Code:

time pigz -4 mysqlbackup.sql
time pbzip2 -9 mysqlbackup.sql
time plzip -3 mysqlbackup.sql

 

then give plzip -5 or -6 a try

also what some folks do is break their backups into highly compressible and not so compressible segments. So for non-binary/graphic files use higher levels of compression but for graphics and stuff that doesn't compress well use lowest level of compression