Want more timely Centmin Mod News Updates?
Become a Member

Featured Multi-threaded compression tools (pigz, pbzip2, lbzip2, plzip, p7zip)

Discussion in 'Other Centmin Mod Installed software' started by eva2000, May 29, 2014.

  1. eva2000

    eva2000 Administrator Staff Member

    30,561
    6,851
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,266
    Local Time:
    3:44 AM
    Nginx 1.13.x
    MariaDB 5.5
    So anyone actually use centmin.sh menu option 17 to install and use the included multi-threaded compression tools (pigz, pbzip2, lbzip2, plzip, p7zip) ? Heaps of benefits particularly for using pigz as opposed to gzip http://vbtechsupport.com/2094/ :)

    Code:
    pigz -h
    Usage: pigz [options] [files ...]
      will compress files in place, adding the suffix '.gz'.  If no files are
      specified, stdin will be compressed to stdout.  pigz does what gzip does,
      but spreads the work over multiple processors and cores when compressing.
    
    Options:
      -0 to -9, -11        Compression level (11 is much slower, a few % better)
      --fast, --best       Compression levels 1 and 9 respectively
      -b, --blocksize mmm  Set compression block size to mmmK (default 128K)
      -c, --stdout         Write all processed output to stdout (won't delete)
      -d, --decompress     Decompress the compressed input
      -f, --force          Force overwrite, compress .gz, links, and to terminal
      -F  --first          Do iterations first, before block split for -11
      -h, --help           Display a help screen and quit
      -i, --independent    Compress blocks independently for damage recovery
      -I, --iterations n   Number of iterations for -11 optimization
      -k, --keep           Do not delete original file after processing
      -K, --zip            Compress to PKWare zip (.zip) single entry format
      -l, --list           List the contents of the compressed input
      -L, --license        Display the pigz license and quit
      -M, --maxsplits n    Maximum number of split blocks for -11
      -n, --no-name        Do not store or restore file name in/from header
      -N, --name           Store/restore file name and mod time in/from header
      -O  --oneblock       Do not split into smaller blocks for -11
      -p, --processes n    Allow up to n compression threads (default is the
                           number of online processors, or 8 if unknown)
      -q, --quiet          Print no messages, even on error
      -r, --recursive      Process the contents of all subdirectories
      -R, --rsyncable      Input-determined block locations for rsync
      -S, --suffix .sss    Use suffix .sss instead of .gz (for compression)
      -t, --test           Test the integrity of the compressed input
      -T, --no-time        Do not store or restore mod time in/from header
      -v, --verbose        Provide more verbose output
      -V  --version        Show the version of pigz
      -z, --zlib           Compress to zlib (.zz) instead of gzip format
      --                   All arguments after "--" are treated as files
     
    • Like Like x 1
  2. Matt

    Matt Moderator Staff Member

    691
    320
    63
    May 25, 2014
    Sheffield, UK
    Ratings:
    +445
    Local Time:
    5:44 PM
    1.7.1
    MariaDB 10
    I use pbzip2 to compress my database backups before offloading to the backup server

    Code:
    # Compress them with BZIP2
    /usr/bin/pbzip2 -f $DIR/*.$TODAY.sql
    sleep 2
     
    • Like Like x 2
  3. Zykov

    Zykov Member

    31
    7
    8
    May 28, 2014
    Ratings:
    +7
    Local Time:
    2:44 AM
    Nginx 1.7.1
    MariaDB 10
    Just installed for pbzip2. Got no errors on beta20, but there was some errors on beta19 (don't remember which ones :rolleyes:)
     
  4. BamaStangGuy

    BamaStangGuy Active Member

    473
    137
    43
    May 25, 2014
    Ratings:
    +181
    Local Time:
    11:44 AM
    I just do a yum install pigz from rpmforge and use it to backup databases and tar files. What is the added benefit of pbzip2?
     
  5. Matt

    Matt Moderator Staff Member

    691
    320
    63
    May 25, 2014
    Sheffield, UK
    Ratings:
    +445
    Local Time:
    5:44 PM
    1.7.1
    MariaDB 10
    http://vbtechsupport.com/1576/
     
    • Like Like x 1
  6. eva2000

    eva2000 Administrator Staff Member

    30,561
    6,851
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,266
    Local Time:
    3:44 AM
    Nginx 1.13.x
    MariaDB 5.5
  7. Matt

    Matt Moderator Staff Member

    691
    320
    63
    May 25, 2014
    Sheffield, UK
    Ratings:
    +445
    Local Time:
    5:44 PM
    1.7.1
    MariaDB 10
    I might have a look at lbzip2 as I've got 32GB of memory on the server.
     
  8. eva2000

    eva2000 Administrator Staff Member

    30,561
    6,851
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,266
    Local Time:
    3:44 AM
    Nginx 1.13.x
    MariaDB 5.5
    yeah was going to say lbzip2 level 1 is faster and compresses more that pbzip2
     
  9. BamaStangGuy

    BamaStangGuy Active Member

    473
    137
    43
    May 25, 2014
    Ratings:
    +181
    Local Time:
    11:44 AM
    Going from pigz -9 to pbzip -9 on backups:

    Screen Shot 2014-05-29 at 2.18.11 AM.png
     
    • Like Like x 1
  10. Matt

    Matt Moderator Staff Member

    691
    320
    63
    May 25, 2014
    Sheffield, UK
    Ratings:
    +445
    Local Time:
    5:44 PM
    1.7.1
    MariaDB 10
    Just swapped it out in the script

    Code:
    COMPRESS="/usr/local/bin/lbzip2"
    
    # Compress them with LBZIP2
    $COMPRESS -f $DIR/*.$TODAY.sql
    sleep 2
     
  11. eva2000

    eva2000 Administrator Staff Member

    30,561
    6,851
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,266
    Local Time:
    3:44 AM
    Nginx 1.13.x
    MariaDB 5.5
  12. eva2000

    eva2000 Administrator Staff Member

    30,561
    6,851
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,266
    Local Time:
    3:44 AM
    Nginx 1.13.x
    MariaDB 5.5
    • Like Like x 1
  13. RoldanLT

    RoldanLT Well-Known Member

    3,931
    960
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,311
    Local Time:
    1:44 AM
    1.11
    10.2
    Looks like lbzip2 is the best :)
    2nd is pbzip2 :)
     
  14. eva2000

    eva2000 Administrator Staff Member

    30,561
    6,851
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,266
    Local Time:
    3:44 AM
    Nginx 1.13.x
    MariaDB 5.5
  15. RoldanLT

    RoldanLT Well-Known Member

    3,931
    960
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,311
    Local Time:
    1:44 AM
    1.11
    10.2
    How about pbzip2 level 9?
    That's what I'm currently using :)
     
  16. eva2000

    eva2000 Administrator Staff Member

    30,561
    6,851
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,266
    Local Time:
    3:44 AM
    Nginx 1.13.x
    MariaDB 5.5
    TBH, probably a waste of cpu time if you want fast backups. Probably get better compression ratio and speed with plzip level 3 or 4.

    Try it just prefix the command with time to compare times and file sizes

    Code:
    time pigz -4 mysqlbackup.sql
    time pbzip2 -9 mysqlbackup.sql
    time plzip -3 mysqlbackup.sql
    
     
  17. RoldanLT

    RoldanLT Well-Known Member

    3,931
    960
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,311
    Local Time:
    1:44 AM
    1.11
    10.2
    For now, I prefer the smallest file size output.
    I don't care about the time :D
     
  18. eva2000

    eva2000 Administrator Staff Member

    30,561
    6,851
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,266
    Local Time:
    3:44 AM
    Nginx 1.13.x
    MariaDB 5.5
    then give plzip -5 or -6 a try :)

    also what some folks do is break their backups into highly compressible and not so compressible segments. So for non-binary/graphic files use higher levels of compression but for graphics and stuff that doesn't compress well use lowest level of compression
     
    Last edited: Nov 12, 2014
    • Like Like x 1
  19. RoldanLT

    RoldanLT Well-Known Member

    3,931
    960
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,311
    Local Time:
    1:44 AM
    1.11
    10.2
    plzip compressed file name extension is .lz?
     
  20. RoldanLT

    RoldanLT Well-Known Member

    3,931
    960
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,311
    Local Time:
    1:44 AM
    1.11
    10.2
    Yeah! The most smallest file size :D