Welcome to Centmin Mod Community
Become a Member

Featured Multi-threaded compression tools (pigz, pbzip2, lbzip2, plzip, p7zip)

Discussion in 'Other Centmin Mod Installed software' started by eva2000, May 29, 2014.

  1. eva2000

    eva2000 Administrator Staff Member

    30,546
    6,849
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,262
    Local Time:
    3:10 AM
    Nginx 1.13.x
    MariaDB 5.5
    yup
    Yeah can't be beat for compression ratios ! But uses alot of memory.
     
  2. RoldanLT

    RoldanLT Well-Known Member

    3,930
    960
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,311
    Local Time:
    1:10 AM
    1.11
    10.2
    Memory meaning RAM? I have plenty :D
    Not too much on cpu?
     
  3. eva2000

    eva2000 Administrator Staff Member

    30,546
    6,849
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,262
    Local Time:
    3:10 AM
    Nginx 1.13.x
    MariaDB 5.5
    all multi-threaded tools use more cpu as they use more threads in parallel hence the speed

    but yes memory = ram :)

    yeah you have alot of idle memory on that 32GB server heh
     
    • Like Like x 1
  4. RoldanLT

    RoldanLT Well-Known Member

    3,930
    960
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,311
    Local Time:
    1:10 AM
    1.11
    10.2
    What is the recommended Compression level #?
    Or recommended multi-threaded compression tool?
    That will not consume a lot of the CPU and just use more RAM?

    I'm using this right now:
    But looks like hogging the cpu when it compress more than 6GB files.
     
  5. eva2000

    eva2000 Administrator Staff Member

    30,546
    6,849
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,262
    Local Time:
    3:10 AM
    Nginx 1.13.x
    MariaDB 5.5
  6. RoldanLT

    RoldanLT Well-Known Member

    3,930
    960
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,311
    Local Time:
    1:10 AM
    1.11
    10.2
    I just remove the hard coded level 5, that way it will automatically consume what is free?
     
  7. eva2000

    eva2000 Administrator Staff Member

    30,546
    6,849
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,262
    Local Time:
    3:10 AM
    Nginx 1.13.x
    MariaDB 5.5
    lbzip2 manual -1 to -9 is block size for compression with -9 being default so leaving -5 off will use -9 AFAIK from the manual LBZIP2

    maybe use -1 -s -u together ?

    also reduce number of threads used

     
    Last edited: Nov 7, 2014
    • Like Like x 1
  8. eva2000

    eva2000 Administrator Staff Member

    30,546
    6,849
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,262
    Local Time:
    3:10 AM
    Nginx 1.13.x
    MariaDB 5.5
    Just re-did some compression tests on tar file as I added xz and pxz compression tools into the mix and tested levels 1 to 6 for each. Trying out new table BBCODE on forums :D

    On

    • Xeon E3-1240v3
    • 32GB RAM
    • 250GB Samsung 840 EVO SSD

    Tar file compression test

    Code:
    -rw-r--r--   1 root root 785M Feb 24 02:38 ccache.tar
    
    method comp lvl compress time decompress time compress cpu % compress max mem compress cs waits decompress cpu % decompress max mem decompress cs waits compression ratio compress size
    zip lvl 1 1 11.46 5.64 100 4768 4 100 7328 2 36 290094
    zip lvl 2 2 12.19 5.50 100 4784 4 99 7328 2 35 283037
    zip lvl 3 3 15.69 5.50 100 4768 3 100 7312 2 34 278277
    zip lvl 4 4 15.42 5.52 99 4784 4 100 7328 2 33 270516
    zip lvl 5 5 20.85 5.40 100 4768 6 99 7312 2 32 264760
    zip lvl 6 6 33.53 5.34 100 4736 10 100 7328 2 32 262171
    gzip lvl 1 1 17.17 6.71 100 3248 6 100 2736 3 35 284476
    gzip lvl 2 2 17.90 6.57 100 3232 5 100 2736 3 34 277441
    gzip lvl 3 3 20.98 6.51 100 3232 5 100 2736 3 34 273699
    gzip lvl 4 4 21.16 6.49 100 3232 7 99 2752 3 33 266843
    gzip lvl 5 5 26.53 6.36 100 3232 6 100 2752 3 32 261646
    gzip lvl 6 6 38.75 6.30 99 3216 10 100 2736 3 32 259173
    bzip2 lvl 1 1 55.64 22.00 100 6832 7 99 4224 4 32 260668
    bzip2 lvl 2 2 51.95 22.76 100 9968 2 99 5792 6 32 259629
    bzip2 lvl 3 3 50.81 23.13 100 13088 4 99 7360 5 32 258250
    bzip2 lvl 4 4 51.00 23.37 100 14656 4 99 8944 5 32 257101
    bzip2 lvl 5 5 50.63 23.51 100 17424 5 99 10496 5 31 256247
    bzip2 lvl 6 6 50.57 23.64 100 22032 4 99 12080 6 31 255263
    pigz lvl 1 1 2.98 2.96 676 36640 11571 143 3760 59280 35 284715
    pigz lvl 2 2 3.16 2.86 668 36784 12844 145 3760 59049 34 277661
    pigz lvl 3 3 3.60 3.06 695 44800 12984 144 3760 59031 34 274043
    pigz lvl 4 4 3.80 2.89 720 44672 13254 145 3760 58857 33 267282
    pigz lvl 5 5 4.64 2.74 756 44768 14935 148 3760 58544 32 262084
    pigz lvl 6 6 6.54 2.69 767 44880 16210 147 3760 58518 32 259653
    pbzip2 lvl 1 1 10.47 3.95 785 118320 1969 783 116272 1864 32 260796
    pbzip2 lvl 2 2 9.97 4.05 783 139680 2210 788 123568 1976 32 259695
    pbzip2 lvl 3 3 9.88 4.19 786 160304 2094 785 135936 1927 32 258607
    pbzip2 lvl 4 4 10.08 4.35 787 182688 2227 786 145696 1952 32 257279
    pbzip2 lvl 5 5 10.11 4.44 786 204656 2025 790 155920 2027 31 257002
    pbzip2 lvl 6 6 10.23 4.77 784 225840 2043 773 187936 1979 31 256430
    lzip lvl 1 1 80.62 18.00 100 51216 18 100 8224 4 28 224924
    lzip lvl 2 2 96.34 17.42 100 75792 22 99 10272 3 27 217384
    lzip lvl 3 3 127.35 16.91 100 96288 29 99 12320 7 26 209650
    lzip lvl 4 4 197.52 16.38 100 145424 43 100 16416 4 23 190331
    lzip lvl 5 5 280.52 15.73 100 186384 63 100 20512 4 22 177522
    lzip lvl 6 6 354.63 15.06 100 366608 85 99 36896 6 20 166226
    plzip lvl 1 1 17.97 8.78 788 557760 817 198 14688 18 28 225938
    plzip lvl 2 2 20.13 8.54 786 879968 653 198 18768 14 27 219164
    plzip lvl 3 3 25.54 8.29 782 1019840 448 199 22816 14 26 211730
    plzip lvl 4 4 37.86 8.16 775 1564912 340 195 31008 17 24 193148
    plzip lvl 5 5 54.27 7.74 771 2022752 259 197 39168 15 22 181550
    plzip lvl 6 6 71.29 7.41 740 3951984 218 197 71920 15 21 170656
    xz lvl 1 1 46.63 17.42 99 8848 8 99 4016 4 27 220122
    xz lvl 2 2 49.01 15.94 99 22816 6 99 5824 4 25 204555
    xz lvl 3 3 260.01 15.05 100 55312 40 99 7856 4 22 184442
    xz lvl 4 4 278.69 14.56 100 102416 66 99 11968 4 22 178454
    xz lvl 5 5 290.57 14.11 100 196640 67 99 20144 4 21 172868
    xz lvl 6 6 337.30 13.57 100 385056 77 99 36544 3 20 165409
    pxz lvl 1 1 12.36 17.62 776 41248 12414 99 3968 5 27 222876
    pxz lvl 2 2 13.97 16.09 794 207024 232 99 5792 5 25 207139
    pxz lvl 3 3 62.46 15.27 752 515952 354 99 7856 4 23 187392
    pxz lvl 4 4 63.27 14.76 744 991088 326 99 11952 4 22 181383
    pxz lvl 5 5 66.13 14.32 731 1941488 531 99 20160 4 21 175915
    pxz lvl 6 6 76.44 13.79 717 3842016 301 99 36528 4 20 168650
     
    Last edited: Feb 24, 2015
    • Like Like x 2
  9. BobbyWibowo

    BobbyWibowo Active Member

    166
    36
    28
    Jul 30, 2015
    Medan, Indonesia
    Ratings:
    +57
    Local Time:
    12:10 AM
    Using pigz for database backups. Just visited their homepage a while ago, laughed hard at this part:
    I actually thought it was pronounced like the plural of pig, lol
     
    • Funny Funny x 2
  10. eva2000

    eva2000 Administrator Staff Member

    30,546
    6,849
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,262
    Local Time:
    3:10 AM
    Nginx 1.13.x
    MariaDB 5.5
    heh always been pig-zee to me :)
     
    • Like Like x 1
    • Funny Funny x 1
  11. RB1

    RB1 Active Member

    281
    72
    28
    Nov 11, 2016
    California
    Ratings:
    +119
    Local Time:
    9:10 AM
    Nginx 1.13.x
    MariaDB 10.1.x
    How can I uninstall these after running CMM menu option 17.
    I don't need these extra things on my server and didn't know it would install as soon as I typed "17" :)
     
  12. eva2000

    eva2000 Administrator Staff Member

    30,546
    6,849
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,262
    Local Time:
    3:10 AM
    Nginx 1.13.x
    MariaDB 5.5
    they don't take up any cpu/mem resources just being installed so just leave them. There isn't really an uninstall routine for source installs :)
     
    • Like Like x 1
  13. RB1

    RB1 Active Member

    281
    72
    28
    Nov 11, 2016
    California
    Ratings:
    +119
    Local Time:
    9:10 AM
    Nginx 1.13.x
    MariaDB 10.1.x
    OK sounds good, thanks for the reply (y)