Join the community today
Become a Member

Sysadmin Compression Comparison Benchmarks: zstd vs brotli vs pigz vs bzip2 vs xz etc

Discussion in 'System Administration' started by eva2000, Sep 3, 2017.

  1. eva2000

    eva2000 Administrator Staff Member

    36,919
    8,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,438
    Local Time:
    5:53 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    No I haven't specifically for pwned pasword database. Though given it's like MySQL dtaabase sql backups being purely texted base, the compression rations might be similar. And well really depends on server hardware/cpu core count if you're focusing on multi-threaded compression tools.
     
  2. Brad Knowles

    Brad Knowles New Member

    4
    2
    3
    May 28, 2018
    Austin, Texas
    Ratings:
    +3
    Local Time:
    7:53 PM
    1.12
    None
    File contents will obviously make a huge difference. The file I referenced contains SHA1 hashes of passwords that have been exposed in one or more breaches, plus the count of the number of breaches in which that password was exposed.

    For example:

    $ head pwned-passwords-ordered-2.0.txt
    000000005AD76BD555C1D6D771DE417A4B87E4B4:3
    00000000A8DAE4228F821FB418F59826079BF368:2
    00000000DD7F2A1C68A35673713783CA390C9E93:630
    00000001E225B908BAC31C56DB04D892E47536E0:5
    00000006BAB7FC3113AA73DE3589630FC08218E7:1
    00000008CD1806EB7B9B46A8F87690B2AC16F617:3
    0000000A0E3B9F25FF41DE4B5AC238C2D545C7A8:14
    0000000A1D4B746FAA3FD526FF6D5BC8052FDB38:16
    0000000CAEF405439D57847A8657218C618160B2:15
    0000000FC1C08E6454BED24F463EA2129E254D43:40 ​

    Sure, it's technically a text file, and those SHA1 hashes can easily be represented in a binary format instead, but other than that, this is going to be a hard nut to crack.

    Of course, this file is also 29GB in size, uncompressed. But you could instead use pwned-passwords-update-1.txt which was the first update after the first full version of this file was produced, and that update is only 548MB in size. But I still think that files of this nature would make a good addition to the tests.

    Thanks!
     
    • Informative Informative x 1
  3. eva2000

    eva2000 Administrator Staff Member

    36,919
    8,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,438
    Local Time:
    5:53 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Unfortunately at 29GB uncompressed size it would take too long to test especially for all the single threaded compression tools for such a test heh. Maybe if I had more free time - unfortunately not right now :)
     
    • Like Like x 1
  4. Brad Knowles

    Brad Knowles New Member

    4
    2
    3
    May 28, 2018
    Austin, Texas
    Ratings:
    +3
    Local Time:
    7:53 PM
    1.12
    None
    Yeah, I figured a 29GB file would be too big. That's why I also mentioned the much smaller 548MB file. ;)

    Anyway, i's just a thought. You're already doing great work for the community, and I'm sure you don't get anywhere near enough credit or thanks!
     
    • Like Like x 1
  5. eva2000

    eva2000 Administrator Staff Member

    36,919
    8,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,438
    Local Time:
    5:53 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Cheers - yeah smaller file might be on the books.
     
    • Like Like x 1
  6. Sanmayce

    Sanmayce New Member

    4
    1
    3
    Jun 9, 2018
    Sofia
    Ratings:
    +3
    Local Time:
    9:53 PM
    Hi,
    me too wanna see how those hashes are ... decompressed.

    >Some folks will know I have a strong focus on performance and efficiency.
    Same here.

    Before running the first 300MB chunk of 'ordered by the hash' file here is the two files. If you wanna see smaller excerpt then just let me know, I chose 300 as it fits in my 8GB laptop's RAM without using virtual such. Gonna run latest TurboBench and latest lzbench along with RAZOR, PPMD, Zstd v1.3.4, Zpaq...

    Code:
    E:\Downloads\z>dir
     Volume in drive E is Kaze_gold
     Volume Serial Number is 4C42-B2C5
    
     Directory of E:\Downloads\z
    
    06/09/2018  05:41 PM    <DIR>          .
    06/09/2018  05:41 PM    <DIR>          ..
    03/17/2018  12:15 AM            12,895 Chunkerito.c
    03/17/2018  12:15 AM            62,976 Chunkerito.exe
    04/26/2018  11:37 AM    <DIR>          Get_The_First_300MB_chunk
    04/26/2018  11:38 AM                30 Get_The_First_300MB_chunk.bat
    04/26/2018  12:14 PM            38,741 Get_The_First_300MB_chunk.zip
    02/17/2018  06:13 AM    31,603,121,046 haveibeenpwned.com_Passwords_pwned-passwords-ordered-2.0.txt
    03/29/2018  12:49 AM             1,631 MokujIN GREEN 224 prompt.lnk
                   6 File(s) 31,603,237,319 bytes
                   3 Dir(s)  176,337,924,096 bytes free
    
    E:\Downloads\z>Get_The_First_300MB_chunk haveibeenpwned.com_Passwords_pwned-passwords-ordered-2.0.txt
    Chunkerito, revision 1+, written by Kaze.
    Purpose: To chunkize/split any file to 'ChunkSize' long chunks.
    Usage: Chunker filename ChunkSize
    Note: For 128MB chunks use ChunkSize = 134217728
    Size of Input TEXTual file: 31,603,121,046
    ^CTerminate batch job (Y/N)? y...
    
    E:\Downloads\z>dir
     Volume in drive E is Kaze_gold
     Volume Serial Number is 4C42-B2C5
    
     Directory of E:\Downloads\z
    
    06/09/2018  05:42 PM    <DIR>          .
    06/09/2018  05:42 PM    <DIR>          ..
    06/09/2018  05:41 PM       314,572,800 Chunkerito.000,001
    06/09/2018  05:42 PM       314,572,800 Chunkerito.000,002
    06/09/2018  05:42 PM       314,572,800 Chunkerito.000,003
    03/17/2018  12:15 AM            12,895 Chunkerito.c
    03/17/2018  12:15 AM            62,976 Chunkerito.exe
    04/26/2018  11:37 AM    <DIR>          Get_The_First_300MB_chunk
    04/26/2018  11:38 AM                30 Get_The_First_300MB_chunk.bat
    04/26/2018  12:14 PM            38,741 Get_The_First_300MB_chunk.zip
    02/17/2018  06:13 AM    31,603,121,046 haveibeenpwned.com_Passwords_pwned-passwords-ordered-2.0.txt
    03/29/2018  12:49 AM             1,631 MokujIN GREEN 224 prompt.lnk
                   9 File(s) 32,546,955,719 bytes
                   3 Dir(s)  175,394,205,696 bytes free
    
    E:\Downloads\z>type "Chunkerito.000,001"|more
    000000005AD76BD555C1D6D771DE417A4B87E4B4:3
    00000000A8DAE4228F821FB418F59826079BF368:2
    00000000DD7F2A1C68A35673713783CA390C9E93:630
    00000001E225B908BAC31C56DB04D892E47536E0:5
    00000006BAB7FC3113AA73DE3589630FC08218E7:1
    00000008CD1806EB7B9B46A8F87690B2AC16F617:3
    0000000A0E3B9F25FF41DE4B5AC238C2D545C7A8:14
    0000000A1D4B746FAA3FD526FF6D5BC8052FDB38:16
    0000000CAEF405439D57847A8657218C618160B2:15
    0000000FC1C08E6454BED24F463EA2129E254D43:40
    00000010F4B38525354491E099EB1796278544B1:3
    0000001225719E346D78D6F493E79FD6EF307A35:2
    0000001306E7C20BE7E0B28B0E8D87EFC634479B:2
    00000016C6C075173C163757BCEA8139D4CC69CF:4
    0000002363B67EAA39B4413802FC10BAC2D0C786:2
    00000028AE3ED9625B6BAD92668069843DB21916:2
    0000002A9AF2DF2267ED92A9FF7FE16E62116CB1:3
    00000030AD61704DDB0AF06B1A64865CA479C1B6:15
    0000003C100967F8714E38BAE61B4DA1B957B947:1
    0000003F2785CA62D59AB905EEAB3533EFFE337A:15
    0000003FA0BC80B317DDE176D6A71F6321CCD35E:2
    000000405907BE8972A2FF97E9A757DDE3BE8EE0:5
    00000042F053B3F16733DFB83D431126D64331FC:6
    00000045ED258F4303903E6AA188652E89D335AD:1
    0000004BDFB224DDAC4CC3FB0FC1A52890354ACC:7
    -- More  --
    
     
    • Informative Informative x 1
  7. Sanmayce

    Sanmayce New Member

    4
    1
    3
    Jun 9, 2018
    Sofia
    Ratings:
    +3
    Local Time:
    9:53 PM
    To reiterate, the [de]compressing of SHA1 lines is quite interesting and in some cases, I daresay, mandatory, simply too many "ununiform/volatile" data can be nicely turned/transformed into such checksums:

    SHA3-512("The quick brown fox jumps over the lazy dog")
    D135BB84D0439DBAC432247EE573A23EA7D3C9DEB2A968EB31D47C4FB45F1EF4422D6C531B5B9BD6F449EBCC449EA94D0A8F05F62130FDA612DA53C79659F609

    BLAKE2b-512("The quick brown fox jumps over the lazy dog")
    A8ADD4BDDDFD93E4877D2746E62817B116364A1FA7BC148D95090BC7333B3673F82401CF7AA2E4CB1ECD90296E3F14CB5413F8ED77BE73045B13914CDCD6A918

    Okay, allow me to share the resultant dumps, the testmachine being i5-7200u @3GHz, 8GB DDR4 2133MHz.

    The lzbench output:
    Code:
    C:\TEXTUAL_MADNESS_bare-minimum>lzbench173 -c4 -i1,15 -o3 -etornado,16/csc,5/gipfeli/blosclz,9/brieflz/fastlz,2/lzo1b,999/lz4fast,1,99/lz4/lz4hc,10,12/lizard,19,29,39,49/lzf,1/lzfse/lzrw,5/lzsse2,17/lzsse4,17/lzsse8,17/quicklz,3/slz_zlib,3/xpack,1,9/shrinker/crush,2/density,3 "Chunkerito.000,001"
    lzbench 1.7.3 (64-bit Windows)   Assembled by P.Skibinski
    Compressor name         Compress. Decompress.  Orig. size  Compr. size  Ratio Filename
    memcpy                  10648 MB/s 11019 MB/s   314572800    314572800 100.00 Chunkerito.000,001
    ...
    done... (cIters=1 dIters=15 cTime=1.0 dTime=2.0 chunkSize=1706MB cSpeed=0MB)
    
    The results sorted by column number 4:
    Compressor name         Compress. Decompress.  Orig. size  Compr. size  Ratio Filename
    csc 2016-10-13 -5        2.17 MB/s    36 MB/s   314572800     96109141  30.55 Chunkerito.000,001
    tornado 0.6a -16         1.55 MB/s    85 MB/s   314572800     99035211  31.48 Chunkerito.000,001
    xpack 2016-06-02 -9        17 MB/s   658 MB/s   314572800    114247320  36.32 Chunkerito.000,001
    lzfse 2017-03-08           44 MB/s   579 MB/s   314572800    116725958  37.11 Chunkerito.000,001
    xpack 2016-06-02 -1       106 MB/s   617 MB/s   314572800    119701473  38.05 Chunkerito.000,001
    density 0.12.5 beta -3    189 MB/s   304 MB/s   314572800    128601768  40.88 Chunkerito.000,001
    crush 1.0 -2             0.68 MB/s   247 MB/s   314572800    132833413  42.23 Chunkerito.000,001
    lzsse2 2016-05-14 -17    0.35 MB/s  2403 MB/s   314572800    135857197  43.19 Chunkerito.000,001
    quicklz 1.5.0 -3           36 MB/s   663 MB/s   314572800    143497637  45.62 Chunkerito.000,001
    lzsse4 2016-05-14 -17    0.34 MB/s  1999 MB/s   314572800    145283631  46.18 Chunkerito.000,001
    lzo1b 2.09 -999            17 MB/s   577 MB/s   314572800    145335371  46.20 Chunkerito.000,001
    lzsse8 2016-05-14 -17    0.35 MB/s  2475 MB/s   314572800    145812467  46.35 Chunkerito.000,001
    lzrw 15-Jul-1991 -5        83 MB/s   436 MB/s   314572800    151526530  48.17 Chunkerito.000,001
    gipfeli 2016-07-13        187 MB/s   299 MB/s   314572800    152543981  48.49 Chunkerito.000,001
    brieflz 1.1.0              93 MB/s   147 MB/s   314572800    154775813  49.20 Chunkerito.000,001
    lz4hc 1.8.0 -12          2.51 MB/s  1794 MB/s   314572800    155682053  49.49 Chunkerito.000,001
    lz4hc 1.8.0 -10            19 MB/s  1817 MB/s   314572800    155692105  49.49 Chunkerito.000,001
    lizard 1.0 -19           2.25 MB/s  2127 MB/s   314572800    155748230  49.51 Chunkerito.000,001
    lzf 3.6 -1                285 MB/s   605 MB/s   314572800    156013044  49.60 Chunkerito.000,001
    lizard 1.0 -29           0.30 MB/s  1652 MB/s   314572800    156559886  49.77 Chunkerito.000,001
    lizard 1.0 -39           2.00 MB/s  2184 MB/s   314572800    156641195  49.79 Chunkerito.000,001
    fastlz 0.1 -2             287 MB/s   498 MB/s   314572800    158119905  50.26 Chunkerito.000,001
    lizard 1.0 -49           0.30 MB/s  1501 MB/s   314572800    158166713  50.28 Chunkerito.000,001
    blosclz 2015-11-10 -9     251 MB/s   646 MB/s   314572800    160165523  50.92 Chunkerito.000,001
    slz_zlib 1.0.0 -3          23 MB/s   273 MB/s   314572800    177293934  56.36 Chunkerito.000,001
    lz4 1.8.0                 697 MB/s  4968 MB/s   314572800    195442959  62.13 Chunkerito.000,001
    lz4fast 1.8.0 -1          698 MB/s  4969 MB/s   314572800    195442959  62.13 Chunkerito.000,001
    lz4fast 1.8.0 -99        1892 MB/s  4303 MB/s   314572800    271688230  86.37 Chunkerito.000,001
    shrinker 0.1               56 MB/s  4950 MB/s   314572800    293099576  93.17 Chunkerito.000,001
    
    The TurboBench output:
    Code:
    C:\TEXTUAL_MADNESS_bare-minimum>"turbobench_v18.05_-_build_04_May_2018.exe" "Chunkerito.000,001" -ememcpy/lz4,1,16/snappy_c/yappy/zlib,1,5,9/bzip2/lzlib,9d29fb273/lzham,4fb258:x4:d29/lzma,9d29:fb273:mf=bt4/libdeflate,1,5,12/oodle,19,49,89,112,114,116,118,129/lzturbo,19,12,10,29,22,20,39,32,30,49,59,59t2,59t4/zstd,1,5,12,22,22d29/lizard,11,19,21,29,31,39,41,49/brotli,1,5,11/brotli,11d29/lzma,9/chameleon,2/density,3/lzham,4/trle/bsc,3,6/zpaq,2,5 -I3 -J31 -k1 -B2G
    TurboBench:  - Sun Jun 10 07:18:36 2018
    
          C Size  ratio%     C MB/s     D MB/s   Name            File
        85871011    27.3       0.31       0.31   zpaq 5                           Chunkerito.000,001
        87731624    27.9       1.12      47.38   lzturbo 49                       Chunkerito.000,001
        90969561    28.9       1.28    1704.88   lzturbo 39                       Chunkerito.000,001
        93322838    29.7       0.52    1248.07   zstd 22                          Chunkerito.000,001
        93324372    29.7       0.48    1245.61   zstd 22d29                       Chunkerito.000,001
        93659859    29.8       0.26     207.99   lzham 4fb258:x4:d29              Chunkerito.000,001
        94395990    30.0       0.45     203.24   brotli 11                        Chunkerito.000,001
        94448425    30.0       0.34     175.30   brotli 11d29                     Chunkerito.000,001
        95368570    30.3       0.60     206.86   lzham 4                          Chunkerito.000,001
        95762613    30.4       0.63      33.15   lzlib 9d29fb273                  Chunkerito.000,001
        96021020    30.5      19.15      14.52   bsc 3                            Chunkerito.000,001
        96625090    30.7       0.94      44.99   lzma 9                           Chunkerito.000,001
        96738102    30.8       0.46    1136.27   oodle 129                        Chunkerito.000,001
        96738102    30.8       0.54    1135.97   oodle 89                         Chunkerito.000,001
        96817005    30.8       6.55     480.71   libdeflate 12                    Chunkerito.000,001
        98387497    31.3     323.86    1438.53   zstd 1                           Chunkerito.000,001
       100242511    31.9      11.45      29.78   bzip2                            Chunkerito.000,001
       100636651    32.0       0.72      47.76   lzma 9d29:fb273:mf=bt4           Chunkerito.000,001
       102336138    32.5      15.21       6.61   bsc 6                            Chunkerito.000,001
       103754533    33.0       6.63      27.82   lzturbo 59                       Chunkerito.000,001
       103755353    33.0       6.92      36.70   lzturbo 59t2                     Chunkerito.000,001
       103756181    33.0       7.30      49.30   lzturbo 59t4                     Chunkerito.000,001
       103886690    33.0     193.81     392.57   brotli 1                         Chunkerito.000,001
       106441065    33.8       0.26     355.95   oodle 19                         Chunkerito.000,001
       109884410    34.9     133.06    1152.55   lizard 41                        Chunkerito.000,001
       110549867    35.1      70.33     720.28   zstd 5                           Chunkerito.000,001
       110599411    35.2      16.65     257.40   brotli 5                         Chunkerito.000,001
       111165601    35.3      13.51     372.55   zstd 12                          Chunkerito.000,001
       113195029    36.0      14.01     232.93   zlib 9                           Chunkerito.000,001
       113967252    36.2       0.32     940.81   lizard 49                        Chunkerito.000,001
       114165753    36.3      20.08     582.03   lzturbo 32                       Chunkerito.000,001
       114186709    36.3      30.25     231.85   zlib 5                           Chunkerito.000,001
       115810528    36.8      96.30     546.45   libdeflate 5                     Chunkerito.000,001
       116400475    37.0     157.89    1251.62   lzturbo 30                       Chunkerito.000,001
       117664639    37.4     138.39     586.06   libdeflate 1                     Chunkerito.000,001
       118908410    37.8     154.97    1251.58   lizard 31                        Chunkerito.000,001
       119552458    38.0      72.83     615.79   zlib 1                           Chunkerito.000,001
       128600598    40.9     275.55     329.62   density 3                        Chunkerito.000,001
       130011833    41.3       1.86      60.88   zpaq 2                           Chunkerito.000,001
       133596888    42.5       1.44     601.32   lzturbo 29                       Chunkerito.000,001
       134797023    42.9       2.34    1516.57   lizard 39                        Chunkerito.000,001
       154993669    49.3      22.35     999.67   lzturbo 22                       Chunkerito.000,001
       155681949    49.5       2.41    2164.26   lz4 16                           Chunkerito.000,001
       155698643    49.5       1.82    2532.30   lzturbo 19                       Chunkerito.000,001
       155748234    49.5       2.40    1995.89   lizard 19                        Chunkerito.000,001
       155768988    49.5       3.47    2498.18   oodle 49                         Chunkerito.000,001
       155980551    49.6       0.53    3790.44   oodle 116                        Chunkerito.000,001
       155983137    49.6       0.57    3791.82   oodle 118                        Chunkerito.000,001
       156559890    49.8       0.32    1863.56   lizard 29                        Chunkerito.000,001
       158655850    50.4      20.44    3763.55   oodle 114                        Chunkerito.000,001
       158709285    50.5      50.82    3086.17   lzturbo 12                       Chunkerito.000,001
       163308151    51.9      59.95    2478.10   oodle 112                        Chunkerito.000,001
       183469433    58.3      93.88    3143.97   yappy                            Chunkerito.000,001
       185920735    59.1     602.28    3446.96   lzturbo 10                       Chunkerito.000,001
       186352774    59.2     421.64    1570.19   snappy_c                         Chunkerito.000,001
       186378354    59.2     555.04    2515.19   lzturbo 20                       Chunkerito.000,001
       187531489    59.6     184.75    3668.44   lizard 11                        Chunkerito.000,001
       192251338    61.1     181.80    4073.30   lizard 21                        Chunkerito.000,001
       195442963    62.1     926.71    5665.63   lz4 1                            Chunkerito.000,001
       197385318    62.7    1577.53    1820.17   chameleon 2                      Chunkerito.000,001
       229165208    72.8     248.56    2461.85   trle                             Chunkerito.000,001
       314572804   100.0   10221.04    9422.30   memcpy                           Chunkerito.000,001
    
    And the 'usual suspects' - PPMD, GZIP, 7zip, BSC, Zpaq, Zstd, RAZOR:
    Code (Text):
    C:\TEXTUAL_MADNESS_bare-minimum>timer64 PPMd_varI_rev2_Intel15_32bit.exe e -o6 -m256 -f"Chunkerito.000,001".O6.PPMd_varI "Chunkerito.000,001"
    Fast PPMII compressor for textual data, variant I, Apr  3 2016
    Chunkerito.000,001:314572800 >99746859, 2.13 bpb, used:252.7MB, speed: 3169 KB/
    
    Kernel  Time =     0.406 =    0%
    User    Time =    95.828 =   98%
    Process Time =    96.234 =   98%    Virtual  Memory =    258 MB
    Global  Time =    97.696 =  100%    Physical Memory =    259 MB
    
    C:\TEXTUAL_MADNESS_bare-minimum>timer64 PPMd_varI_rev2_Intel15_32bit.exe e -o16 -m256 -f"Chunkerito.000,001".O16.PPMd_varI "Chunkerito.000,001"
    Fast PPMII compressor for textual data, variant I, Apr  3 2016
    Chunkerito.000,001:314572800 >98346467, 2.09 bpb, used: 11.2MB, speed: 3090 KB/
    
    Kernel  Time =     0.328 =    0%
    User    Time =    98.718 =   99%
    Process Time =    99.046 =   99%    Virtual  Memory =    258 MB
    Global  Time =    99.427 =  100%    Physical Memory =    259 MB
    
    C:\TEXTUAL_MADNESS_bare-minimum>timer64 "7za_v16.04_x64.exe" a -tgzip -mx9 "Chunkerito.000,001".MX9.zip "Chunkerito.000,001"
    
    7-Zip (a) [64] 16.04 : Copyright (c) 1999-2016 Igor Pavlov : 2016-10-04
    
    Scanning the drive:
    1 file, 314572800 bytes (300 MiB)
    
    Creating archive: Chunkerito.000,001.MX9.zip
    
    Items to compress: 1
    
    Files read from disk: 1
    Archive size: 96934873 bytes (93 MiB)
    Everything is Ok
    
    Kernel  Time =     0.140 =    0%
    User    Time =   228.625 =   99%
    Process Time =   228.765 =   99%    Virtual  Memory =      6 MB
    Global  Time =   229.141 =  100%    Physical Memory =      8 MB
    
    C:\TEXTUAL_MADNESS_bare-minimum>timer64 "7za_v16.04_x64.exe" a -t7z -mx9 -md=29 "Chunkerito.000,001".MX9Dict512.7z "Chunkerito.000,001"
    
    7-Zip (a) [64] 16.04 : Copyright (c) 1999-2016 Igor Pavlov : 2016-10-04
    
    Scanning the drive:
    1 file, 314572800 bytes (300 MiB)
    
    Creating archive: Chunkerito.000,001.MX9Dict512.7z
    
    Items to compress: 1
    
    Files read from disk: 1
    Archive size: 96637109 bytes (93 MiB)
    Everything is Ok
    
    Kernel  Time =     2.093 =    0%
    User    Time =   406.453 =  132%
    Process Time =   408.546 =  133%    Virtual  Memory =   4177 MB
    Global  Time =   307.107 =  100%    Physical Memory =   3226 MB
    
    C:\TEXTUAL_MADNESS_bare-minimum>timer64 "xz_v5.2.3_x64.exe" -z -k -f -9 -e -v -v --lzma2=dict=512MiB --threads=1 "Chunkerito.000,001"
    xz_v5.2.3_x64: Filter chain: --lzma2=dict=512MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
    xz_v5.2.3_x64: 5,378 MiB of memory is required. The limiter is disabled.
    xz_v5.2.3_x64: Decompression will need 513 MiB of memory.
    Chunkerito.000,001 (1/1)
      100 %        92.1 MiB / 300.0 MiB = 0.307   776 KiB/s       6:36
    
    Kernel  Time =     1.859 =    0%
    User    Time =   391.625 =   98%
    Process Time =   393.484 =   99%    Virtual  Memory =   5389 MB
    Global  Time =   396.105 =  100%    Physical Memory =   2710 MB
    
    C:\TEXTUAL_MADNESS_bare-minimum>timer64 "bsc_v3.1.0_x64.exe" e "Chunkerito.000,001" "Chunkerito.000,001".ST6Block512.bsc -b512 -m6 -cp -Tt
    This is bsc, Block Sorting Compressor. Version 3.1.0. 8 July 2012.
    Copyright (c) 2009-2012 Ilya Grebnov <[email protected]>.
    
    Chunkerito.000,001 compressed 314572800 into 92270528 in 19.062 seconds.
    
    Kernel  Time =     0.687 =    3%
    User    Time =    17.671 =   92%
    Process Time =    18.359 =   95%    Virtual  Memory =   1579 MB
    Global  Time =    19.156 =  100%    Physical Memory =   1571 MB
    
    C:\TEXTUAL_MADNESS_bare-minimum>timer64 "zpaq_v7.05_x64.exe" add "Chunkerito.000,001".method29.zpaq "Chunkerito.000,001" -method 29 -threads 1
    zpaq v7.05 journaling archiver, compiled Apr 17 2015
    Adding 314.572800 MB in 1 files -method 29 -threads 1 at 2018-06-11 04:22:31.
    100.00% 0:00:00 + Chunkerito.000,001 314572800
    100.00% 0:00:00 [1..4565] 314591068 -method 29,239,0
    1 +added, 0 -removed.
    
    0.000000 + (314.572800 -> 314.572800 -> 126.796303) = 126.796303 MB
    198.641 seconds (all OK)
    
    Kernel  Time =     0.687 =    0%
    User    Time =   197.734 =   99%
    Process Time =   198.421 =   99%    Virtual  Memory =   2169 MB
    Global  Time =   198.655 =  100%    Physical Memory =   1892 MB
    
    C:\TEXTUAL_MADNESS_bare-minimum>timer64 "zpaq_v7.05_x64.exe" add "Chunkerito.000,001".method59.zpaq "Chunkerito.000,001" -method 59 -threads 1
    zpaq v7.05 journaling archiver, compiled Apr 17 2015
    Adding 314.572800 MB in 1 files -method 59 -threads 1 at 2018-06-11 04:25:50.
    100.00% 0:00:00 + Chunkerito.000,001 314572800
    100.00% 0:00:00 [1..4565] 314591068 -method 59,239,0
    1 +added, 0 -removed.
    
    0.000000 + (314.572800 -> 314.572800 -> 85.992745) = 85.992745 MB
    1026.625 seconds (all OK)
    
    Kernel  Time =     1.765 =    0%
    User    Time =  1024.328 =   99%
    Process Time =  1026.093 =   99%    Virtual  Memory =   5568 MB
    Global  Time =  1026.641 =  100%    Physical Memory =   4215 MB
    
    C:\TEXTUAL_MADNESS_bare-minimum>timer64 zstd-v1.3.4-win64.exe --ultra -22 --zstd=wlog=29,clog=30,hlog=30,slog=26 "Chunkerito.000,001"
    Chunkerito.000,001   : 29.67%   (314572800 => 93324464 bytes, Chunkerito.000,001.zst)
    
    Kernel  Time =     4.843 =    0%
    User    Time = 32158.031 =   99%
    Process Time = 32162.875 =   99%    Virtual  Memory =  18479 MB
    Global  Time = 32169.834 =  100%    Physical Memory =   7183 MB
    
    C:\TEXTUAL_MADNESS_bare-minimum>timer64 rz_1.01.exe a -d 512M "Chunkerito.000,001".512M.rz "Chunkerito.000,001"
    
     *** RAZOR Archiver 1.01 (2017-09-14) - DEMO/TEST version ***
     *** (c) Christian Martelock ([email protected]) ***
    
     Scanning c:\textual_madness_bare-minimum\chunkerito.000,001
     Found 0 dirs, 1 files, 314572800 bytes.
    
     Creating archive Chunkerito.000,001.512M.rz
     Window : 307200K (2048M..512G)
     Header : 52
     Size   : 92853683
    
     Archive ok. Added 0 dirs, 1 files, 314572800 bytes.
     CPU time = 1273.953s / wall time = 879.841s
    
    Kernel  Time =     3.140 =    0%
    User    Time =  1273.953 =  144%
    Process Time =  1277.093 =  145%    Virtual  Memory =   3770 MB
    Global  Time =   879.943 =  100%    Physical Memory =   3260 MB
    

    For more info and to download the package allowing replication (to run all the benches above):
    The Zennish Microdeduplicator · Issue #1 · Sanmayce/Nakamichi

    LzTurbo decompresses SHA1 sums 1704.88/0.31=5499x faster than Zpaq 3532.gif
     
    Last edited: Jun 11, 2018
  8. eva2000

    eva2000 Administrator Staff Member

    36,919
    8,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,438
    Local Time:
    5:53 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    nice.. maybe better if you started a new thread of your own in System Administration as your juicy results are buried in this thread on page 2 right now :)

    Also for posting code or output from commands to keep the formatting, you might want to use CODEB tags for code as it has wider frame than CODE tag How to use forum BBCODE code tags :)
     
..