Sysadmin Compression Comparison Benchmarks: zstd vs brotli vs pigz vs bzip2 vs xz etc

eva2000 · May 28, 2018

Brad Knowles said: ↑

So, I'm curious -- have you tried adding some really large files to the benchmark? For example, pwned-passwords-ordered-2.0.txt from the page at Have I Been Pwned: Pwned Passwords?
Click to expand...

No I haven't specifically for pwned pasword database. Though given it's like MySQL dtaabase sql backups being purely texted base, the compression rations might be similar. And well really depends on server hardware/cpu core count if you're focusing on multi-threaded compression tools.

Brad Knowles · May 28, 2018

File contents will obviously make a huge difference. The file I referenced contains SHA1 hashes of passwords that have been exposed in one or more breaches, plus the count of the number of breaches in which that password was exposed.

For example:

$ head pwned-passwords-ordered-2.0.txt
000000005AD76BD555C1D6D771DE417A4B87E4B4:3
00000000A8DAE4228F821FB418F59826079BF368:2
00000000DD7F2A1C68A35673713783CA390C9E93:630
00000001E225B908BAC31C56DB04D892E47536E0:5
00000006BAB7FC3113AA73DE3589630FC08218E7:1
00000008CD1806EB7B9B46A8F87690B2AC16F617:3
0000000A0E3B9F25FF41DE4B5AC238C2D545C7A8:14
0000000A1D4B746FAA3FD526FF6D5BC8052FDB38:16
0000000CAEF405439D57847A8657218C618160B2:15
0000000FC1C08E6454BED24F463EA2129E254D43:40

Sure, it's technically a text file, and those SHA1 hashes can easily be represented in a binary format instead, but other than that, this is going to be a hard nut to crack.

Of course, this file is also 29GB in size, uncompressed. But you could instead use pwned-passwords-update-1.txt which was the first update after the first full version of this file was produced, and that update is only 548MB in size. But I still think that files of this nature would make a good addition to the tests.

Thanks!

eva2000 · May 28, 2018

Unfortunately at 29GB uncompressed size it would take too long to test especially for all the single threaded compression tools for such a test heh. Maybe if I had more free time - unfortunately not right now

Brad Knowles · May 28, 2018

Yeah, I figured a 29GB file would be too big. That's why I also mentioned the much smaller 548MB file.

Anyway, i's just a thought. You're already doing great work for the community, and I'm sure you don't get anywhere near enough credit or thanks!

eva2000 · May 28, 2018

Cheers - yeah smaller file might be on the books.

Sanmayce · Jun 10, 2018

Hi,
me too wanna see how those hashes are ... decompressed.

>Some folks will know I have a strong focus on performance and efficiency.
Same here.

Before running the first 300MB chunk of 'ordered by the hash' file here is the two files. If you wanna see smaller excerpt then just let me know, I chose 300 as it fits in my 8GB laptop's RAM without using virtual such. Gonna run latest TurboBench and latest lzbench along with RAZOR, PPMD, Zstd v1.3.4, Zpaq...

Code:

E:\Downloads\z>dir
 Volume in drive E is Kaze_gold
 Volume Serial Number is 4C42-B2C5

 Directory of E:\Downloads\z

06/09/2018  05:41 PM    <DIR>          .
06/09/2018  05:41 PM    <DIR>          ..
03/17/2018  12:15 AM            12,895 Chunkerito.c
03/17/2018  12:15 AM            62,976 Chunkerito.exe
04/26/2018  11:37 AM    <DIR>          Get_The_First_300MB_chunk
04/26/2018  11:38 AM                30 Get_The_First_300MB_chunk.bat
04/26/2018  12:14 PM            38,741 Get_The_First_300MB_chunk.zip
02/17/2018  06:13 AM    31,603,121,046 haveibeenpwned.com_Passwords_pwned-passwords-ordered-2.0.txt
03/29/2018  12:49 AM             1,631 MokujIN GREEN 224 prompt.lnk
               6 File(s) 31,603,237,319 bytes
               3 Dir(s)  176,337,924,096 bytes free

E:\Downloads\z>Get_The_First_300MB_chunk haveibeenpwned.com_Passwords_pwned-passwords-ordered-2.0.txt
Chunkerito, revision 1+, written by Kaze.
Purpose: To chunkize/split any file to 'ChunkSize' long chunks.
Usage: Chunker filename ChunkSize
Note: For 128MB chunks use ChunkSize = 134217728
Size of Input TEXTual file: 31,603,121,046
^CTerminate batch job (Y/N)? y...

E:\Downloads\z>dir
 Volume in drive E is Kaze_gold
 Volume Serial Number is 4C42-B2C5

 Directory of E:\Downloads\z

06/09/2018  05:42 PM    <DIR>          .
06/09/2018  05:42 PM    <DIR>          ..
06/09/2018  05:41 PM       314,572,800 Chunkerito.000,001
06/09/2018  05:42 PM       314,572,800 Chunkerito.000,002
06/09/2018  05:42 PM       314,572,800 Chunkerito.000,003
03/17/2018  12:15 AM            12,895 Chunkerito.c
03/17/2018  12:15 AM            62,976 Chunkerito.exe
04/26/2018  11:37 AM    <DIR>          Get_The_First_300MB_chunk
04/26/2018  11:38 AM                30 Get_The_First_300MB_chunk.bat
04/26/2018  12:14 PM            38,741 Get_The_First_300MB_chunk.zip
02/17/2018  06:13 AM    31,603,121,046 haveibeenpwned.com_Passwords_pwned-passwords-ordered-2.0.txt
03/29/2018  12:49 AM             1,631 MokujIN GREEN 224 prompt.lnk
               9 File(s) 32,546,955,719 bytes
               3 Dir(s)  175,394,205,696 bytes free

E:\Downloads\z>type "Chunkerito.000,001"|more
000000005AD76BD555C1D6D771DE417A4B87E4B4:3
00000000A8DAE4228F821FB418F59826079BF368:2
00000000DD7F2A1C68A35673713783CA390C9E93:630
00000001E225B908BAC31C56DB04D892E47536E0:5
00000006BAB7FC3113AA73DE3589630FC08218E7:1
00000008CD1806EB7B9B46A8F87690B2AC16F617:3
0000000A0E3B9F25FF41DE4B5AC238C2D545C7A8:14
0000000A1D4B746FAA3FD526FF6D5BC8052FDB38:16
0000000CAEF405439D57847A8657218C618160B2:15
0000000FC1C08E6454BED24F463EA2129E254D43:40
00000010F4B38525354491E099EB1796278544B1:3
0000001225719E346D78D6F493E79FD6EF307A35:2
0000001306E7C20BE7E0B28B0E8D87EFC634479B:2
00000016C6C075173C163757BCEA8139D4CC69CF:4
0000002363B67EAA39B4413802FC10BAC2D0C786:2
00000028AE3ED9625B6BAD92668069843DB21916:2
0000002A9AF2DF2267ED92A9FF7FE16E62116CB1:3
00000030AD61704DDB0AF06B1A64865CA479C1B6:15
0000003C100967F8714E38BAE61B4DA1B957B947:1
0000003F2785CA62D59AB905EEAB3533EFFE337A:15
0000003FA0BC80B317DDE176D6A71F6321CCD35E:2
000000405907BE8972A2FF97E9A757DDE3BE8EE0:5
00000042F053B3F16733DFB83D431126D64331FC:6
00000045ED258F4303903E6AA188652E89D335AD:1
0000004BDFB224DDAC4CC3FB0FC1A52890354ACC:7
-- More  --

Sanmayce · Jun 11, 2018

To reiterate, the [de]compressing of SHA1 lines is quite interesting and in some cases, I daresay, mandatory, simply too many "ununiform/volatile" data can be nicely turned/transformed into such checksums:

SHA3-512("The quick brown fox jumps over the lazy dog")
D135BB84D0439DBAC432247EE573A23EA7D3C9DEB2A968EB31D47C4FB45F1EF4422D6C531B5B9BD6F449EBCC449EA94D0A8F05F62130FDA612DA53C79659F609

BLAKE2b-512("The quick brown fox jumps over the lazy dog")
A8ADD4BDDDFD93E4877D2746E62817B116364A1FA7BC148D95090BC7333B3673F82401CF7AA2E4CB1ECD90296E3F14CB5413F8ED77BE73045B13914CDCD6A918

Okay, allow me to share the resultant dumps, the testmachine being i5-7200u @3GHz, 8GB DDR4 2133MHz.

The lzbench output:

Code:

C:\TEXTUAL_MADNESS_bare-minimum>lzbench173 -c4 -i1,15 -o3 -etornado,16/csc,5/gipfeli/blosclz,9/brieflz/fastlz,2/lzo1b,999/lz4fast,1,99/lz4/lz4hc,10,12/lizard,19,29,39,49/lzf,1/lzfse/lzrw,5/lzsse2,17/lzsse4,17/lzsse8,17/quicklz,3/slz_zlib,3/xpack,1,9/shrinker/crush,2/density,3 "Chunkerito.000,001"
lzbench 1.7.3 (64-bit Windows)   Assembled by P.Skibinski
Compressor name         Compress. Decompress.  Orig. size  Compr. size  Ratio Filename
memcpy                  10648 MB/s 11019 MB/s   314572800    314572800 100.00 Chunkerito.000,001
...
done... (cIters=1 dIters=15 cTime=1.0 dTime=2.0 chunkSize=1706MB cSpeed=0MB)

The results sorted by column number 4:
Compressor name         Compress. Decompress.  Orig. size  Compr. size  Ratio Filename
csc 2016-10-13 -5        2.17 MB/s    36 MB/s   314572800     96109141  30.55 Chunkerito.000,001
tornado 0.6a -16         1.55 MB/s    85 MB/s   314572800     99035211  31.48 Chunkerito.000,001
xpack 2016-06-02 -9        17 MB/s   658 MB/s   314572800    114247320  36.32 Chunkerito.000,001
lzfse 2017-03-08           44 MB/s   579 MB/s   314572800    116725958  37.11 Chunkerito.000,001
xpack 2016-06-02 -1       106 MB/s   617 MB/s   314572800    119701473  38.05 Chunkerito.000,001
density 0.12.5 beta -3    189 MB/s   304 MB/s   314572800    128601768  40.88 Chunkerito.000,001
crush 1.0 -2             0.68 MB/s   247 MB/s   314572800    132833413  42.23 Chunkerito.000,001
lzsse2 2016-05-14 -17    0.35 MB/s  2403 MB/s   314572800    135857197  43.19 Chunkerito.000,001
quicklz 1.5.0 -3           36 MB/s   663 MB/s   314572800    143497637  45.62 Chunkerito.000,001
lzsse4 2016-05-14 -17    0.34 MB/s  1999 MB/s   314572800    145283631  46.18 Chunkerito.000,001
lzo1b 2.09 -999            17 MB/s   577 MB/s   314572800    145335371  46.20 Chunkerito.000,001
lzsse8 2016-05-14 -17    0.35 MB/s  2475 MB/s   314572800    145812467  46.35 Chunkerito.000,001
lzrw 15-Jul-1991 -5        83 MB/s   436 MB/s   314572800    151526530  48.17 Chunkerito.000,001
gipfeli 2016-07-13        187 MB/s   299 MB/s   314572800    152543981  48.49 Chunkerito.000,001
brieflz 1.1.0              93 MB/s   147 MB/s   314572800    154775813  49.20 Chunkerito.000,001
lz4hc 1.8.0 -12          2.51 MB/s  1794 MB/s   314572800    155682053  49.49 Chunkerito.000,001
lz4hc 1.8.0 -10            19 MB/s  1817 MB/s   314572800    155692105  49.49 Chunkerito.000,001
lizard 1.0 -19           2.25 MB/s  2127 MB/s   314572800    155748230  49.51 Chunkerito.000,001
lzf 3.6 -1                285 MB/s   605 MB/s   314572800    156013044  49.60 Chunkerito.000,001
lizard 1.0 -29           0.30 MB/s  1652 MB/s   314572800    156559886  49.77 Chunkerito.000,001
lizard 1.0 -39           2.00 MB/s  2184 MB/s   314572800    156641195  49.79 Chunkerito.000,001
fastlz 0.1 -2             287 MB/s   498 MB/s   314572800    158119905  50.26 Chunkerito.000,001
lizard 1.0 -49           0.30 MB/s  1501 MB/s   314572800    158166713  50.28 Chunkerito.000,001
blosclz 2015-11-10 -9     251 MB/s   646 MB/s   314572800    160165523  50.92 Chunkerito.000,001
slz_zlib 1.0.0 -3          23 MB/s   273 MB/s   314572800    177293934  56.36 Chunkerito.000,001
lz4 1.8.0                 697 MB/s  4968 MB/s   314572800    195442959  62.13 Chunkerito.000,001
lz4fast 1.8.0 -1          698 MB/s  4969 MB/s   314572800    195442959  62.13 Chunkerito.000,001
lz4fast 1.8.0 -99        1892 MB/s  4303 MB/s   314572800    271688230  86.37 Chunkerito.000,001
shrinker 0.1               56 MB/s  4950 MB/s   314572800    293099576  93.17 Chunkerito.000,001

The TurboBench output:

Code:

C:\TEXTUAL_MADNESS_bare-minimum>"turbobench_v18.05_-_build_04_May_2018.exe" "Chunkerito.000,001" -ememcpy/lz4,1,16/snappy_c/yappy/zlib,1,5,9/bzip2/lzlib,9d29fb273/lzham,4fb258:x4:d29/lzma,9d29:fb273:mf=bt4/libdeflate,1,5,12/oodle,19,49,89,112,114,116,118,129/lzturbo,19,12,10,29,22,20,39,32,30,49,59,59t2,59t4/zstd,1,5,12,22,22d29/lizard,11,19,21,29,31,39,41,49/brotli,1,5,11/brotli,11d29/lzma,9/chameleon,2/density,3/lzham,4/trle/bsc,3,6/zpaq,2,5 -I3 -J31 -k1 -B2G
TurboBench:  - Sun Jun 10 07:18:36 2018

      C Size  ratio%     C MB/s     D MB/s   Name            File
    85871011    27.3       0.31       0.31   zpaq 5                           Chunkerito.000,001
    87731624    27.9       1.12      47.38   lzturbo 49                       Chunkerito.000,001
    90969561    28.9       1.28    1704.88   lzturbo 39                       Chunkerito.000,001
    93322838    29.7       0.52    1248.07   zstd 22                          Chunkerito.000,001
    93324372    29.7       0.48    1245.61   zstd 22d29                       Chunkerito.000,001
    93659859    29.8       0.26     207.99   lzham 4fb258:x4:d29              Chunkerito.000,001
    94395990    30.0       0.45     203.24   brotli 11                        Chunkerito.000,001
    94448425    30.0       0.34     175.30   brotli 11d29                     Chunkerito.000,001
    95368570    30.3       0.60     206.86   lzham 4                          Chunkerito.000,001
    95762613    30.4       0.63      33.15   lzlib 9d29fb273                  Chunkerito.000,001
    96021020    30.5      19.15      14.52   bsc 3                            Chunkerito.000,001
    96625090    30.7       0.94      44.99   lzma 9                           Chunkerito.000,001
    96738102    30.8       0.46    1136.27   oodle 129                        Chunkerito.000,001
    96738102    30.8       0.54    1135.97   oodle 89                         Chunkerito.000,001
    96817005    30.8       6.55     480.71   libdeflate 12                    Chunkerito.000,001
    98387497    31.3     323.86    1438.53   zstd 1                           Chunkerito.000,001
   100242511    31.9      11.45      29.78   bzip2                            Chunkerito.000,001
   100636651    32.0       0.72      47.76   lzma 9d29:fb273:mf=bt4           Chunkerito.000,001
   102336138    32.5      15.21       6.61   bsc 6                            Chunkerito.000,001
   103754533    33.0       6.63      27.82   lzturbo 59                       Chunkerito.000,001
   103755353    33.0       6.92      36.70   lzturbo 59t2                     Chunkerito.000,001
   103756181    33.0       7.30      49.30   lzturbo 59t4                     Chunkerito.000,001
   103886690    33.0     193.81     392.57   brotli 1                         Chunkerito.000,001
   106441065    33.8       0.26     355.95   oodle 19                         Chunkerito.000,001
   109884410    34.9     133.06    1152.55   lizard 41                        Chunkerito.000,001
   110549867    35.1      70.33     720.28   zstd 5                           Chunkerito.000,001
   110599411    35.2      16.65     257.40   brotli 5                         Chunkerito.000,001
   111165601    35.3      13.51     372.55   zstd 12                          Chunkerito.000,001
   113195029    36.0      14.01     232.93   zlib 9                           Chunkerito.000,001
   113967252    36.2       0.32     940.81   lizard 49                        Chunkerito.000,001
   114165753    36.3      20.08     582.03   lzturbo 32                       Chunkerito.000,001
   114186709    36.3      30.25     231.85   zlib 5                           Chunkerito.000,001
   115810528    36.8      96.30     546.45   libdeflate 5                     Chunkerito.000,001
   116400475    37.0     157.89    1251.62   lzturbo 30                       Chunkerito.000,001
   117664639    37.4     138.39     586.06   libdeflate 1                     Chunkerito.000,001
   118908410    37.8     154.97    1251.58   lizard 31                        Chunkerito.000,001
   119552458    38.0      72.83     615.79   zlib 1                           Chunkerito.000,001
   128600598    40.9     275.55     329.62   density 3                        Chunkerito.000,001
   130011833    41.3       1.86      60.88   zpaq 2                           Chunkerito.000,001
   133596888    42.5       1.44     601.32   lzturbo 29                       Chunkerito.000,001
   134797023    42.9       2.34    1516.57   lizard 39                        Chunkerito.000,001
   154993669    49.3      22.35     999.67   lzturbo 22                       Chunkerito.000,001
   155681949    49.5       2.41    2164.26   lz4 16                           Chunkerito.000,001
   155698643    49.5       1.82    2532.30   lzturbo 19                       Chunkerito.000,001
   155748234    49.5       2.40    1995.89   lizard 19                        Chunkerito.000,001
   155768988    49.5       3.47    2498.18   oodle 49                         Chunkerito.000,001
   155980551    49.6       0.53    3790.44   oodle 116                        Chunkerito.000,001
   155983137    49.6       0.57    3791.82   oodle 118                        Chunkerito.000,001
   156559890    49.8       0.32    1863.56   lizard 29                        Chunkerito.000,001
   158655850    50.4      20.44    3763.55   oodle 114                        Chunkerito.000,001
   158709285    50.5      50.82    3086.17   lzturbo 12                       Chunkerito.000,001
   163308151    51.9      59.95    2478.10   oodle 112                        Chunkerito.000,001
   183469433    58.3      93.88    3143.97   yappy                            Chunkerito.000,001
   185920735    59.1     602.28    3446.96   lzturbo 10                       Chunkerito.000,001
   186352774    59.2     421.64    1570.19   snappy_c                         Chunkerito.000,001
   186378354    59.2     555.04    2515.19   lzturbo 20                       Chunkerito.000,001
   187531489    59.6     184.75    3668.44   lizard 11                        Chunkerito.000,001
   192251338    61.1     181.80    4073.30   lizard 21                        Chunkerito.000,001
   195442963    62.1     926.71    5665.63   lz4 1                            Chunkerito.000,001
   197385318    62.7    1577.53    1820.17   chameleon 2                      Chunkerito.000,001
   229165208    72.8     248.56    2461.85   trle                             Chunkerito.000,001
   314572804   100.0   10221.04    9422.30   memcpy                           Chunkerito.000,001

And the 'usual suspects' - PPMD, GZIP, 7zip, BSC, Zpaq, Zstd, RAZOR:

Code (Text):

C:\TEXTUAL_MADNESS_bare-minimum>timer64 PPMd_varI_rev2_Intel15_32bit.exe e -o6 -m256 -f"Chunkerito.000,001".O6.PPMd_varI "Chunkerito.000,001"
Fast PPMII compressor for textual data, variant I, Apr  3 2016
Chunkerito.000,001:314572800 >99746859, 2.13 bpb, used:252.7MB, speed: 3169 KB/

Kernel  Time =     0.406 =    0%
User    Time =    95.828 =   98%
Process Time =    96.234 =   98%    Virtual  Memory =    258 MB
Global  Time =    97.696 =  100%    Physical Memory =    259 MB

C:\TEXTUAL_MADNESS_bare-minimum>timer64 PPMd_varI_rev2_Intel15_32bit.exe e -o16 -m256 -f"Chunkerito.000,001".O16.PPMd_varI "Chunkerito.000,001"
Fast PPMII compressor for textual data, variant I, Apr  3 2016
Chunkerito.000,001:314572800 >98346467, 2.09 bpb, used: 11.2MB, speed: 3090 KB/

Kernel  Time =     0.328 =    0%
User    Time =    98.718 =   99%
Process Time =    99.046 =   99%    Virtual  Memory =    258 MB
Global  Time =    99.427 =  100%    Physical Memory =    259 MB

C:\TEXTUAL_MADNESS_bare-minimum>timer64 "7za_v16.04_x64.exe" a -tgzip -mx9 "Chunkerito.000,001".MX9.zip "Chunkerito.000,001"

7-Zip (a) [64] 16.04 : Copyright (c) 1999-2016 Igor Pavlov : 2016-10-04

Scanning the drive:
1 file, 314572800 bytes (300 MiB)

Creating archive: Chunkerito.000,001.MX9.zip

Items to compress: 1

Files read from disk: 1
Archive size: 96934873 bytes (93 MiB)
Everything is Ok

Kernel  Time =     0.140 =    0%
User    Time =   228.625 =   99%
Process Time =   228.765 =   99%    Virtual  Memory =      6 MB
Global  Time =   229.141 =  100%    Physical Memory =      8 MB

C:\TEXTUAL_MADNESS_bare-minimum>timer64 "7za_v16.04_x64.exe" a -t7z -mx9 -md=29 "Chunkerito.000,001".MX9Dict512.7z "Chunkerito.000,001"

7-Zip (a) [64] 16.04 : Copyright (c) 1999-2016 Igor Pavlov : 2016-10-04

Scanning the drive:
1 file, 314572800 bytes (300 MiB)

Creating archive: Chunkerito.000,001.MX9Dict512.7z

Items to compress: 1

Files read from disk: 1
Archive size: 96637109 bytes (93 MiB)
Everything is Ok

Kernel  Time =     2.093 =    0%
User    Time =   406.453 =  132%
Process Time =   408.546 =  133%    Virtual  Memory =   4177 MB
Global  Time =   307.107 =  100%    Physical Memory =   3226 MB

C:\TEXTUAL_MADNESS_bare-minimum>timer64 "xz_v5.2.3_x64.exe" -z -k -f -9 -e -v -v --lzma2=dict=512MiB --threads=1 "Chunkerito.000,001"
xz_v5.2.3_x64: Filter chain: --lzma2=dict=512MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
xz_v5.2.3_x64: 5,378 MiB of memory is required. The limiter is disabled.
xz_v5.2.3_x64: Decompression will need 513 MiB of memory.
Chunkerito.000,001 (1/1)
  100 %        92.1 MiB / 300.0 MiB = 0.307   776 KiB/s       6:36

Kernel  Time =     1.859 =    0%
User    Time =   391.625 =   98%
Process Time =   393.484 =   99%    Virtual  Memory =   5389 MB
Global  Time =   396.105 =  100%    Physical Memory =   2710 MB

C:\TEXTUAL_MADNESS_bare-minimum>timer64 "bsc_v3.1.0_x64.exe" e "Chunkerito.000,001" "Chunkerito.000,001".ST6Block512.bsc -b512 -m6 -cp -Tt
This is bsc, Block Sorting Compressor. Version 3.1.0. 8 July 2012.
Copyright (c) 2009-2012 Ilya Grebnov <[email protected]>.

Chunkerito.000,001 compressed 314572800 into 92270528 in 19.062 seconds.

Kernel  Time =     0.687 =    3%
User    Time =    17.671 =   92%
Process Time =    18.359 =   95%    Virtual  Memory =   1579 MB
Global  Time =    19.156 =  100%    Physical Memory =   1571 MB

C:\TEXTUAL_MADNESS_bare-minimum>timer64 "zpaq_v7.05_x64.exe" add "Chunkerito.000,001".method29.zpaq "Chunkerito.000,001" -method 29 -threads 1
zpaq v7.05 journaling archiver, compiled Apr 17 2015
Adding 314.572800 MB in 1 files -method 29 -threads 1 at 2018-06-11 04:22:31.
100.00% 0:00:00 + Chunkerito.000,001 314572800
100.00% 0:00:00 [1..4565] 314591068 -method 29,239,0
1 +added, 0 -removed.

0.000000 + (314.572800 -> 314.572800 -> 126.796303) = 126.796303 MB
198.641 seconds (all OK)

Kernel  Time =     0.687 =    0%
User    Time =   197.734 =   99%
Process Time =   198.421 =   99%    Virtual  Memory =   2169 MB
Global  Time =   198.655 =  100%    Physical Memory =   1892 MB

C:\TEXTUAL_MADNESS_bare-minimum>timer64 "zpaq_v7.05_x64.exe" add "Chunkerito.000,001".method59.zpaq "Chunkerito.000,001" -method 59 -threads 1
zpaq v7.05 journaling archiver, compiled Apr 17 2015
Adding 314.572800 MB in 1 files -method 59 -threads 1 at 2018-06-11 04:25:50.
100.00% 0:00:00 + Chunkerito.000,001 314572800
100.00% 0:00:00 [1..4565] 314591068 -method 59,239,0
1 +added, 0 -removed.

0.000000 + (314.572800 -> 314.572800 -> 85.992745) = 85.992745 MB
1026.625 seconds (all OK)

Kernel  Time =     1.765 =    0%
User    Time =  1024.328 =   99%
Process Time =  1026.093 =   99%    Virtual  Memory =   5568 MB
Global  Time =  1026.641 =  100%    Physical Memory =   4215 MB

C:\TEXTUAL_MADNESS_bare-minimum>timer64 zstd-v1.3.4-win64.exe --ultra -22 --zstd=wlog=29,clog=30,hlog=30,slog=26 "Chunkerito.000,001"
Chunkerito.000,001   : 29.67%   (314572800 => 93324464 bytes, Chunkerito.000,001.zst)

Kernel  Time =     4.843 =    0%
User    Time = 32158.031 =   99%
Process Time = 32162.875 =   99%    Virtual  Memory =  18479 MB
Global  Time = 32169.834 =  100%    Physical Memory =   7183 MB

C:\TEXTUAL_MADNESS_bare-minimum>timer64 rz_1.01.exe a -d 512M "Chunkerito.000,001".512M.rz "Chunkerito.000,001"

 *** RAZOR Archiver 1.01 (2017-09-14) - DEMO/TEST version ***
 *** (c) Christian Martelock ([email protected]) ***

 Scanning c:\textual_madness_bare-minimum\chunkerito.000,001
 Found 0 dirs, 1 files, 314572800 bytes.

 Creating archive Chunkerito.000,001.512M.rz
 Window : 307200K (2048M..512G)
 Header : 52
 Size   : 92853683

 Archive ok. Added 0 dirs, 1 files, 314572800 bytes.
 CPU time = 1273.953s / wall time = 879.841s

Kernel  Time =     3.140 =    0%
User    Time =  1273.953 =  144%
Process Time =  1277.093 =  145%    Virtual  Memory =   3770 MB
Global  Time =   879.943 =  100%    Physical Memory =   3260 MB

For more info and to download the package allowing replication (to run all the benches above):
The Zennish Microdeduplicator · Issue #1 · Sanmayce/Nakamichi

LzTurbo decompresses SHA1 sums 1704.88/0.31=5499x faster than Zpaq

eva2000 · Jun 11, 2018

nice.. maybe better if you started a new thread of your own in System Administration as your juicy results are buried in this thread on page 2 right now

Also for posting code or output from commands to keep the formatting, you might want to use CODEB tags for code as it has wider frame than CODE tag How to use forum BBCODE code tags

Log in / Register

Forums

Welcome to Centmin Mod Community

Sysadmin Compression Comparison Benchmarks: zstd vs brotli vs pigz vs bzip2 vs xz etc

eva2000 Administrator Staff Member

Brad Knowles New Member

eva2000 Administrator Staff Member

Brad Knowles New Member

eva2000 Administrator Staff Member

Sanmayce New Member

Sanmayce New Member

eva2000 Administrator Staff Member

.

Log in / Register

Useful Searches

Welcome to Centmin Mod Community

Sysadmin Compression Comparison Benchmarks: zstd vs brotli vs pigz vs bzip2 vs xz etc

eva2000 Administrator Staff Member

Brad Knowles New Member

eva2000 Administrator Staff Member

Brad Knowles New Member

eva2000 Administrator Staff Member

Sanmayce New Member

Sanmayce New Member

eva2000 Administrator Staff Member

.