No I haven't specifically for pwned pasword database. Though given it's like MySQL dtaabase sql backups being purely texted base, the compression rations might be similar. And well really depends on server hardware/cpu core count if you're focusing on multi-threaded compression tools.
File contents will obviously make a huge difference. The file I referenced contains SHA1 hashes of passwords that have been exposed in one or more breaches, plus the count of the number of breaches in which that password was exposed. For example: $ head pwned-passwords-ordered-2.0.txt 000000005AD76BD555C1D6D771DE417A4B87E4B4:3 00000000A8DAE4228F821FB418F59826079BF368:2 00000000DD7F2A1C68A35673713783CA390C9E93:630 00000001E225B908BAC31C56DB04D892E47536E0:5 00000006BAB7FC3113AA73DE3589630FC08218E7:1 00000008CD1806EB7B9B46A8F87690B2AC16F617:3 0000000A0E3B9F25FF41DE4B5AC238C2D545C7A8:14 0000000A1D4B746FAA3FD526FF6D5BC8052FDB38:16 0000000CAEF405439D57847A8657218C618160B2:15 0000000FC1C08E6454BED24F463EA2129E254D43:40 Sure, it's technically a text file, and those SHA1 hashes can easily be represented in a binary format instead, but other than that, this is going to be a hard nut to crack. Of course, this file is also 29GB in size, uncompressed. But you could instead use pwned-passwords-update-1.txt which was the first update after the first full version of this file was produced, and that update is only 548MB in size. But I still think that files of this nature would make a good addition to the tests. Thanks!
Unfortunately at 29GB uncompressed size it would take too long to test especially for all the single threaded compression tools for such a test heh. Maybe if I had more free time - unfortunately not right now
Yeah, I figured a 29GB file would be too big. That's why I also mentioned the much smaller 548MB file. Anyway, i's just a thought. You're already doing great work for the community, and I'm sure you don't get anywhere near enough credit or thanks!
Hi, me too wanna see how those hashes are ... decompressed. >Some folks will know I have a strong focus on performance and efficiency. Same here. Before running the first 300MB chunk of 'ordered by the hash' file here is the two files. If you wanna see smaller excerpt then just let me know, I chose 300 as it fits in my 8GB laptop's RAM without using virtual such. Gonna run latest TurboBench and latest lzbench along with RAZOR, PPMD, Zstd v1.3.4, Zpaq... Code: E:\Downloads\z>dir Volume in drive E is Kaze_gold Volume Serial Number is 4C42-B2C5 Directory of E:\Downloads\z 06/09/2018 05:41 PM <DIR> . 06/09/2018 05:41 PM <DIR> .. 03/17/2018 12:15 AM 12,895 Chunkerito.c 03/17/2018 12:15 AM 62,976 Chunkerito.exe 04/26/2018 11:37 AM <DIR> Get_The_First_300MB_chunk 04/26/2018 11:38 AM 30 Get_The_First_300MB_chunk.bat 04/26/2018 12:14 PM 38,741 Get_The_First_300MB_chunk.zip 02/17/2018 06:13 AM 31,603,121,046 haveibeenpwned.com_Passwords_pwned-passwords-ordered-2.0.txt 03/29/2018 12:49 AM 1,631 MokujIN GREEN 224 prompt.lnk 6 File(s) 31,603,237,319 bytes 3 Dir(s) 176,337,924,096 bytes free E:\Downloads\z>Get_The_First_300MB_chunk haveibeenpwned.com_Passwords_pwned-passwords-ordered-2.0.txt Chunkerito, revision 1+, written by Kaze. Purpose: To chunkize/split any file to 'ChunkSize' long chunks. Usage: Chunker filename ChunkSize Note: For 128MB chunks use ChunkSize = 134217728 Size of Input TEXTual file: 31,603,121,046 ^CTerminate batch job (Y/N)? y... E:\Downloads\z>dir Volume in drive E is Kaze_gold Volume Serial Number is 4C42-B2C5 Directory of E:\Downloads\z 06/09/2018 05:42 PM <DIR> . 06/09/2018 05:42 PM <DIR> .. 06/09/2018 05:41 PM 314,572,800 Chunkerito.000,001 06/09/2018 05:42 PM 314,572,800 Chunkerito.000,002 06/09/2018 05:42 PM 314,572,800 Chunkerito.000,003 03/17/2018 12:15 AM 12,895 Chunkerito.c 03/17/2018 12:15 AM 62,976 Chunkerito.exe 04/26/2018 11:37 AM <DIR> Get_The_First_300MB_chunk 04/26/2018 11:38 AM 30 Get_The_First_300MB_chunk.bat 04/26/2018 12:14 PM 38,741 Get_The_First_300MB_chunk.zip 02/17/2018 06:13 AM 31,603,121,046 haveibeenpwned.com_Passwords_pwned-passwords-ordered-2.0.txt 03/29/2018 12:49 AM 1,631 MokujIN GREEN 224 prompt.lnk 9 File(s) 32,546,955,719 bytes 3 Dir(s) 175,394,205,696 bytes free E:\Downloads\z>type "Chunkerito.000,001"|more 000000005AD76BD555C1D6D771DE417A4B87E4B4:3 00000000A8DAE4228F821FB418F59826079BF368:2 00000000DD7F2A1C68A35673713783CA390C9E93:630 00000001E225B908BAC31C56DB04D892E47536E0:5 00000006BAB7FC3113AA73DE3589630FC08218E7:1 00000008CD1806EB7B9B46A8F87690B2AC16F617:3 0000000A0E3B9F25FF41DE4B5AC238C2D545C7A8:14 0000000A1D4B746FAA3FD526FF6D5BC8052FDB38:16 0000000CAEF405439D57847A8657218C618160B2:15 0000000FC1C08E6454BED24F463EA2129E254D43:40 00000010F4B38525354491E099EB1796278544B1:3 0000001225719E346D78D6F493E79FD6EF307A35:2 0000001306E7C20BE7E0B28B0E8D87EFC634479B:2 00000016C6C075173C163757BCEA8139D4CC69CF:4 0000002363B67EAA39B4413802FC10BAC2D0C786:2 00000028AE3ED9625B6BAD92668069843DB21916:2 0000002A9AF2DF2267ED92A9FF7FE16E62116CB1:3 00000030AD61704DDB0AF06B1A64865CA479C1B6:15 0000003C100967F8714E38BAE61B4DA1B957B947:1 0000003F2785CA62D59AB905EEAB3533EFFE337A:15 0000003FA0BC80B317DDE176D6A71F6321CCD35E:2 000000405907BE8972A2FF97E9A757DDE3BE8EE0:5 00000042F053B3F16733DFB83D431126D64331FC:6 00000045ED258F4303903E6AA188652E89D335AD:1 0000004BDFB224DDAC4CC3FB0FC1A52890354ACC:7 -- More --
To reiterate, the [de]compressing of SHA1 lines is quite interesting and in some cases, I daresay, mandatory, simply too many "ununiform/volatile" data can be nicely turned/transformed into such checksums: SHA3-512("The quick brown fox jumps over the lazy dog") D135BB84D0439DBAC432247EE573A23EA7D3C9DEB2A968EB31D47C4FB45F1EF4422D6C531B5B9BD6F449EBCC449EA94D0A8F05F62130FDA612DA53C79659F609 BLAKE2b-512("The quick brown fox jumps over the lazy dog") A8ADD4BDDDFD93E4877D2746E62817B116364A1FA7BC148D95090BC7333B3673F82401CF7AA2E4CB1ECD90296E3F14CB5413F8ED77BE73045B13914CDCD6A918 Okay, allow me to share the resultant dumps, the testmachine being i5-7200u @3GHz, 8GB DDR4 2133MHz. The lzbench output: Code: C:\TEXTUAL_MADNESS_bare-minimum>lzbench173 -c4 -i1,15 -o3 -etornado,16/csc,5/gipfeli/blosclz,9/brieflz/fastlz,2/lzo1b,999/lz4fast,1,99/lz4/lz4hc,10,12/lizard,19,29,39,49/lzf,1/lzfse/lzrw,5/lzsse2,17/lzsse4,17/lzsse8,17/quicklz,3/slz_zlib,3/xpack,1,9/shrinker/crush,2/density,3 "Chunkerito.000,001" lzbench 1.7.3 (64-bit Windows) Assembled by P.Skibinski Compressor name Compress. Decompress. Orig. size Compr. size Ratio Filename memcpy 10648 MB/s 11019 MB/s 314572800 314572800 100.00 Chunkerito.000,001 ... done... (cIters=1 dIters=15 cTime=1.0 dTime=2.0 chunkSize=1706MB cSpeed=0MB) The results sorted by column number 4: Compressor name Compress. Decompress. Orig. size Compr. size Ratio Filename csc 2016-10-13 -5 2.17 MB/s 36 MB/s 314572800 96109141 30.55 Chunkerito.000,001 tornado 0.6a -16 1.55 MB/s 85 MB/s 314572800 99035211 31.48 Chunkerito.000,001 xpack 2016-06-02 -9 17 MB/s 658 MB/s 314572800 114247320 36.32 Chunkerito.000,001 lzfse 2017-03-08 44 MB/s 579 MB/s 314572800 116725958 37.11 Chunkerito.000,001 xpack 2016-06-02 -1 106 MB/s 617 MB/s 314572800 119701473 38.05 Chunkerito.000,001 density 0.12.5 beta -3 189 MB/s 304 MB/s 314572800 128601768 40.88 Chunkerito.000,001 crush 1.0 -2 0.68 MB/s 247 MB/s 314572800 132833413 42.23 Chunkerito.000,001 lzsse2 2016-05-14 -17 0.35 MB/s 2403 MB/s 314572800 135857197 43.19 Chunkerito.000,001 quicklz 1.5.0 -3 36 MB/s 663 MB/s 314572800 143497637 45.62 Chunkerito.000,001 lzsse4 2016-05-14 -17 0.34 MB/s 1999 MB/s 314572800 145283631 46.18 Chunkerito.000,001 lzo1b 2.09 -999 17 MB/s 577 MB/s 314572800 145335371 46.20 Chunkerito.000,001 lzsse8 2016-05-14 -17 0.35 MB/s 2475 MB/s 314572800 145812467 46.35 Chunkerito.000,001 lzrw 15-Jul-1991 -5 83 MB/s 436 MB/s 314572800 151526530 48.17 Chunkerito.000,001 gipfeli 2016-07-13 187 MB/s 299 MB/s 314572800 152543981 48.49 Chunkerito.000,001 brieflz 1.1.0 93 MB/s 147 MB/s 314572800 154775813 49.20 Chunkerito.000,001 lz4hc 1.8.0 -12 2.51 MB/s 1794 MB/s 314572800 155682053 49.49 Chunkerito.000,001 lz4hc 1.8.0 -10 19 MB/s 1817 MB/s 314572800 155692105 49.49 Chunkerito.000,001 lizard 1.0 -19 2.25 MB/s 2127 MB/s 314572800 155748230 49.51 Chunkerito.000,001 lzf 3.6 -1 285 MB/s 605 MB/s 314572800 156013044 49.60 Chunkerito.000,001 lizard 1.0 -29 0.30 MB/s 1652 MB/s 314572800 156559886 49.77 Chunkerito.000,001 lizard 1.0 -39 2.00 MB/s 2184 MB/s 314572800 156641195 49.79 Chunkerito.000,001 fastlz 0.1 -2 287 MB/s 498 MB/s 314572800 158119905 50.26 Chunkerito.000,001 lizard 1.0 -49 0.30 MB/s 1501 MB/s 314572800 158166713 50.28 Chunkerito.000,001 blosclz 2015-11-10 -9 251 MB/s 646 MB/s 314572800 160165523 50.92 Chunkerito.000,001 slz_zlib 1.0.0 -3 23 MB/s 273 MB/s 314572800 177293934 56.36 Chunkerito.000,001 lz4 1.8.0 697 MB/s 4968 MB/s 314572800 195442959 62.13 Chunkerito.000,001 lz4fast 1.8.0 -1 698 MB/s 4969 MB/s 314572800 195442959 62.13 Chunkerito.000,001 lz4fast 1.8.0 -99 1892 MB/s 4303 MB/s 314572800 271688230 86.37 Chunkerito.000,001 shrinker 0.1 56 MB/s 4950 MB/s 314572800 293099576 93.17 Chunkerito.000,001 The TurboBench output: Code: C:\TEXTUAL_MADNESS_bare-minimum>"turbobench_v18.05_-_build_04_May_2018.exe" "Chunkerito.000,001" -ememcpy/lz4,1,16/snappy_c/yappy/zlib,1,5,9/bzip2/lzlib,9d29fb273/lzham,4fb258:x4:d29/lzma,9d29:fb273:mf=bt4/libdeflate,1,5,12/oodle,19,49,89,112,114,116,118,129/lzturbo,19,12,10,29,22,20,39,32,30,49,59,59t2,59t4/zstd,1,5,12,22,22d29/lizard,11,19,21,29,31,39,41,49/brotli,1,5,11/brotli,11d29/lzma,9/chameleon,2/density,3/lzham,4/trle/bsc,3,6/zpaq,2,5 -I3 -J31 -k1 -B2G TurboBench: - Sun Jun 10 07:18:36 2018 C Size ratio% C MB/s D MB/s Name File 85871011 27.3 0.31 0.31 zpaq 5 Chunkerito.000,001 87731624 27.9 1.12 47.38 lzturbo 49 Chunkerito.000,001 90969561 28.9 1.28 1704.88 lzturbo 39 Chunkerito.000,001 93322838 29.7 0.52 1248.07 zstd 22 Chunkerito.000,001 93324372 29.7 0.48 1245.61 zstd 22d29 Chunkerito.000,001 93659859 29.8 0.26 207.99 lzham 4fb258:x4:d29 Chunkerito.000,001 94395990 30.0 0.45 203.24 brotli 11 Chunkerito.000,001 94448425 30.0 0.34 175.30 brotli 11d29 Chunkerito.000,001 95368570 30.3 0.60 206.86 lzham 4 Chunkerito.000,001 95762613 30.4 0.63 33.15 lzlib 9d29fb273 Chunkerito.000,001 96021020 30.5 19.15 14.52 bsc 3 Chunkerito.000,001 96625090 30.7 0.94 44.99 lzma 9 Chunkerito.000,001 96738102 30.8 0.46 1136.27 oodle 129 Chunkerito.000,001 96738102 30.8 0.54 1135.97 oodle 89 Chunkerito.000,001 96817005 30.8 6.55 480.71 libdeflate 12 Chunkerito.000,001 98387497 31.3 323.86 1438.53 zstd 1 Chunkerito.000,001 100242511 31.9 11.45 29.78 bzip2 Chunkerito.000,001 100636651 32.0 0.72 47.76 lzma 9d29:fb273:mf=bt4 Chunkerito.000,001 102336138 32.5 15.21 6.61 bsc 6 Chunkerito.000,001 103754533 33.0 6.63 27.82 lzturbo 59 Chunkerito.000,001 103755353 33.0 6.92 36.70 lzturbo 59t2 Chunkerito.000,001 103756181 33.0 7.30 49.30 lzturbo 59t4 Chunkerito.000,001 103886690 33.0 193.81 392.57 brotli 1 Chunkerito.000,001 106441065 33.8 0.26 355.95 oodle 19 Chunkerito.000,001 109884410 34.9 133.06 1152.55 lizard 41 Chunkerito.000,001 110549867 35.1 70.33 720.28 zstd 5 Chunkerito.000,001 110599411 35.2 16.65 257.40 brotli 5 Chunkerito.000,001 111165601 35.3 13.51 372.55 zstd 12 Chunkerito.000,001 113195029 36.0 14.01 232.93 zlib 9 Chunkerito.000,001 113967252 36.2 0.32 940.81 lizard 49 Chunkerito.000,001 114165753 36.3 20.08 582.03 lzturbo 32 Chunkerito.000,001 114186709 36.3 30.25 231.85 zlib 5 Chunkerito.000,001 115810528 36.8 96.30 546.45 libdeflate 5 Chunkerito.000,001 116400475 37.0 157.89 1251.62 lzturbo 30 Chunkerito.000,001 117664639 37.4 138.39 586.06 libdeflate 1 Chunkerito.000,001 118908410 37.8 154.97 1251.58 lizard 31 Chunkerito.000,001 119552458 38.0 72.83 615.79 zlib 1 Chunkerito.000,001 128600598 40.9 275.55 329.62 density 3 Chunkerito.000,001 130011833 41.3 1.86 60.88 zpaq 2 Chunkerito.000,001 133596888 42.5 1.44 601.32 lzturbo 29 Chunkerito.000,001 134797023 42.9 2.34 1516.57 lizard 39 Chunkerito.000,001 154993669 49.3 22.35 999.67 lzturbo 22 Chunkerito.000,001 155681949 49.5 2.41 2164.26 lz4 16 Chunkerito.000,001 155698643 49.5 1.82 2532.30 lzturbo 19 Chunkerito.000,001 155748234 49.5 2.40 1995.89 lizard 19 Chunkerito.000,001 155768988 49.5 3.47 2498.18 oodle 49 Chunkerito.000,001 155980551 49.6 0.53 3790.44 oodle 116 Chunkerito.000,001 155983137 49.6 0.57 3791.82 oodle 118 Chunkerito.000,001 156559890 49.8 0.32 1863.56 lizard 29 Chunkerito.000,001 158655850 50.4 20.44 3763.55 oodle 114 Chunkerito.000,001 158709285 50.5 50.82 3086.17 lzturbo 12 Chunkerito.000,001 163308151 51.9 59.95 2478.10 oodle 112 Chunkerito.000,001 183469433 58.3 93.88 3143.97 yappy Chunkerito.000,001 185920735 59.1 602.28 3446.96 lzturbo 10 Chunkerito.000,001 186352774 59.2 421.64 1570.19 snappy_c Chunkerito.000,001 186378354 59.2 555.04 2515.19 lzturbo 20 Chunkerito.000,001 187531489 59.6 184.75 3668.44 lizard 11 Chunkerito.000,001 192251338 61.1 181.80 4073.30 lizard 21 Chunkerito.000,001 195442963 62.1 926.71 5665.63 lz4 1 Chunkerito.000,001 197385318 62.7 1577.53 1820.17 chameleon 2 Chunkerito.000,001 229165208 72.8 248.56 2461.85 trle Chunkerito.000,001 314572804 100.0 10221.04 9422.30 memcpy Chunkerito.000,001 And the 'usual suspects' - PPMD, GZIP, 7zip, BSC, Zpaq, Zstd, RAZOR: Code (Text): C:\TEXTUAL_MADNESS_bare-minimum>timer64 PPMd_varI_rev2_Intel15_32bit.exe e -o6 -m256 -f"Chunkerito.000,001".O6.PPMd_varI "Chunkerito.000,001" Fast PPMII compressor for textual data, variant I, Apr 3 2016 Chunkerito.000,001:314572800 >99746859, 2.13 bpb, used:252.7MB, speed: 3169 KB/ Kernel Time = 0.406 = 0% User Time = 95.828 = 98% Process Time = 96.234 = 98% Virtual Memory = 258 MB Global Time = 97.696 = 100% Physical Memory = 259 MB C:\TEXTUAL_MADNESS_bare-minimum>timer64 PPMd_varI_rev2_Intel15_32bit.exe e -o16 -m256 -f"Chunkerito.000,001".O16.PPMd_varI "Chunkerito.000,001" Fast PPMII compressor for textual data, variant I, Apr 3 2016 Chunkerito.000,001:314572800 >98346467, 2.09 bpb, used: 11.2MB, speed: 3090 KB/ Kernel Time = 0.328 = 0% User Time = 98.718 = 99% Process Time = 99.046 = 99% Virtual Memory = 258 MB Global Time = 99.427 = 100% Physical Memory = 259 MB C:\TEXTUAL_MADNESS_bare-minimum>timer64 "7za_v16.04_x64.exe" a -tgzip -mx9 "Chunkerito.000,001".MX9.zip "Chunkerito.000,001" 7-Zip (a) [64] 16.04 : Copyright (c) 1999-2016 Igor Pavlov : 2016-10-04 Scanning the drive: 1 file, 314572800 bytes (300 MiB) Creating archive: Chunkerito.000,001.MX9.zip Items to compress: 1 Files read from disk: 1 Archive size: 96934873 bytes (93 MiB) Everything is Ok Kernel Time = 0.140 = 0% User Time = 228.625 = 99% Process Time = 228.765 = 99% Virtual Memory = 6 MB Global Time = 229.141 = 100% Physical Memory = 8 MB C:\TEXTUAL_MADNESS_bare-minimum>timer64 "7za_v16.04_x64.exe" a -t7z -mx9 -md=29 "Chunkerito.000,001".MX9Dict512.7z "Chunkerito.000,001" 7-Zip (a) [64] 16.04 : Copyright (c) 1999-2016 Igor Pavlov : 2016-10-04 Scanning the drive: 1 file, 314572800 bytes (300 MiB) Creating archive: Chunkerito.000,001.MX9Dict512.7z Items to compress: 1 Files read from disk: 1 Archive size: 96637109 bytes (93 MiB) Everything is Ok Kernel Time = 2.093 = 0% User Time = 406.453 = 132% Process Time = 408.546 = 133% Virtual Memory = 4177 MB Global Time = 307.107 = 100% Physical Memory = 3226 MB C:\TEXTUAL_MADNESS_bare-minimum>timer64 "xz_v5.2.3_x64.exe" -z -k -f -9 -e -v -v --lzma2=dict=512MiB --threads=1 "Chunkerito.000,001" xz_v5.2.3_x64: Filter chain: --lzma2=dict=512MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0 xz_v5.2.3_x64: 5,378 MiB of memory is required. The limiter is disabled. xz_v5.2.3_x64: Decompression will need 513 MiB of memory. Chunkerito.000,001 (1/1) 100 % 92.1 MiB / 300.0 MiB = 0.307 776 KiB/s 6:36 Kernel Time = 1.859 = 0% User Time = 391.625 = 98% Process Time = 393.484 = 99% Virtual Memory = 5389 MB Global Time = 396.105 = 100% Physical Memory = 2710 MB C:\TEXTUAL_MADNESS_bare-minimum>timer64 "bsc_v3.1.0_x64.exe" e "Chunkerito.000,001" "Chunkerito.000,001".ST6Block512.bsc -b512 -m6 -cp -Tt This is bsc, Block Sorting Compressor. Version 3.1.0. 8 July 2012. Copyright (c) 2009-2012 Ilya Grebnov <Ilya.Grebnov@gmail.com>. Chunkerito.000,001 compressed 314572800 into 92270528 in 19.062 seconds. Kernel Time = 0.687 = 3% User Time = 17.671 = 92% Process Time = 18.359 = 95% Virtual Memory = 1579 MB Global Time = 19.156 = 100% Physical Memory = 1571 MB C:\TEXTUAL_MADNESS_bare-minimum>timer64 "zpaq_v7.05_x64.exe" add "Chunkerito.000,001".method29.zpaq "Chunkerito.000,001" -method 29 -threads 1 zpaq v7.05 journaling archiver, compiled Apr 17 2015 Adding 314.572800 MB in 1 files -method 29 -threads 1 at 2018-06-11 04:22:31. 100.00% 0:00:00 + Chunkerito.000,001 314572800 100.00% 0:00:00 [1..4565] 314591068 -method 29,239,0 1 +added, 0 -removed. 0.000000 + (314.572800 -> 314.572800 -> 126.796303) = 126.796303 MB 198.641 seconds (all OK) Kernel Time = 0.687 = 0% User Time = 197.734 = 99% Process Time = 198.421 = 99% Virtual Memory = 2169 MB Global Time = 198.655 = 100% Physical Memory = 1892 MB C:\TEXTUAL_MADNESS_bare-minimum>timer64 "zpaq_v7.05_x64.exe" add "Chunkerito.000,001".method59.zpaq "Chunkerito.000,001" -method 59 -threads 1 zpaq v7.05 journaling archiver, compiled Apr 17 2015 Adding 314.572800 MB in 1 files -method 59 -threads 1 at 2018-06-11 04:25:50. 100.00% 0:00:00 + Chunkerito.000,001 314572800 100.00% 0:00:00 [1..4565] 314591068 -method 59,239,0 1 +added, 0 -removed. 0.000000 + (314.572800 -> 314.572800 -> 85.992745) = 85.992745 MB 1026.625 seconds (all OK) Kernel Time = 1.765 = 0% User Time = 1024.328 = 99% Process Time = 1026.093 = 99% Virtual Memory = 5568 MB Global Time = 1026.641 = 100% Physical Memory = 4215 MB C:\TEXTUAL_MADNESS_bare-minimum>timer64 zstd-v1.3.4-win64.exe --ultra -22 --zstd=wlog=29,clog=30,hlog=30,slog=26 "Chunkerito.000,001" Chunkerito.000,001 : 29.67% (314572800 => 93324464 bytes, Chunkerito.000,001.zst) Kernel Time = 4.843 = 0% User Time = 32158.031 = 99% Process Time = 32162.875 = 99% Virtual Memory = 18479 MB Global Time = 32169.834 = 100% Physical Memory = 7183 MB C:\TEXTUAL_MADNESS_bare-minimum>timer64 rz_1.01.exe a -d 512M "Chunkerito.000,001".512M.rz "Chunkerito.000,001" *** RAZOR Archiver 1.01 (2017-09-14) - DEMO/TEST version *** *** (c) Christian Martelock (christian.martelock@web.de) *** Scanning c:\textual_madness_bare-minimum\chunkerito.000,001 Found 0 dirs, 1 files, 314572800 bytes. Creating archive Chunkerito.000,001.512M.rz Window : 307200K (2048M..512G) Header : 52 Size : 92853683 Archive ok. Added 0 dirs, 1 files, 314572800 bytes. CPU time = 1273.953s / wall time = 879.841s Kernel Time = 3.140 = 0% User Time = 1273.953 = 144% Process Time = 1277.093 = 145% Virtual Memory = 3770 MB Global Time = 879.943 = 100% Physical Memory = 3260 MB For more info and to download the package allowing replication (to run all the benches above): The Zennish Microdeduplicator · Issue #1 · Sanmayce/Nakamichi LzTurbo decompresses SHA1 sums 1704.88/0.31=5499x faster than Zpaq
nice.. maybe better if you started a new thread of your own in System Administration as your juicy results are buried in this thread on page 2 right now Also for posting code or output from commands to keep the formatting, you might want to use CODEB tags for code as it has wider frame than CODE tag How to use forum BBCODE code tags
FYI, folks wanting to use zstd compression can also check out my custom tar RPM build with native zstd support at Beta Branch - CentOS 7.x - Custom Tar Archiver RPM Build With Facebook Zstd Compression Support