Featured Multi-threaded compression tools (pigz, pbzip2, lbzip2, plzip, p7zip)

yup

Yeah can't be beat for compression ratios ! But uses alot of memory.

---

 

all multi-threaded tools use more cpu as they use more threads in parallel hence the speed

but yes memory = ram

yeah you have alot of idle memory on that 32GB server heh

 

faster compression & better compression ratios = more cpu and memory usage nothing you can do about that really

full table and analysis of which is best is at gziptest.sh part 2: multi-threaded compression benchmarks | vbtechsupport.com

 

lbzip2 manual -1 to -9 is block size for compression with -9 being default so leaving -5 off will use -9 AFAIK from the manual LBZIP2

maybe use -1 -s -u together ?

also reduce number of threads used

 

Just re-did some compression tests on tar file as I added xz and pxz compression tools into the mix and tested levels 1 to 6 for each. Trying out new table BBCODE on forums

On

• Xeon E3-1240v3
• 32GB RAM
• 250GB Samsung 840 EVO SSD

Tar file compression test

Code:

-rw-r--r--   1 root root 785M Feb 24 02:38 ccache.tar

method	comp lvl	compress time	decompress time	compress cpu %	compress max mem	compress cs waits	decompress cpu %	decompress max mem	decompress cs waits	compression ratio	compress size
zip lvl 1	1	11.46	5.64	100	4768	4	100	7328	2	36	290094
zip lvl 2	2	12.19	5.50	100	4784	4	99	7328	2	35	283037
zip lvl 3	3	15.69	5.50	100	4768	3	100	7312	2	34	278277
zip lvl 4	4	15.42	5.52	99	4784	4	100	7328	2	33	270516
zip lvl 5	5	20.85	5.40	100	4768	6	99	7312	2	32	264760
zip lvl 6	6	33.53	5.34	100	4736	10	100	7328	2	32	262171
gzip lvl 1	1	17.17	6.71	100	3248	6	100	2736	3	35	284476
gzip lvl 2	2	17.90	6.57	100	3232	5	100	2736	3	34	277441
gzip lvl 3	3	20.98	6.51	100	3232	5	100	2736	3	34	273699
gzip lvl 4	4	21.16	6.49	100	3232	7	99	2752	3	33	266843
gzip lvl 5	5	26.53	6.36	100	3232	6	100	2752	3	32	261646
gzip lvl 6	6	38.75	6.30	99	3216	10	100	2736	3	32	259173
bzip2 lvl 1	1	55.64	22.00	100	6832	7	99	4224	4	32	260668
bzip2 lvl 2	2	51.95	22.76	100	9968	2	99	5792	6	32	259629
bzip2 lvl 3	3	50.81	23.13	100	13088	4	99	7360	5	32	258250
bzip2 lvl 4	4	51.00	23.37	100	14656	4	99	8944	5	32	257101
bzip2 lvl 5	5	50.63	23.51	100	17424	5	99	10496	5	31	256247
bzip2 lvl 6	6	50.57	23.64	100	22032	4	99	12080	6	31	255263
pigz lvl 1	1	2.98	2.96	676	36640	11571	143	3760	59280	35	284715
pigz lvl 2	2	3.16	2.86	668	36784	12844	145	3760	59049	34	277661
pigz lvl 3	3	3.60	3.06	695	44800	12984	144	3760	59031	34	274043
pigz lvl 4	4	3.80	2.89	720	44672	13254	145	3760	58857	33	267282
pigz lvl 5	5	4.64	2.74	756	44768	14935	148	3760	58544	32	262084
pigz lvl 6	6	6.54	2.69	767	44880	16210	147	3760	58518	32	259653
pbzip2 lvl 1	1	10.47	3.95	785	118320	1969	783	116272	1864	32	260796
pbzip2 lvl 2	2	9.97	4.05	783	139680	2210	788	123568	1976	32	259695
pbzip2 lvl 3	3	9.88	4.19	786	160304	2094	785	135936	1927	32	258607
pbzip2 lvl 4	4	10.08	4.35	787	182688	2227	786	145696	1952	32	257279
pbzip2 lvl 5	5	10.11	4.44	786	204656	2025	790	155920	2027	31	257002
pbzip2 lvl 6	6	10.23	4.77	784	225840	2043	773	187936	1979	31	256430
lzip lvl 1	1	80.62	18.00	100	51216	18	100	8224	4	28	224924
lzip lvl 2	2	96.34	17.42	100	75792	22	99	10272	3	27	217384
lzip lvl 3	3	127.35	16.91	100	96288	29	99	12320	7	26	209650
lzip lvl 4	4	197.52	16.38	100	145424	43	100	16416	4	23	190331
lzip lvl 5	5	280.52	15.73	100	186384	63	100	20512	4	22	177522
lzip lvl 6	6	354.63	15.06	100	366608	85	99	36896	6	20	166226
plzip lvl 1	1	17.97	8.78	788	557760	817	198	14688	18	28	225938
plzip lvl 2	2	20.13	8.54	786	879968	653	198	18768	14	27	219164
plzip lvl 3	3	25.54	8.29	782	1019840	448	199	22816	14	26	211730
plzip lvl 4	4	37.86	8.16	775	1564912	340	195	31008	17	24	193148
plzip lvl 5	5	54.27	7.74	771	2022752	259	197	39168	15	22	181550
plzip lvl 6	6	71.29	7.41	740	3951984	218	197	71920	15	21	170656
xz lvl 1	1	46.63	17.42	99	8848	8	99	4016	4	27	220122
xz lvl 2	2	49.01	15.94	99	22816	6	99	5824	4	25	204555
xz lvl 3	3	260.01	15.05	100	55312	40	99	7856	4	22	184442
xz lvl 4	4	278.69	14.56	100	102416	66	99	11968	4	22	178454
xz lvl 5	5	290.57	14.11	100	196640	67	99	20144	4	21	172868
xz lvl 6	6	337.30	13.57	100	385056	77	99	36544	3	20	165409
pxz lvl 1	1	12.36	17.62	776	41248	12414	99	3968	5	27	222876
pxz lvl 2	2	13.97	16.09	794	207024	232	99	5792	5	25	207139
pxz lvl 3	3	62.46	15.27	752	515952	354	99	7856	4	23	187392
pxz lvl 4	4	63.27	14.76	744	991088	326	99	11952	4	22	181383
pxz lvl 5	5	66.13	14.32	731	1941488	531	99	20160	4	21	175915
pxz lvl 6	6	76.44	13.79	717	3842016	301	99	36528	4	20	168650

 

heh always been pig-zee to me

 

they don't take up any cpu/mem resources just being installed so just leave them. There isn't really an uninstall routine for source installs

 

thanks for heads up just updated 123.09beta01 with the fix, so run cmupdate command to update local centmin mod code on your server

Code (Text):

cmupdate
No local changes to save
remote: Enumerating objects: 13, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (1/1), done.
remote: Total 5 (delta 4), reused 5 (delta 4), pack-reused 0
Unpacking objects: 100% (5/5), done.
From https://github.com/centminmod/centminmod
   697c6eb..e5f2c84  123.09beta01 -> origin/123.09beta01
Updating 697c6eb..e5f2c84
Fast-forward
 centmin.sh       | 2 +-
 inc/compress.inc | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

and then run centmin.sh menu option 17

 

compiles fine for me

 

strange I just tried again and works for me

what is output for these commands

Code (Text):

ls -lah /svr-setup | grep lbzip2

and does it work if you delete your existing lbzip2-2.5.tar.gz and retry centmin.sh menu option 17

Code (Text):

cd /svr-setup
rm /svr-setup/lbzip2-2.5*