Thank you for visiting this page, this page has been update in another link Parallel compression utilities on Linux
In the article of Compression tools on linux, I compared 4 different compression utilities which using different compression algorithms. Here I'll compare 3 popular parallel compression utilities. lbzip2, pbzip2 and pigz. lbzip2 and pbzip2 uses the same compression algorithm as bzip's, while pigz uese zlib which gzip uses. Here are home pages of three compression utilities. http://lbzip2.org/ http://compression.ca/pbzip2/ http://zlib.net/pigz/ The test bed was on the machine that has 2 sockets, 8 Intel(R) Core(TM) i7 CPUs 870 @ 2.93GHz 16GB memory, 2 sets of mirrored local SATA disks. Linux kernel 2.6.32-358.18.1.el6.x86_64 lbzip2-2.2-1.el6.x86_64 pbzip2-1.1.6-1.el6.x86_64 pigz-2.2.5-1.el6.x86_64 Even though there is no disk bottle nect, I load file into memory before each test, so that there is no io concern at all. Each
tool utility has compression levels, I test level(fast) 1, 6(default)
and 9(best). The ratio is the percentage reduction for each file
compressed or decompressed(compatible with gzip -v)
Realiability testThe second is to compare their stability. So, I used the following simple script.The
purpose is that for each compression utility, I run 100 times compress
and uncompress. with mdsum and filesize check. echo "$tool compress start" for i in `seq 1 100` { echo "test No. $i" cp -f ${testfile}.orig $testfile rm -f ${testfile}.bz2 ./to_mem $testfile time -p $tool -v -9 $testfile ls -l ${testfile}.bz2 >>${tool}.chk md5sum -b ${testfile}.bz2 >>${tool}.chk time -p $tool -v -d ${testfile}.bz2 md5sum -b $testfile >>${tool}.chk sleep 3 } echo "$tool tests end" test results are from 100 round tests
Compatibility testlbzip2 and pbzip2 use the same compression algorithm, so I also cross tested format compatibilities of bzip2, pbzip2 and lbzip2, they are all good, all passed md5sum file check.As for pigz, there is one interesting thing, every time it yields out different compressed file, same file size, but md5sum, it does can recover the compressed file back to origional file with correct md5sum. I tested it with small files, I don't see this type of 'issue'. As for compability with gzip, I don't see any issue, all good. Note: I don't have any parallel compression utility which use LZMA algothrim, xz looks is heading to this. |