hi i wondered about multithreaded rar archive file extraction on linux and after some googleing it seems that 7zip seems support multhithrading also also at least reading rars.
so I wondered if it would be possible to add an option to choose 7zip instead of unrar for linux so small server with multicore can benefit.
Especially since its the slowest bottleneck so far
(at least on my small server with raid / 16GB ram and atom quad core rar takes by far longest since it uses only one core and those atom boards don't have a big GHrz per CPU amount... multithreaded par runs fine so far so it would be nice and since the 16GB ram alsmot cache 100% of the download and the raid write speed isn't maxed out only the single core power is slowing the process)
linux rar 7zip support for multithreading
Re: linux rar 7zip support for multithreading
Did you test the speed differences?
I did:
versus
So no difference in real time.
I did:
Code: Select all
$ time unrar x *01.rar
UNRAR 5.00 beta 8 freeware Copyright (c) 1993-2013 Alexander Roshal
Extracting from bigrandomfile.part01.rar
<...>
All OK
real 0m13.926s
user 0m0.608s
sys 0m0.916s
Code: Select all
$ time 7z x *01.rar
7-Zip 9.20 Copyright (c) 1999-2010 Igor Pavlov 2010-11-18
p7zip Version 9.20 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,1 CPU)
Processing archive: bigrandomfile.part01.rar
Extracting random-300MB.bin
Everything is Ok
Size: 314572800
Compressed: 314575528
real 0m13.484s
user 0m1.200s
sys 0m0.708s
Re: linux rar 7zip support for multithreading
Sander: in the log 7zip says '1 CPU'.
Maybe it does improve with bigger file and that switch to enable more CPU?
Maybe it does improve with bigger file and that switch to enable more CPU?
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
Re: linux rar 7zip support for multithreading
Ah, good point; I was running it on my VPS. I'll find a multi-core system, with enough diskspace.safihre wrote:Sander: in the log 7zip says '1 CPU'.
Maybe it does improve with bigger file and that switch to enable more CPU?
On a dual-core Atom:
First create a 300MB random file, then create the rar files:
Code: Select all
dd if=/dev/urandom of=random-300MB.bin bs=1M count=300
rar a bigrandomfile -v10m -m0 random-300MB.bin
Code: Select all
sander@netbook:~/rar-kul$ time unrar x *01.rar
UNRAR 5.00 beta 8 freeware Copyright (c) 1993-2013 Alexander Roshal
Extracting from bigrandomfile.part01.rar
... random-300MB.bin OK
All OK
real 0m6.192s
user 0m1.940s
sys 0m1.932s
Code: Select all
sander@netbook:~/rar-kul$ time 7z x *01.rar
7-Zip 9.20 Copyright (c) 1999-2010 Igor Pavlov 2010-11-18
p7zip Version 9.20 (locale=C,Utf16=off,HugeFiles=on,2 CPUs)
Processing archive: bigrandomfile.part01.rar
Extracting random-300MB.bin
Everything is Ok
Size: 314572800
Compressed: 314575528
real 0m7.043s
user 0m2.628s
sys 0m1.852s
Re: linux rar 7zip support for multithreading
It's actually slightly faster in processing time -- compare the sys values. But the user time and remaining difference (wait time) is slower. This is likely because all threads were in contention for the handle for that single file.sander wrote:vsCode: Select all
real 0m6.192s user 0m1.940s sys 0m1.932s
So ... 7z is slower?Code: Select all
real 0m7.043s user 0m2.628s sys 0m1.852s
Test this on a multi-part rar and/or a rar with many files and multiple parts and you will likely see faster values for 7z, at least on the second test.