linux rar 7zip support for multithreading

Want something added? Ask for it here.
Post Reply
nonenix
Newbie
Newbie
Posts: 1
Joined: March 28th, 2016, 1:19 pm

linux rar 7zip support for multithreading

Post by nonenix »

hi i wondered about multithreaded rar archive file extraction on linux and after some googleing it seems that 7zip seems support multhithrading also also at least reading rars.

so I wondered if it would be possible to add an option to choose 7zip instead of unrar for linux so small server with multicore can benefit.

Especially since its the slowest bottleneck so far
(at least on my small server with raid / 16GB ram and atom quad core rar takes by far longest since it uses only one core and those atom boards don't have a big GHrz per CPU amount... multithreaded par runs fine so far so it would be nice and since the 16GB ram alsmot cache 100% of the download and the raid write speed isn't maxed out only the single core power is slowing the process)
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: linux rar 7zip support for multithreading

Post by sander »

Did you test the speed differences?

I did:

Code: Select all

$ time unrar x *01.rar

UNRAR 5.00 beta 8 freeware      Copyright (c) 1993-2013 Alexander Roshal


Extracting from bigrandomfile.part01.rar
<...>
All OK

real	0m13.926s
user	0m0.608s
sys	0m0.916s
versus

Code: Select all

$ time 7z x *01.rar

7-Zip 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
p7zip Version 9.20 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,1 CPU)

Processing archive: bigrandomfile.part01.rar

Extracting  random-300MB.bin

Everything is Ok

Size:       314572800
Compressed: 314575528

real	0m13.484s
user	0m1.200s
sys	0m0.708s
So no difference in real time.
User avatar
safihre
Administrator
Administrator
Posts: 5338
Joined: April 30th, 2015, 7:35 am
Contact:

Re: linux rar 7zip support for multithreading

Post by safihre »

Sander: in the log 7zip says '1 CPU'.

Maybe it does improve with bigger file and that switch to enable more CPU?
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: linux rar 7zip support for multithreading

Post by sander »

safihre wrote:Sander: in the log 7zip says '1 CPU'.

Maybe it does improve with bigger file and that switch to enable more CPU?
Ah, good point; I was running it on my VPS. I'll find a multi-core system, with enough diskspace.

On a dual-core Atom:

First create a 300MB random file, then create the rar files:

Code: Select all

dd if=/dev/urandom of=random-300MB.bin bs=1M count=300
rar a bigrandomfile -v10m -m0 random-300MB.bin
Then unrar the files:

Code: Select all

sander@netbook:~/rar-kul$ time unrar x *01.rar

UNRAR 5.00 beta 8 freeware      Copyright (c) 1993-2013 Alexander Roshal


Extracting from bigrandomfile.part01.rar


...         random-300MB.bin                                          OK 
All OK

real	0m6.192s
user	0m1.940s
sys	0m1.932s
and

Code: Select all

sander@netbook:~/rar-kul$ time 7z x *01.rar

7-Zip 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
p7zip Version 9.20 (locale=C,Utf16=off,HugeFiles=on,2 CPUs)

Processing archive: bigrandomfile.part01.rar

Extracting  random-300MB.bin

Everything is Ok

Size:       314572800
Compressed: 314575528

real	0m7.043s
user	0m2.628s
sys	0m1.852s
So ... 7z is slower? ???
fysa
Newbie
Newbie
Posts: 2
Joined: February 13th, 2009, 3:57 pm

Re: linux rar 7zip support for multithreading

Post by fysa »

sander wrote:

Code: Select all

real	0m6.192s
user	0m1.940s
sys	0m1.932s
vs

Code: Select all

real	0m7.043s
user	0m2.628s
sys	0m1.852s
So ... 7z is slower? ???
It's actually slightly faster in processing time -- compare the sys values. But the user time and remaining difference (wait time) is slower. This is likely because all threads were in contention for the handle for that single file.
Test this on a multi-part rar and/or a rar with many files and multiple parts and you will likely see faster values for 7z, at least on the second test.
Post Reply