Howto: multicore par2 on Debian, Ubuntu & derivatives

Support for the Debian/Ubuntu package, created by JCFP.
Forum rules
Help us help you:
  • Are you using the latest stable version of SABnzbd? Downloads page.
  • Tell us what system you run SABnzbd on.
  • Adhere to the forum rules.
  • Do you experience problems during downloading?
    Check your connection in Status and Interface settings window.
    Use Test Server in Config > Servers.
    We will probably ask you to do a test using only basic settings.
  • Do you experience problems during repair or unpacking?
    Enable +Debug logging in the Status and Interface settings window and share the relevant parts of the log here using [ code ] sections.
blindpet
Newbie
Newbie
Posts: 28
Joined: July 13th, 2014, 7:51 am
Contact:

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by blindpet »

Thank you for the clarification jcfp, I wasn't sure if Sabnbzbd needed to start the threads or not. Will be posting the instructions for others to use and linking here. Thanks for your work on this.
blindpet
Newbie
Newbie
Posts: 28
Joined: July 13th, 2014, 7:51 am
Contact:

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by blindpet »

Just to clarify, are you cool with me posting this as a guide on my site to spread your work? Links to your github and this thread will be included in the post of course.
User avatar
jcfp
Release Testers
Release Testers
Posts: 985
Joined: February 7th, 2008, 12:45 pm

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by jcfp »

No problem.
blindpet
Newbie
Newbie
Posts: 28
Joined: July 13th, 2014, 7:51 am
Contact:

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by blindpet »

About to post this and doing some tests, do you know if the repair time in Sabnbzd includes the verification time? Using the same nzb file with intentionally damaged blocks for consistency I get these times which is an improvement but not as significant as I would have hoped

Single core par2cmdline 198 seconds
Dual core par2cmdline 166 seconds

Verification time is 105 seconds which would make it
Single core 93
Dual core 61
Skyrider
Newbie
Newbie
Posts: 6
Joined: December 11th, 2015, 6:03 am

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by Skyrider »

According to: http://www.htpcguides.com/install-par2- ... ntu-linux/

The guide was made with the help of people's replies in this thread. (jcfp and sander).

I'm using a Pi2 model B, and it appears the entire compile fails:

http://paste.debian.net/hidden/ddc4d151/

After installing automake1.10, still fails:

http://paste.debian.net/hidden/579c06d5/
blindpet
Newbie
Newbie
Posts: 28
Joined: July 13th, 2014, 7:51 am
Contact:

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by blindpet »

Skyrider, try installing autoconf too
Skyrider
Newbie
Newbie
Posts: 6
Joined: December 11th, 2015, 6:03 am

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by Skyrider »

Thanks, but:

autoconf is already the newest version.

Forgot to mention that libtbb-dev is NOT available for raspbarian. Despite it being it being mentioned in the tutorial.
User avatar
jcfp
Release Testers
Release Testers
Posts: 985
Joined: February 7th, 2008, 12:45 pm

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by jcfp »

Your automake problem is likely caused by the instructions at htpcguides.com telling you to run configure/make by hand rather than actually building the debian package, thereby skipping steps included in the latter such as autoreconf (and gcc hardening later on). Retry with the instruction in the topic start.

I'm not sure whether that will make that "compilation requires an ARMv7-a architecture" error go away though. What os and release are you using? How did you install tbb and its header files if not by installing libtbb-dev and libtbb2?
Skyrider
Newbie
Newbie
Posts: 6
Joined: December 11th, 2015, 6:03 am

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by Skyrider »

jcfp wrote:Your automake problem is likely caused by the instructions at htpcguides.com telling you to run configure/make by hand rather than actually building the debian package, thereby skipping steps included in the latter such as autoreconf (and gcc hardening later on). Retry with the instruction in the topic start.

I'm not sure whether that will make that "compilation requires an ARMv7-a architecture" error go away though. What os and release are you using? How did you install tbb and its header files if not by installing libtbb-dev and libtbb2?
Thanks for the reply. I'm using:
Distributor ID: Raspbian
Description: Raspbian GNU/Linux 8.0 (jessie)
Release: 8.0
Codename: jessie
As for libtbb-dev, I was unable to apt-get install the package file, So I installed the DEB files instead:

pool/main/t/tbb/libtbb-dev_4.2~20140122-5_armhf.deb
pool/main/t/tbb/libtbb2_4.2~20140122-5_armhf.deb

Located at: http://ftp.iinet.net.au/debian/debian/
User avatar
safihre
Administrator
Administrator
Posts: 5335
Joined: April 30th, 2015, 7:35 am
Contact:

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by safihre »

I think the Wiki should be updated, since it still points to the ChuChuSoft website that disappeared.
http://wiki.sabnzbd.org/configure-switc ... multi-core
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
User avatar
shypike
Administrator
Administrator
Posts: 19774
Joined: January 18th, 2008, 12:49 pm

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by shypike »

safihre wrote:I think the Wiki should be updated, since it still points to the ChuChuSoft website that disappeared.
http://wiki.sabnzbd.org/configure-switc ... multi-core
Fixed.
User avatar
safihre
Administrator
Administrator
Posts: 5335
Joined: April 30th, 2015, 7:35 am
Contact:

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by safihre »

Also on github in the README needs the new link!
Both for 0.8 and 0.7 probably!
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
User avatar
sander
Release Testers
Release Testers
Posts: 8808
Joined: January 22nd, 2008, 2:22 pm

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by sander »

The script gets this file:
https://github.com/jcfp/par2tbb-chuchus ... 503.tar.gz

Does that means it's par2cmdline 0.4, whereas par2cmdline is already at 0.6.14 (with the "-N" option)?
User avatar
an3k
Newbie
Newbie
Posts: 31
Joined: June 16th, 2015, 10:49 am

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by an3k »

How come SABnzbd NEVER uses the multithreaded version? I entered -t+ into the web config setting "Extra PAR2 Parameters" but when SABnzbd is verifying or repairing the executed command is just /usr/bin/par2 r /path/to/first_file.par2 /path/to/first_file.rar ...

I checked the code and found this nonsense:

newsunpack.py:1163 of 0.8.0 Beta 4

Code: Select all

 1    classic = classic or not cfg.par2_multicore()
 2    if sabnzbd.WIN32:
 3        # If filenames are UTF-8 then we must use par2-tbb, unless this is a retry with classic
 4        tbb = (sabnzbd.assembler.GetMD5Hashes(parfile, True)[1] and not classic) or not PAR2C_COMMAND
 5    else:
 6        tbb = False
 7    if tbb and cfg.par_option():
 8        command = [str(PAR2_COMMAND), cmd, str(cfg.par_option().strip()), parfile]
 9    else:
10        if classic:
11            command = [str(PAR2C_COMMAND), cmd, parfile]
12        else:
13            command = [str(PAR2_COMMAND), cmd, parfile]
14    logging.debug('Par2-classic = %s', classic)
So if we're not on Windows the if clause (line 7) which should set the extra par2 parameters (line 8) never gets executed but instead its else clause (line 9 to 13). In fact only one command is executed: line 13.

Replace line 13 with

Code: Select all

command = [str(PAR2_COMMAND), cmd, str(cfg.par_option().strip()), parfile]
and you get your extra par2 parameters passed to par2 and thus real multi-threading.

Enable debug logging and check your logfiles for Starting par2 to see what command line sabnzbd used.
Before my dirty hack it was
2016-01-10 02:52:04,437::DEBUG::[newsunpack:1195] Starting par2: ['/usr/bin/par2', 'r', '/var/tmp/some.home.videos.par2']
after my dirty hack it is
2016-01-10 02:52:04,437::DEBUG::[newsunpack:1195] Starting par2: ['/usr/bin/par2', 'r', '-t+', '/var/tmp/some.home.videos.par2']

Finally this makes sense "Processing verifications and repairs concurrently using up to 24 logical CPUs." 8)
User avatar
sander
Release Testers
Release Testers
Posts: 8808
Joined: January 22nd, 2008, 2:22 pm

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by sander »

I posted about that yesterday: http://forums.sabnzbd.org/viewtopic.php ... 98#p103720

So did you read that and then posted about it here?

Anyway: It has been fixed in the git version of SAB: https://github.com/sabnzbd/sabnzbd/comm ... 20f3d3e1aa
Post Reply