Howto: multicore par2 on Debian, Ubuntu & derivatives

Support for the Debian/Ubuntu package, created by JCFP.
Forum rules
Help us help you:
  • Are you using the latest stable version of SABnzbd? Downloads page.
  • Tell us what system you run SABnzbd on.
  • Adhere to the forum rules.
  • Do you experience problems during downloading?
    Check your connection in Status and Interface settings window.
    Use Test Server in Config > Servers.
    We will probably ask you to do a test using only basic settings.
  • Do you experience problems during repair or unpacking?
    Enable +Debug logging in the Status and Interface settings window and share the relevant parts of the log here using [ code ] sections.
blindpet
Newbie
Newbie
Posts: 28
Joined: July 13th, 2014, 7:51 am
Contact:

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by blindpet » November 2nd, 2015, 12:01 pm

Thank you for the clarification jcfp, I wasn't sure if Sabnbzbd needed to start the threads or not. Will be posting the instructions for others to use and linking here. Thanks for your work on this.

blindpet
Newbie
Newbie
Posts: 28
Joined: July 13th, 2014, 7:51 am
Contact:

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by blindpet » November 3rd, 2015, 6:46 pm

Just to clarify, are you cool with me posting this as a guide on my site to spread your work? Links to your github and this thread will be included in the post of course.

User avatar
jcfp
Release Testers
Release Testers
Posts: 907
Joined: February 7th, 2008, 12:45 pm

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by jcfp » November 4th, 2015, 8:22 am

No problem.

blindpet
Newbie
Newbie
Posts: 28
Joined: July 13th, 2014, 7:51 am
Contact:

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by blindpet » November 17th, 2015, 9:35 am

About to post this and doing some tests, do you know if the repair time in Sabnbzd includes the verification time? Using the same nzb file with intentionally damaged blocks for consistency I get these times which is an improvement but not as significant as I would have hoped

Single core par2cmdline 198 seconds
Dual core par2cmdline 166 seconds

Verification time is 105 seconds which would make it
Single core 93
Dual core 61

Skyrider
Newbie
Newbie
Posts: 5
Joined: December 11th, 2015, 6:03 am

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by Skyrider » December 11th, 2015, 6:15 am

According to: http://www.htpcguides.com/install-par2- ... ntu-linux/

The guide was made with the help of people's replies in this thread. (jcfp and sander).

I'm using a Pi2 model B, and it appears the entire compile fails:

http://paste.debian.net/hidden/ddc4d151/

After installing automake1.10, still fails:

http://paste.debian.net/hidden/579c06d5/

blindpet
Newbie
Newbie
Posts: 28
Joined: July 13th, 2014, 7:51 am
Contact:

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by blindpet » December 11th, 2015, 6:29 am

Skyrider, try installing autoconf too

Skyrider
Newbie
Newbie
Posts: 5
Joined: December 11th, 2015, 6:03 am

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by Skyrider » December 11th, 2015, 6:30 am

Thanks, but:

autoconf is already the newest version.

Forgot to mention that libtbb-dev is NOT available for raspbarian. Despite it being it being mentioned in the tutorial.

User avatar
jcfp
Release Testers
Release Testers
Posts: 907
Joined: February 7th, 2008, 12:45 pm

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by jcfp » December 11th, 2015, 7:09 am

Your automake problem is likely caused by the instructions at htpcguides.com telling you to run configure/make by hand rather than actually building the debian package, thereby skipping steps included in the latter such as autoreconf (and gcc hardening later on). Retry with the instruction in the topic start.

I'm not sure whether that will make that "compilation requires an ARMv7-a architecture" error go away though. What os and release are you using? How did you install tbb and its header files if not by installing libtbb-dev and libtbb2?

Skyrider
Newbie
Newbie
Posts: 5
Joined: December 11th, 2015, 6:03 am

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by Skyrider » December 11th, 2015, 7:58 am

jcfp wrote:Your automake problem is likely caused by the instructions at htpcguides.com telling you to run configure/make by hand rather than actually building the debian package, thereby skipping steps included in the latter such as autoreconf (and gcc hardening later on). Retry with the instruction in the topic start.

I'm not sure whether that will make that "compilation requires an ARMv7-a architecture" error go away though. What os and release are you using? How did you install tbb and its header files if not by installing libtbb-dev and libtbb2?
Thanks for the reply. I'm using:
Distributor ID: Raspbian
Description: Raspbian GNU/Linux 8.0 (jessie)
Release: 8.0
Codename: jessie
As for libtbb-dev, I was unable to apt-get install the package file, So I installed the DEB files instead:

pool/main/t/tbb/libtbb-dev_4.2~20140122-5_armhf.deb
pool/main/t/tbb/libtbb2_4.2~20140122-5_armhf.deb

Located at: http://ftp.iinet.net.au/debian/debian/

User avatar
safihre
Administrator
Administrator
Posts: 3373
Joined: April 30th, 2015, 7:35 am
Location: Switzerland
Contact:

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by safihre » December 29th, 2015, 3:20 pm

I think the Wiki should be updated, since it still points to the ChuChuSoft website that disappeared.
http://wiki.sabnzbd.org/configure-switc ... multi-core

User avatar
shypike
Administrator
Administrator
Posts: 21039
Joined: January 18th, 2008, 12:49 pm

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by shypike » December 30th, 2015, 5:29 am

safihre wrote:I think the Wiki should be updated, since it still points to the ChuChuSoft website that disappeared.
http://wiki.sabnzbd.org/configure-switc ... multi-core
Fixed.

User avatar
safihre
Administrator
Administrator
Posts: 3373
Joined: April 30th, 2015, 7:35 am
Location: Switzerland
Contact:

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by safihre » December 30th, 2015, 8:38 am

Also on github in the README needs the new link!
Both for 0.8 and 0.7 probably!

User avatar
sander
Release Testers
Release Testers
Posts: 6758
Joined: January 22nd, 2008, 2:22 pm

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by sander » January 9th, 2016, 3:07 pm

The script gets this file:
https://github.com/jcfp/par2tbb-chuchus ... 503.tar.gz

Does that means it's par2cmdline 0.4, whereas par2cmdline is already at 0.6.14 (with the "-N" option)?

an3k
Newbie
Newbie
Posts: 18
Joined: June 16th, 2015, 10:49 am

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by an3k » January 9th, 2016, 9:37 pm

How come SABnzbd NEVER uses the multithreaded version? I entered -t+ into the web config setting "Extra PAR2 Parameters" but when SABnzbd is verifying or repairing the executed command is just /usr/bin/par2 r /path/to/first_file.par2 /path/to/first_file.rar ...

I checked the code and found this nonsense:

newsunpack.py:1163 of 0.8.0 Beta 4

Code: Select all

 1    classic = classic or not cfg.par2_multicore()
 2    if sabnzbd.WIN32:
 3        # If filenames are UTF-8 then we must use par2-tbb, unless this is a retry with classic
 4        tbb = (sabnzbd.assembler.GetMD5Hashes(parfile, True)[1] and not classic) or not PAR2C_COMMAND
 5    else:
 6        tbb = False
 7    if tbb and cfg.par_option():
 8        command = [str(PAR2_COMMAND), cmd, str(cfg.par_option().strip()), parfile]
 9    else:
10        if classic:
11            command = [str(PAR2C_COMMAND), cmd, parfile]
12        else:
13            command = [str(PAR2_COMMAND), cmd, parfile]
14    logging.debug('Par2-classic = %s', classic)
So if we're not on Windows the if clause (line 7) which should set the extra par2 parameters (line 8) never gets executed but instead its else clause (line 9 to 13). In fact only one command is executed: line 13.

Replace line 13 with

Code: Select all

command = [str(PAR2_COMMAND), cmd, str(cfg.par_option().strip()), parfile]
and you get your extra par2 parameters passed to par2 and thus real multi-threading.

Enable debug logging and check your logfiles for Starting par2 to see what command line sabnzbd used.
Before my dirty hack it was
2016-01-10 02:52:04,437::DEBUG::[newsunpack:1195] Starting par2: ['/usr/bin/par2', 'r', '/var/tmp/some.home.videos.par2']
after my dirty hack it is
2016-01-10 02:52:04,437::DEBUG::[newsunpack:1195] Starting par2: ['/usr/bin/par2', 'r', '-t+', '/var/tmp/some.home.videos.par2']

Finally this makes sense "Processing verifications and repairs concurrently using up to 24 logical CPUs." 8)

User avatar
sander
Release Testers
Release Testers
Posts: 6758
Joined: January 22nd, 2008, 2:22 pm

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by sander » January 10th, 2016, 2:08 am

I posted about that yesterday: http://forums.sabnzbd.org/viewtopic.php ... 98#p103720

So did you read that and then posted about it here?

Anyway: It has been fixed in the git version of SAB: https://github.com/sabnzbd/sabnzbd/comm ... 20f3d3e1aa

Post Reply