Howto: multicore par2 on Debian, Ubuntu & derivatives

Support for the Debian/Ubuntu package, created by JCFP.
Forum rules
Help us help you:
  • Are you using the latest stable version of SABnzbd? Downloads page.
  • Tell us what system you run SABnzbd on.
  • Adhere to the forum rules.
  • Do you experience problems during downloading?
    Check your connection in Status and Interface settings window.
    Use Test Server in Config > Servers.
    We will probably ask you to do a test using only basic settings.
  • Do you experience problems during repair or unpacking?
    Enable +Debug logging in the Status and Interface settings window and share the relevant parts of the log here using [ code ] sections.
User avatar
an3k
Newbie
Newbie
Posts: 31
Joined: June 16th, 2015, 10:49 am

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by an3k »

Oh, it finally got fixed? I talked about the issue with shypike and offered my help more than six month ago but see below:
Summarizing: the 64bit and Multicore options make only sense for Windows and OSX where we can distribute the binaries for par2 and unrar.
All other platforms are too much out of our control.
Maybe it would make more sense if the user could enter an alternative name (for non-Win non-OSX).
User avatar
safihre
Administrator
Administrator
Posts: 5335
Joined: April 30th, 2015, 7:35 am
Contact:

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by safihre »

@shypyke, is it really not worth to do a check after the module-detection if this -t and/or -N parameters are supported and then update par2-options config accordingly?

It seems this is part of the main par2 modules we support and definitely bumps performance without much effort..
We don't have the luxury like NZBGet to compile par2 ourselves, this seems a good start :)
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
User avatar
jcfp
Release Testers
Release Testers
Posts: 985
Joined: February 7th, 2008, 12:45 pm

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by jcfp »

sander wrote:The script gets this file:
https://github.com/jcfp/par2tbb-chuchus ... 503.tar.gz

Does that means it's par2cmdline 0.4, whereas par2cmdline is already at 0.6.14 (with the "-N" option)?
The chuchusoft releases are forked from the old (2004 or so) par2 0.4, primarily adding tbb support; so yes, that's 0.4. Any version of par2 greater than 0.4 is from (or based on) the github-fork-turned-official hosted at https://github.com/Parchive/par2cmdline/ Their 0.6.8 release changed the way the program tries to find misplaced blocks, 0.6.13 added the -S and -N options. The latter re-enables the old file scanning behaviour found in pre-0.6.8, including any and all versions that identify themselves as 0.4.

The ones you have to be careful with are 0.6.8-0.6.12 (inclusive), they suffer from a bug that assumes the offset of a misplaced block is always positive (see this bug report) and don't have any way to adjust or override the new bahaviour.
TioSolid
Newbie
Newbie
Posts: 2
Joined: March 15th, 2016, 7:52 pm

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by TioSolid »

Im having trouble building the multicore par2 in the ARM architeture too. The thing is that the problem seems to be related to the par2 package itself (verificationhashtable.h).

Here is my output (from the last command in the tutorial in the first post of this thread):

Code: Select all

In file included from par2cmdline.h:615:0,
                 from par1repairersourcefile.cpp:20:
verificationhashtable.h:429:49: warning: suggest parentheses around '&&' within '||' [-Wparentheses]
                        checksummer.ShortBlock() && checksummer.BlockLength() != nextentry->GetDataBlock()->GetLength()
                                                 ^
g++ -DHAVE_CONFIG_H -I.   -D_FORTIFY_SOURCE=2 -Wall -I./../tbb43_20141023oss/include   -std=c++11 -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -c -o par2creator.o par2creator.cpp
In file included from par2cmdline.h:615:0,
                 from par2creator.cpp:32:
verificationhashtable.h: In member function 'bool VerificationHashEntry::operator<(const VerificationHashEntry&) const':
verificationhashtable.h:69:40: warning: suggest parentheses around '&&' within '||' [-Wparentheses]
     return crc < r.crc || crc == r.crc && hash < r.hash;
                                        ^
verificationhashtable.h: In member function 'bool VerificationHashEntry::operator>(const VerificationHashEntry&) const':
verificationhashtable.h:73:40: warning: suggest parentheses around '&&' within '||' [-Wparentheses]
     return crc > r.crc || crc == r.crc && hash > r.hash;
                                        ^
verificationhashtable.h: In static member function 'static const VerificationHashEntry* VerificationHashEntry::Search(const VerificationHashEntry*, const MD5Hash&)':
verificationhashtable.h:186:47: warning: suggest parentheses around '&&' within '||' [-Wparentheses]
     if (entry->crc < crc || entry->crc == crc && entry->hash < hash)
                                               ^
verificationhashtable.h:190:52: warning: suggest parentheses around '&&' within '||' [-Wparentheses]
     else if (entry->crc > crc || entry->crc == crc && entry->hash > hash)
                                                    ^
verificationhashtable.h: In member function 'const VerificationHashEntry* VerificationHashTable::FindMatch(const VerificationHashEntry*, const Par2RepairerSourceFile*, const FileCheckSummer&, bool&) const':
verificationhashtable.h:405:54: warning: suggest parentheses around '&&' within '||' [-Wparentheses]
                             checksummer.ShortBlock() && checksummer.BlockLength() != currententry->GetDataBlock()->GetLength()
                                                      ^
verificationhashtable.h:412:56: warning: suggest parentheses around '&&' within '||' [-Wparentheses]
                               checksummer.ShortBlock() && checksummer.BlockLength() != currententry->GetDataBlock()->GetLength()
                                                        ^
In file included from par2cmdline.h:615:0,
                 from par2creator.cpp:32:
verificationhashtable.h:429:49: warning: suggest parentheses around '&&' within '||' [-Wparentheses]
                        checksummer.ShortBlock() && checksummer.BlockLength() != nextentry->GetDataBlock()->GetLength()
                                                 ^
/tmp/cco7GNtB.s: Assembler messages:
/tmp/cco7GNtB.s:4993: Error: thumb conditional instruction should be in IT block -- `cmpeq r1,r3'
/tmp/cco7GNtB.s:4994: Error: thumb conditional instruction should be in IT block -- `strexdeq lr,r2,r3,[fp]'
/tmp/cco7GNtB.s:5098: Error: thumb conditional instruction should be in IT block -- `cmpeq r3,r5'
/tmp/cco7GNtB.s:5099: Error: thumb conditional instruction should be in IT block -- `strexdeq r1,r4,r5,[fp]'
/tmp/cco7GNtB.s:5165: Error: thumb conditional instruction should be in IT block -- `strexeq r2,r10,[r3]'
/tmp/cco7GNtB.s:26751: Error: thumb conditional instruction should be in IT block -- `cmpeq r7,r3'
/tmp/cco7GNtB.s:26752: Error: thumb conditional instruction should be in IT block -- `strexdeq lr,r0,r1,[r5]'
make[2]: *** [par2creator.o] Error 1
make[2]: Leaving directory `/root/par2tbb/debpkg-par2tbb/par2cmdline-tbb-0.4+20150503'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/root/par2tbb/debpkg-par2tbb/par2cmdline-tbb-0.4+20150503'
dh_auto_build: make -j1 returned exit code 2
make: *** [build] Error 2
dpkg-buildpackage: error: debian/rules build gave error exit status 2
Is there any way to solve this?

Edit: Sorry, here is my specs:

- Ubuntu 14.04.4 LTS
- Odroid X2 (1.7GHz Quad core ARM Cortex-A9 MPCore)
User avatar
jcfp
Release Testers
Release Testers
Posts: 985
Joined: February 7th, 2008, 12:45 pm

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by jcfp »

Looks like a compiler and/or assembler issue. The internets suggest passing -Wa,-mimplicit-it=thumb to the build process, which you could do by editing the debian/rules file just after the uscan step in the tutorial. Insert something like this above the line that starts with a percent sign:

Code: Select all

CXXFLAGS:=$(shell dpkg-buildflags --get CXXFLAGS) -Wa,-mimplicit-it=thumb
export CXXFLAGS
And with that, I'm out of suggestions....
TioSolid
Newbie
Newbie
Posts: 2
Joined: March 15th, 2016, 7:52 pm

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by TioSolid »

Using this command gives me a "shell - command not found" error

I tried issuing
dpkg-buildpackage -S -us -uc -d -Wa -mimplicit-it=thumb

after the uscan command, but it gives me a "dpkg-buildflags: unknown option `-Wa'" error too :(
sandersonjobs
Newbie
Newbie
Posts: 2
Joined: July 23rd, 2016, 3:12 am

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by sandersonjobs »

Hi all. Did anyone ever figure this out? I have a Raspberry Pi 3 ARMv8 Quad core and Sabnzb is really sucking in single core mode. I have been using this app a long time, but from what I hear, nzbget has this functionality built in already. Anyone have a work-around, or should I just make the switch?
kcallis
Newbie
Newbie
Posts: 8
Joined: November 8th, 2010, 6:21 pm

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by kcallis »

jcfp wrote:
sander wrote:The script gets this file:
https://github.com/jcfp/par2tbb-chuchus ... 503.tar.gz

Does that means it's par2cmdline 0.4, whereas par2cmdline is already at 0.6.14 (with the "-N" option)?
The chuchusoft releases are forked from the old (2004 or so) par2 0.4, primarily adding tbb support; so yes, that's 0.4. Any version of par2 greater than 0.4 is from (or based on) the github-fork-turned-official hosted at https://github.com/Parchive/par2cmdline/ Their 0.6.8 release changed the way the program tries to find misplaced blocks, 0.6.13 added the -S and -N options. The latter re-enables the old file scanning behaviour found in pre-0.6.8, including any and all versions that identify themselves as 0.4.

The ones you have to be careful with are 0.6.8-0.6.12 (inclusive), they suffer from a bug that assumes the offset of a misplaced block is always positive (see this bug report) and don't have any way to adjust or override the new bahaviour.

So how does one get the update version on the Ubuntu box. I thought i could just use git and the new URL, but when I do the uscan I get the same issue:

Code: Select all

kcc@cortana:/tmp/tmp.SCY2wr3ZeE/debpkg-par2tbb$ uscan --force-downloaduscan warning: In watchfile debian/watch, reading webpage
  https://github.com/jcfp/par2tbb-chuchusoft-sources/releases failed: 500 Can't connect to 127.0.0.1:8888
sandersonjobs
Newbie
Newbie
Posts: 2
Joined: July 23rd, 2016, 3:12 am

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by sandersonjobs »

Sorry...I s witched to nzbget and have not had a single issue. I'm now just going to recommend the switch since nzbget makes it's own par2 that works with multiple architectures.
User avatar
safihre
Administrator
Administrator
Posts: 5335
Joined: April 30th, 2015, 7:35 am
Contact:

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by safihre »

EDITED: par2cmdline-mt doesn't work well with SABnzbd, because often it complains about files being 'out of basepath'.
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
piSAB
Newbie
Newbie
Posts: 5
Joined: February 22nd, 2015, 7:37 am

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by piSAB »

Hi Jcfp

Unfortunately this procedure no longer seems to work on a Raspberry PI 2 (Jessie build).
I posted what's listed below on http://www.htpcguides.com/install-par2- ... ntu-linux/
but was prompted to contact you here:


Unfortunately this procedure no longer seems to work on a Raspberry PI 2 (Jessie build)
Even after switching the installed dependencies to use libopencv-dev instead of libtbb-dev:

sudo apt-get install devscripts build-essential git debhelper dh-autoreconf libwww-perl libopencv-dev -y

The Install PAR2 Multicore on ARMv7 procedure fails with a fatal error at the make command:

sudo make -j2

par2cmdline.h:495:39: fatal error: tbb/task_scheduler_init.h: No such file or directory
#include "tbb/task_scheduler_init.h"
^
In file included from par2cmdline.cpp:26:0:
par2cmdline.h:495:39: fatal error: tbb/task_scheduler_init.h: No such file or directory
#include "tbb/task_scheduler_init.h"
^
compilation terminated.
compilation terminated.
Makefile:445: recipe for target 'buffer.o' failed
make[1]: *** [buffer.o] Error 1
make[1]: *** Waiting for unfinished jobs....
Makefile:445: recipe for target 'par2cmdline.o' failed
make[1]: *** [par2cmdline.o] Error 1
make[1]: Leaving directory '/home/pi/par2tbb/debpkg-par2tbb/par2cmdline-tbb-0.4+20150503'
Makefile:327: recipe for target 'all' failed
make: *** [all] Error 2

Any help here would be appreciated. Thanks in advance.

Mike Mod
It could be a temporarily buggy script Simon, you can try asking the developer of the script on the sabnzbd forum
User avatar
jcfp
Release Testers
Release Testers
Posts: 985
Joined: February 7th, 2008, 12:45 pm

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by jcfp »

piSAB wrote:libopencv-dev
An image processing library? Why?? You must install libtbb and its development headers for this version of par2. As it says right next to the instructions: supported architectures and operating systems are probably limited to whatever has a recent version of libtbb available. Everything else leads to this:
piSAB wrote:fatal error: tbb/task_scheduler_init.h: No such file or directory
Please stick to the build instructions in the topic start: no installing libopencv as if it were a replacement for tbb, no manual configure runs, no 'sudo make' with parellel build options set. It's hard enough already to guess what's going on all these different platforms and hardware architectures.
User avatar
safihre
Administrator
Administrator
Posts: 5335
Joined: April 30th, 2015, 7:35 am
Contact:

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by safihre »

I also tried many ways to get tbb on my RP3, but never succeeded.
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
User avatar
jcfp
Release Testers
Release Testers
Posts: 985
Joined: February 7th, 2008, 12:45 pm

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by jcfp »

Maybe the hardware isn't compatible, maybe the os provides only a subset of debian packages (remember users having to build their own unrar?). I dunno.
piSAB
Newbie
Newbie
Posts: 5
Joined: February 22nd, 2015, 7:37 am

Re: Howto: multicore par2 on Debian, Ubuntu & derivatives

Post by piSAB »

Thanks for your replies. I was simply following the guide on: http://www.htpcguides.com/install-par2- ... ntu-linux/

I made the replacement for libtbb-dev as suggested in the above mentioned guide as others had reported success in doing so.

libtbb-dev is not not available on the PI2 so as you suggest this is now most probably a no go for Raspberry PI.
Post Reply