Repair failed, not enough repair blocks nearly everytime

Get help with all aspects of SABnzbd
Forum rules
Help us help you:
  • Are you using the latest stable version of SABnzbd? Downloads page.
  • Tell us what system you run SABnzbd on.
  • Adhere to the forum rules.
  • Do you experience problems during downloading?
    Check your connection in Status and Interface settings window.
    Use Test Server in Config > Servers.
    We will probably ask you to do a test using only basic settings.
  • Do you experience problems during repair or unpacking?
    Enable +Debug logging in the Status and Interface settings window and share the relevant parts of the log here using [ code ] sections.
SigHunter
Newbie
Newbie
Posts: 5
Joined: January 8th, 2016, 4:45 am

Repair failed, not enough repair blocks nearly everytime

Post by SigHunter »

Hi guys

I know there are some similar threads to this one, but I created a new one since I'm not exactly sure the problem is the same.

Sabnzbd fails to repair nearly every nzb I give it. When I manually repair (via par2repair -N *.par2 *) it always repairs without any problems and I can extract the files.
Downloading from another box with newsbin works flawlessly
I just reset my sabnzbd.ini but still happens. This did not happen 2 months ago.
Sabnzbdplus is running on Ubuntu 15.10, Provider is Eweka, Version 0.8.0Beta3 [0b6395f]


Example:

Code: Select all

Name The.Big.Bang.Theory.S09E12.1080p.HDTV.X264-DIMENSION
Completed 08/01/2016 10:33
Status Failed
Size 1.1 GB
Category Default
Path /mnt/storage/home/sighunter/Downloads/incomplete/The.Big.Bang.Theory.S09E12.1080p.HDTV.X264-DIMENSION
Download Downloaded in 1 min 27 seconds at an average of 13.3 MB/s
Age: 4h
3 articles were malformed
3 articles were missing
Source https://nzbindex.com/download/129942615/IyDmZwVOrEDzlIcNNUOsgXRWkXbXMR-0118-t9dXuaJHipCn3uW3So55rl1bhOwVzo.par2-1.09-GB.nzb
Servers newsreader96.eweka.nl=1.1 GB
Repair [t9dXuaJHipCn3uW3So55rl1bhOwVzo] Repair failed, not enough repair blocks (234 short)
par2verify says

Code: Select all

Repair is required.
2 file(s) exist but are damaged.
6 file(s) are ok.
You have 2566 out of 2947 data blocks available.
You have 145 recovery blocks available.
Repair is not possible.
You need 236 more recovery blocks to be able to repair.
However I can without any problem (always!) repair it with par2repair -N *.par2 *

Code: Select all

Repair is required.
2 file(s) exist but are damaged.
6 file(s) are ok.
You have 2943 out of 2947 data blocks available.
You have 145 recovery blocks available.
Repair is possible.
You have an excess of 141 recovery blocks.
4 recovery blocks will be used to repair.

Computing Reed Solomon matrix.
Constructing: done.
Solving: done.

Wrote 314572800 bytes to disk

Verifying repaired files:

Target: "t9dXuaJHipCn3uW3So55rl1bhOwVzo.part3.rar" - found.
Target: "t9dXuaJHipCn3uW3So55rl1bhOwVzo.part7.rar" - found.

Repair complete.

I have not had to do this ever before, just happens since about 2 months.
Maybe it was introduced with the update to Ubuntu 15.10 (from 15.04), I don't know

Any thoughts?
User avatar
shypike
Administrator
Administrator
Posts: 19774
Joined: January 18th, 2008, 12:49 pm

Re: Repair failed, not enough repair blocks nearly everytime

Post by shypike »

SigHunter wrote: However I can without any problem (always!) repair it with par2repair -N *.par2 *
I wonder what kind of par2 command you have installed.
-N is not a known option for par2.
Can you post its sign-on?
User avatar
jcfp
Release Testers
Release Testers
Posts: 986
Joined: February 7th, 2008, 12:45 pm

Re: Repair failed, not enough repair blocks nearly everytime

Post by jcfp »

Same as discussed in https://forums.sabnzbd.org/viewtopic.ph ... 15#p101219

In the meantime, I've also seen a few downloads myself where this made a difference. May make sense to pass that option by default for the relevant versions of par2?
User avatar
sander
Release Testers
Release Testers
Posts: 8812
Joined: January 22nd, 2008, 2:22 pm

Re: Repair failed, not enough repair blocks nearly everytime

Post by sander »

It seems "-N" is there in par2repair version 0.6.14 (which is there on Ubuntu 15.10):

Code: Select all

# par2repair -V       
par2cmdline version 0.6.14
# par2repair --help | grep " \-N"
  -N       : No data skipping (find badly mispositioned data blocks)
#
Not in 0.6.11 (on Ubuntu 15.04):

Code: Select all

$ par2repair -V
par2cmdline version 0.6.11
$ par2repair --help | grep " \-N"
$

HTH
User avatar
sander
Release Testers
Release Testers
Posts: 8812
Joined: January 22nd, 2008, 2:22 pm

Re: Repair failed, not enough repair blocks nearly everytime

Post by sander »

@SigHunter:

Under Config -> Switches, SABnzbd contains a field "Extra PAR2 Parameters" ... maybe you can try to fill out the "-N" option there and see if that works & helps?
See http://wiki.sabnzbd.org/configure-switches#multi-core
User avatar
shypike
Administrator
Administrator
Posts: 19774
Joined: January 18th, 2008, 12:49 pm

Re: Repair failed, not enough repair blocks nearly everytime

Post by shypike »

OK, I can see how this helps.
Let's see what we can do.
User avatar
sander
Release Testers
Release Testers
Posts: 8812
Joined: January 22nd, 2008, 2:22 pm

Re: Repair failed, not enough repair blocks nearly everytime

Post by sander »

FWIW: if you want to update to the newest par2 / par2repair:

Code: Select all

sudo apt-get install git build-essential automake
git clone https://github.com/Parchive/par2cmdline.git
cd par2cmdline/
aclocal && autoconf && autoheader && automake --add-missing && ./configure
time make
./par2 r -N
./par2 --help
sudo make install
Result:

Code: Select all

sander@haring:~/git/par2cmdline$ ./par2 -V
par2cmdline version 0.6.14

sander@haring:~/git/par2cmdline$ ./par2 r -N 
You must specify a Recovery file.

sander@haring:~/git/par2cmdline$ ./par2 --help | grep " \-N"
  -N       : No data skipping (find badly mispositioned data blocks)
sander@haring:~/git/par2cmdline$ 
User avatar
sander
Release Testers
Release Testers
Posts: 8812
Joined: January 22nd, 2008, 2:22 pm

Re: Repair failed, not enough repair blocks nearly everytime

Post by sander »

More remarks:

Code: Select all

sander@haring:~$ cat .sabnzbd/logs/sabnzbd.log | grep -B5 "\-N"
Scanning: "Big.Buck.Bunny.2008.1080p.x264-60fps.part01.rar": 97.1%
Scanning: "Big.Buck.Bunny.2008.1080p.x264-60fps.part01.rar": 98.4%
Scanning: "Big.Buck.Bunny.2008.1080p.x264-60fps.part01.rar": 99.7%
File: "Big.Buck.Bunny.2008.1080p.x264-60fps.part01.rar" - no data found.
14966208 bytes of data were skipped whilst scanning.
If there are not enough blocks found to repair: try again with the -N option.
Note that last line:

Code: Select all

If there are not enough blocks found to repair: try again with the -N option.
Interesting, isn't it?

But:

Code: Select all

sander@haring:~$ cat .sabnzbd/sabnzbd.ini | grep par_opt
par_option = -N
So the option is there, but it seems not to be used by SABnzbd:

Code: Select all

sander@haring:~$ cat .sabnzbd/logs/sabnzbd.log | grep "Starting par2" | cut -c-200

2016-01-08 23:53:33,259::DEBUG::[newsunpack:1195] Starting par2: ['/usr/local/bin/par2', 'r', '/home/sander/Downloads/incomplete/buck bunny - bad/Big.Buck.Bunny.2008.1080p.x264-60fps.par2', '/home/san
2016-01-08 23:53:52,365::DEBUG::[newsunpack:1195] Starting par2: ['/usr/local/bin/par2', 'r', '/home/sander/Downloads/incomplete/buck bunny - bad/Big.Buck.Bunny.2008.1080p.x264-60fps.par2', '/home/san
Only 'r', but no '-N'? ???
User avatar
shypike
Administrator
Administrator
Posts: 19774
Joined: January 18th, 2008, 12:49 pm

Re: Repair failed, not enough repair blocks nearly everytime

Post by shypike »

Extra parameters are working on other systems than Windows.
I'm working on a solution.
User avatar
sander
Release Testers
Release Testers
Posts: 8812
Joined: January 22nd, 2008, 2:22 pm

Re: Repair failed, not enough repair blocks nearly everytime

Post by sander »

shypike wrote:Extra parameters are working on other systems than Windows.
So it should work on my Linux? Which it doesn't, AFAIK (see logging)
User avatar
shypike
Administrator
Administrator
Posts: 19774
Joined: January 18th, 2008, 12:49 pm

Re: Repair failed, not enough repair blocks nearly everytime

Post by shypike »

sander wrote:
shypike wrote:Extra parameters are working on other systems than Windows.
So it should work on my Linux? Which it doesn't, AFAIK (see logging)
Typo: extra parameters work ONLY on WIndows.
I'll fix that too.
User avatar
sander
Release Testers
Release Testers
Posts: 8812
Joined: January 22nd, 2008, 2:22 pm

Re: Repair failed, not enough repair blocks nearly everytime

Post by sander »

I created a NZB with missing segments (see https://gist.github.com/sanderjo/aa1f9d4720696cc11640 ), and let SAB 0.8.0 Beta3 download it. The download failed: "Repair failed, not enough repair blocks (47 short)"

With a plain "par2repair *par2", repair is not possible (as SAB itself discovered & reported):

Code: Select all

sander@haring:~/Downloads/incomplete/buck bunny - bad 2$ par2repair *par2
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.par2".
Loaded 50 new packets
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol93+92.par2".
Loaded 92 new packets including 92 recovery blocks
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol00+01.par2".
Loaded 1 new packets including 1 recovery blocks
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol03+03.par2".
Loaded 3 new packets including 3 recovery blocks
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol06+06.par2".
Loaded 6 new packets including 6 recovery blocks
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol01+02.par2".
Loaded 2 new packets including 2 recovery blocks
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol47+46.par2".
Loaded 46 new packets including 46 recovery blocks
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol12+12.par2".
Loaded 12 new packets including 12 recovery blocks
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol24+23.par2".
Loaded 23 new packets including 23 recovery blocks
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.par2".
No new packets found
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol00+01.par2".
No new packets found
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol01+02.par2".
No new packets found
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol03+03.par2".
No new packets found
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol06+06.par2".
No new packets found
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol12+12.par2".
No new packets found
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol24+23.par2".
No new packets found
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol47+46.par2".
No new packets found
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol93+92.par2".
No new packets found

There are 24 recoverable files and 0 other files.
The block size used was 196608 bytes.
There are a total of 1811 data blocks.
The total size of the data files is 355860040 bytes.

Verifying source files:

Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.nfo" - found.
File: "Big.Buck.Bunny.2008.1080p.x264-60fps.part01.rar" - no data found.
14966208 bytes of data were skipped whilst scanning.
If there are not enough blocks found to repair: try again with the -N option.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part02.rar" - damaged. Found 11 of 80 data blocks.
12789568 bytes of data were skipped whilst scanning.
If there are not enough blocks found to repair: try again with the -N option.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part03.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part04.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part05.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part06.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part07.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part08.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part09.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part10.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part11.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part12.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part13.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part14.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part15.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part16.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part17.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part18.rar" - damaged. Found 35 of 80 data blocks.
6539072 bytes of data were skipped whilst scanning.
If there are not enough blocks found to repair: try again with the -N option.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part19.rar" - damaged. Found 42 of 80 data blocks.
5547456 bytes of data were skipped whilst scanning.
If there are not enough blocks found to repair: try again with the -N option.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part20.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part21.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part22.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part23.rar" - found.

Scanning extra files:


Repair is required.
4 file(s) exist but are damaged.
20 file(s) are ok.
You have 1579 out of 1811 data blocks available.
You have 185 recovery blocks available.
Repair is not possible.
You need 47 more recovery blocks to be able to repair.
sander@haring:~/Downloads/incomplete/buck bunny - bad 2$

So:

Code: Select all

Repair is not possible.
You need 47 more recovery blocks to be able to repair.
as reported by SABnzbd.

With "-N" repair is possible:

Code: Select all

sander@haring:~/Downloads/incomplete/buck bunny - bad 2$ par2repair -N *par2
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.par2".
Loaded 50 new packets
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol93+92.par2".
Loaded 92 new packets including 92 recovery blocks
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol00+01.par2".
Loaded 1 new packets including 1 recovery blocks
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol03+03.par2".
Loaded 3 new packets including 3 recovery blocks
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol06+06.par2".
Loaded 6 new packets including 6 recovery blocks
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol01+02.par2".
Loaded 2 new packets including 2 recovery blocks
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol47+46.par2".
Loaded 46 new packets including 46 recovery blocks
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol12+12.par2".
Loaded 12 new packets including 12 recovery blocks
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol24+23.par2".
Loaded 23 new packets including 23 recovery blocks
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.par2".
No new packets found
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol00+01.par2".
No new packets found
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol01+02.par2".
No new packets found
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol03+03.par2".
No new packets found
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol06+06.par2".
No new packets found
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol12+12.par2".
No new packets found
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol24+23.par2".
No new packets found
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol47+46.par2".
No new packets found
Loading "Big.Buck.Bunny.2008.1080p.x264-60fps.vol93+92.par2".
No new packets found

There are 24 recoverable files and 0 other files.
The block size used was 196608 bytes.
There are a total of 1811 data blocks.
The total size of the data files is 355860040 bytes.

Verifying source files:

Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.nfo" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part01.rar" - damaged. Found 76 of 80 data blocks.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part02.rar" - damaged. Found 74 of 80 data blocks.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part03.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part04.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part05.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part06.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part07.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part08.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part09.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part10.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part11.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part12.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part13.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part14.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part15.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part16.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part17.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part18.rar" - damaged. Found 68 of 80 data blocks.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part19.rar" - damaged. Found 69 of 80 data blocks.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part20.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part21.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part22.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part23.rar" - found.

Scanning extra files:


Repair is required.
4 file(s) exist but are damaged.
20 file(s) are ok.
You have 1778 out of 1811 data blocks available.
You have 185 recovery blocks available.
Repair is possible.
You have an excess of 152 recovery blocks.
33 recovery blocks will be used to repair.

Computing Reed Solomon matrix.
Constructing: done.
Solving: done.

Wrote 62914560 bytes to disk

Verifying repaired files:

Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part01.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part02.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part18.rar" - found.
Target: "Big.Buck.Bunny.2008.1080p.x264-60fps.part19.rar" - found.

Repair complete.
@Shypike:
Would it be safe to always run "par2repair -N *par2", or can that have bad side effects, and is first a plain "par2repair *par2" needed?
User avatar
sander
Release Testers
Release Testers
Posts: 8812
Joined: January 22nd, 2008, 2:22 pm

Re: Repair failed, not enough repair blocks nearly everytime

Post by sander »

FWIW: In Beta4, in newsunpack.py I changed the if line from

Code: Select all

    if tbb and cfg.par_option():
        command = [str(PAR2_COMMAND), cmd, str(cfg.par_option().strip()), parfile]
to

Code: Select all

   if cfg.par_option():
        command = [str(PAR2_COMMAND), cmd, str(cfg.par_option().strip()), parfile]
(so: leaving out the tbb check) and ran SABnzbd on the missing-segemts-NZB. Result:

Code: Select all

sander@haring:~/git/sabnzbd$ cat ~/.sabnzbd/logs/sabnzbd.log | grep "Starting par2" | cut -c-200
2016-01-09 16:20:01,670::DEBUG::[newsunpack:1195] Starting par2: ['/usr/local/bin/par2', 'r', '-N', '/home/sander/Downloads/incomplete/buck bunny - bad because of missing segments/Big.Buck.Bunny.2008.
2016-01-09 16:20:13,178::DEBUG::[newsunpack:1195] Starting par2: ['/usr/local/bin/par2', 'r', '-N', '/home/sander/Downloads/incomplete/buck bunny - bad because of missing segments/Big.Buck.Bunny.2008.
so the "-N" is in the command line (good), and the download is correct. :) :) :)

Note: as said, I don't know if "-N" can have a bad side effect, and thus wether a plain par2repair should be run first.
User avatar
shypike
Administrator
Administrator
Posts: 19774
Joined: January 18th, 2008, 12:49 pm

Re: Repair failed, not enough repair blocks nearly everytime

Post by shypike »

sander wrote: Note: as said, I don't know if "-N" can have a bad side effect, and thus whether a plain par2repair should be run first.
I wouldn't worry about that.
If the blocks are not "badly mispositioned", it doesn't matter.
If they are, it's worth a try.
Doing a double run is a bit over the top.
User avatar
sander
Release Testers
Release Testers
Posts: 8812
Joined: January 22nd, 2008, 2:22 pm

Re: Repair failed, not enough repair blocks nearly everytime

Post by sander »

OK.

So SABnzbd should check if the par2 knows the option -N / No data skipping, and if so, always use it?

So a check like this:

Code: Select all

>>> "".join(os.popen('par2 --help').readlines()).find(' -N')
1261
>>> "".join(os.popen('par2 --help').readlines()).find('No data skipping')
1273
par2 does not know -N:

Code: Select all

>>> "".join(os.popen('par2 --help').readlines()).find(' -N')
-1
>>> "".join(os.popen('par2 --help').readlines()).find('No data skipping')
-1

Or just do a dry-run with -N:
-N is known

Code: Select all

$ par2 r -N
You must specify a Recovery file.
-N is unknown

Code: Select all

$ par2 r -N
Invalid option specified: -N


Or is there something more beautiful in python?
Post Reply