unpacking/repair is failing a lot lately

[ionicencode]

More info .... managed to snatch another release where SABnzbd seemingly ignored the par2 files. Running the par2 program on my server, par2cmdline version 0.6.12, failed. It reported:

Repair is required.
9 file(s) exist but are damaged.
35 file(s) are ok.
You have 815 out of 913 data blocks available.
You have 91 recovery blocks available.
Repair is not possible.
You need 7 more recovery blocks to be able to repair.

Par2 program on my client, QuickPar 0.9.1 http://www.quickpar.org.uk, says it can repair using 17/91 recovery blocks.

The two par2 programs agree on the damaged files but disagree wrt the number of complete data blocks. TeraCopy v2.27 http://www.codesector.com verifies the repaired files using sfv file. I'm left wondering if the file "repaired" by QuickPar 0.9.1 is valid or not but passing the SFV validation seems to indicate it's good. Also the archive file tested using IZArc v4.1.6 says the archive is valid. Maybe recent problems have something to do with par2cmdline version 0.6.12?

BTW QuickPar reports that the par2 files were created with par2cmdline version 0.4

EDIT: Downgrading to par2cmdline version 0.6.3 results in successful repair. Upgrading to 0.6.13 continues to result in failure. Will try changing postprocess script to try downgraded par2cmdline when the original fails.

EDIT2: My package manager upgraded par2cmdline June 13 of this year. Yup. That's about when I noticed things went downhill.

> qlop -l |grep par2
Wed Jan 30 16:05:58 2013 >>> app-arch/par2cmdline-0.4-r2
Sat Mar 8 20:50:07 2014 >>> app-arch/par2cmdline-0.6.3
Sat Jun 13 23:30:40 2015 >>> app-arch/par2cmdline-0.6.12

[shypike]

Disturbing news for Linux users.
It's not a problem you can solve in a user script, because that runs after SABnzbd is done.
Well you can, but only if you also do any unpacking in that script and allow failed jobs
to be handled by the script.

Can you email some NZB examples of the described problem cases?
[email protected]
Please add the URL of this message.

[jcfp]

Might be a known bug in par2 between 0.6.8 and 0.6.12 (inclusive), I'm thinking of this one that was introduced in response to an issue dealing with a possible DoS against par2. If this is the case, version 0.6.13 should do better with default settings, and also comes with a new command line options allowing users to opt for the old [pre 0.6.8] approach.

[shypike]

What can we do for anything between 0.6.8 and 0.6.12? Issue a warning?
Do we need the compatibility switch in 0.6.13?

[ionicencode]

Disturbing news for Linux users.

I'm running gentoo which adds significant variability because all packages are compiled locally using site specific flags.

> emerge --info
Portage 2.2.20 (python 3.4.1-final-0, default/linux/amd64/13.0, gcc-4.8.4, glibc-2.20-r2, 4.0.5-gentoo x86_64)
...
ACCEPT_KEYWORDS="amd64"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=core2 -O2 -pipe"
...

Can you email some NZB examples of the described problem cases?

Yes. I have a backup of the "incomplete" folder including __ADMIN__ from a failed repair by par2cmdline-0.6.12 but succeeds with par2cmdline-0.6.3. I will mail the NZB but also have the entire 2.2G failed fileset.

[shypike]

I will mail the NZB but also have the entire 2.2G failed fileset.

One would be nice to begin with.

[ionicencode]

Done. Check your mail. In case you are not able to re-produce from enclosed NZB, the fileset I have saved is from a sequence of failed RAR and PAR2 files for one media file from astraweb.
From: ionicencode
Subject: re: unpacking/repair is failing a lot lately
Date Sent: Sat, 25 Jul 2015 13:29:37 -0400

[ionicencode]

Might be a known bug in par2 between 0.6.8 and 0.6.12 (inclusive), I'm thinking of this one that was introduced in response to an issue dealing with a possible DoS against par2. If this is the case, version 0.6.13 should do better with default settings, and also comes with a new command line options allowing users to opt for the old [pre 0.6.8] approach.

Looks promising. I'll check the command line options for 0.6.12 and 0.6.13 and run manually against my failed, but repairable using 0.6.3, download.

[shypike]

Done. Check your mail. In case you are not able to re-produce from enclosed NZB, the fileset I have saved is from a sequence of failed RAR and PAR2 files for one media file from astraweb.
From: ionicencode
Subject: re: unpacking/repair is failing a lot lately
Date Sent: Sat, 25 Jul 2015 13:29:37 -0400

I'm sorry: nothing in the mailbox
[email protected]

[ionicencode]

Might be a known bug in par2 between 0.6.8 and 0.6.12 (inclusive), I'm thinking of this one that was introduced in response to an issue dealing with a possible DoS against par2. If this is the case, version 0.6.13 should do better with default settings, and also comes with a new command line options allowing users to opt for the old [pre 0.6.8] approach.

It appears your suspicions are correct. Thank you very much your help. This gives me more options to work around the situation. Partial log of my par2 testing using a broken download follows:

par2cmdline-0.6.3 v -v File.720p-dhd.nfo/File.720p-dhd.par2
Repair is possible.
17 recovery blocks will be used to repair.

par2cmdline-0.6.12 v -v File.720p-dhd.nfo/File.720p-dhd.par2
Repair is not possible.
You need 7 more recovery blocks to be able to repair.

par2cmdline-0.6.13 v -v File.720p-dhd.nfo/File.720p-dhd.par2
Repair is not possible.
You need 7 more recovery blocks to be able to repair.

par2cmdline-0.6.13 v -v -N File.720p-dhd.nfo/File.720p-dhd.par2
Repair is possible.
17 recovery blocks will be used to repair.

I'm sorry: nothing in the mailbox
[email protected]

Try 2 done. I also sent a link from a throwaway drop-box account.

From: ionicencode
Subject: re: unpacking/repair is failing a lot lately
Date Sent: Tue, 28 Jul 2015 12:48:54 -0400

[jcfp]

What can we do for anything between 0.6.8 and 0.6.12? Issue a warning?
Do we need the compatibility switch in 0.6.13?

As for 0.6.8 and 0.6.12: if only one knew how often the flaws in the scanning behaviour described in par2 issue #50 actually cause problems in real life...

The -N option is the only way (in >= 0.6.13 anyways) to scan the entire file for data blocks, needed for some repairs with significantly displaced data blocks as this thread shows. Trade-off is a potential dos via crafted files - I have no idea about the hacking skills required to actually pull that off though. For background info including the new options see the commit message at https://github.com/Parchive/par2cmdline ... 2e715ec79b

It appears your suspicions are correct. Thank you very much your help. This gives me more options to work around the situation. Partial log of my par2 testing using a broken download follows:

Looks like it, your output is the expected result if you have files with data blocks offset more than the range newer par2 releases scan by default. Note that you can set extra par2 params in sab, if you want to make the -N switch the default behaviour for your usenet downloads.