PAR2 Files Disappearing During Download if Post-Processing Is Enabled [SOLVED]

[shypike]

OK, have Python2.5.1 running now. Removed the old Python, that seemed to help.
Other modules are a piece of cake.

And now testing...

[bob511]

I got all excited, because your problems installing a reasonably up-to-date version of python reminded me that I had (in order to not break anything in CC proper) installed 2.5 to /usr/local/bin/python2.5, and redirected SABnzbd.py there--without thinking to change the reference of any of the sub-scripts in sabnzbd/.  Maybe one of those--say, newsunpack.py--was trying the python2.3 at /usr/bin/python, and dying silently?

Unfortunately not; changed each individual script to point at the proper python binary, restarted SABnzbd+, ran the download again, and the par-verify problem remains.  Goddamn it, I thought that might have been it.  But I bring up the parallel install, in case you might see some other way SABnzbd+ might get confused about the right python binary in a way that vanilla SABnzbd wouldn't.

[shypike]

When you start the base script (SABnzbd.py) with a particular Python release, everything is run under that release of Python.
This way you can have multiple Python releases in parallel.
So python2.5 SABnzbd.py will force the correct Python.
As long as you run the installations of the Python modules (like CherryPy, yEnc etc.) as:
  python2.5 setup.py install
you can be sure that they install into the right Python release.

(It looks like compilation of Python does not follow this rule, because I couldn't get Python2.5.1 compiled and installed without first removing Python2.3).
Sometime this week I can start testing on CC4.2 (got some more stuff to do for SAB).

BTW: I did have a startup problem, because SAB crashed on an unresolvable hostname (clark.sabnzbd.lan).
(so, another code fix required).

[shypike]

Bob,

Got SABnzbd running on CC4.2, done a series of downloads that needed repairs and extra par2 fetching.
No problems found at all so far.

[bob511]

Yeah, my luck.  Thanks for going to such great lengths on my account, though.

If I'm reading things right, it looks like you changed nothing at all in the PAR2 section from the 0.2.5 newsunpack.py to 0.3.1, yes?

[shypike]

We did a bunch of tests with dubious NZB files.
So far the only scenario that may explain things is when the first PAR2 file is so damaged that no useful information comes back from par2cmdline. But looking at the code, SABnzbd should even recover from this situation.

There may be problems with par2cmdline itself (like "main packet not found"), but those problems are persistent.

We're still keeping an eye open for these problems, and we would appreciate further problem reports accompanied by logs and nzb file.

BTW: I am working on a fix that will download all par2 files even when SABnzbd predicts that repairs won't succeed.
Gives you more chance on manual repair, using other sources.
Also, post-processing will be blocked if repair is not possible.

[bob511]

So far the only scenario that may explain things is when the first PAR2 file is so damaged that no useful information comes back from par2cmdline. But looking at the code, SABnzbd should even recover from this situation.

And that doesn't explain why a manual run of par2cmdline on the very same PAR2 spits out the expected, useful info and repairs successfully, right?

Tried the current trunk of 0.4.x, to the same non-repair/CRC error result.  Lord knows I don't want to give up your nifty UNIX Permissions and kB/s-speed-limit options in going back to a working 0.2.5, so I'm still pouring over .py files with my feeble Python comprehension trying to see any place at which the PAR2 handling is different from the 0.3.x/0.4.x -plus versions.

[bob511]

[bob511]

That is my catharsis post, before I explain what's been going wrong, you see.

[bob511]

Some clues, to build up the suspense:

It's quite suspicous, because normally one or two INFO lines are logged when par2cmdline does its work.
You would see either "all files correct", "repair is required" or something along those lines.
Now nothing at all is logged (except the final conclusion), which is very odd.

Output of par2cmdline looks OK to me, exactly what is to be expected.

- Where did you get the modules from (par2, yenc, cherry etc).
    Especially par2cmdlin.

Self-compiled

I had (in order to not break anything in CC proper) installed 2.5 to /usr/local/bin/python2.5, and redirected SABnzbd.py there

2008-02-17 09:12:12,436::INFO::_yenc module... found!
2008-02-17 09:12:12,436::INFO::celementtree module... found!
2008-02-17 09:12:12,436::INFO::par2 binary... NOT found!
2008-02-17 09:12:12,437::INFO::rar binary... found (/usr/bin/rar)
2008-02-17 09:12:12,437::INFO::unzip binary... found (/usr/bin/unzip)
2008-02-17 09:12:12,437::INFO::Perl interpreter... found (['/usr/bin/perl', '/mnt/SABnzbdplus/sabnzbd/utils/sendEmail.pl'])

And the most important clue:

I'm going to feel like such an ass when you go through this heroic effort and it turns out to be some dumb, piddling oversight on my part.

[bob511]

Ok, no, that second-to-last one is the most important clue.  How did I miss the "par2 binary... NOT found!" while trying to diagnose a problem with the par2 binary?  SABnzbd's logging even capitalizes the "not" so helpfully, but still I just breezed by.

In compiling most of the things I had to self-compile for my somewhat-dated distro, I let things install to Autotools' favorite directory /usr/local/bin, instead of redirecting it to /usr/bin (which Python, for instance, might have conflicted with the 2.3 binary, like you ran into).  No big deal, /usr/local/bin is in the PATH, like it practically always is, right?

Except the bit at the beginning of newsunpack.py, for algorithmic reasons I have really thought through yet, doesn't seem to come across /usr/local/bin as a possible binary hide-out (I stuck some extra logging statements in there, just to be sure).  There was no problem with par2cmdline, it just wasn't anywhere SABnzbd+ expected it to be.

At first glance, it doesn't seem like the 0.2.5 path-finding code looks all that different from the 0.3.x/0.4.x code, so I'm not sure why the old, vanilla version was able to come across the par2 binary just fine, but a symlink in /usr/bin and SABnzbd+'s logs say it found par2 and I just got my first "[PAR-INFO] xxxx: => Verified in 30.7s, all files correct" entry in my history.

I really appreciate all the time and thought you put into trying to help me track this down, and I apologize that it seems to have resulted from a quirk of my setup.

Renaming the (poorly titled) thread as SOLVED.

EDIT:  More specifically (after a bit more tinkering), SABnzbd+ was not getting the correct PATH environment variable when launching--so, again, clearly a misconfig on my end, that SABnzbd+ was not expecting.

[shypike]

I should have a look at how SABnzbd behaves in post-processing when it cannot find par2.
The logging during post-processing is not very clear.

Ah, well. Your problem is solved at least.
I was able to fix one problem due to my excercise with CC4.2 (problem with unresolvable hostname)
and got a hint that I should look at post-proc logging.

[mortimer]

I had the same problem with "par verify failed on". The logs did NOT show  the error "par2 binary... NOT found" so the par2.exe could be found OK.
I was using an UNC path to a network disk:
    \\wd1\x\downloads\incomplete.
After I changed this to a drive letter:
      x:\downloads\incomplete
the par verify worked fine again.
Obviously the par2.exe somehow fails to handle UNC paths correctly.

Perhaps a tip for others that experience this problem.