Unknown error while running par2_repair / rar_unpack

[pmneo]

Hello!

I've a big Problem with the SABnzbd+ on a Ubuntu machine ...

The par2 repair and unrar process does not work, it says everytimes


Stage Par2
    [PAR-INFO] xxxxx: => Unknown error while running par2_repair, see logfile

Stage Unrar
    [RAR-INFO] xxxxx: => Unknown error while running rar_unpack, see logfile

But the Log file doesn't says more:

Code: Select all

2008-08-31 10:28:51,568::INFO::All processes started
2008-08-31 10:28:51,568::INFO::_yenc module... found!
2008-08-31 10:28:51,571::INFO::celementtree module... found!
2008-08-31 10:28:51,572::INFO::par2 binary... found (/usr/bin/par2)
2008-08-31 10:28:51,572::INFO::rar binary... found (/usr/bin/unrar)
2008-08-31 10:28:51,572::INFO::unzip binary... found (/usr/bin/unzip)
2008-08-31 10:28:51,572::INFO::pyOpenSSL... found (True)
2008-08-31 10:28:51,575::DEBUG::log_dir: /media/sda/_sabnzb/logs
2008-08-31 10:28:51,575::INFO::Web dir is /usr/share/sabnzbdplus/interfaces/smpl
2008-08-31 10:28:51,575::INFO::Web dir is /usr/share/sabnzbdplus/interfaces/Plush
...
...
2008-08-31 10:39:25,304::ERROR::[newsunpack] Unknown error while running par2_repair on set xxxxxx
2008-08-31 10:39:25,305::INFO::[postproc] Par2 check finished on xxxxxx.nzb
2008-08-31 10:39:25,393::INFO::[sabnzbd.misc] Creating directories: /media/sda/_complete/xxxxxx
2008-08-31 10:39:25,394::INFO::[postproc] Running unpack_magic on xxxxxx.nzb
2008-08-31 10:39:25,399::DEBUG::[newsunpack] build_filelists(): joinables: []
2008-08-31 10:39:25,400::DEBUG::[newsunpack] build_filelists(): zips: []
2008-08-31 10:39:25,400::DEBUG::[newsunpack] build_filelists(): rars: ['/media/sda/_tmp/xxxxxx/xxxxxx.part01.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part02.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part03.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part04.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part05.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part06.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part07.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part08.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part09.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part10.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part11.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part12.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part13.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part14.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part15.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part16.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part17.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part18.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part19.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part20.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part21.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part22.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part23.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part24.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part25.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part26.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part27.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part28.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part29.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part30.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part31.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part32.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part33.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part34.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part35.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part36.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part37.rar']
2008-08-31 10:39:25,402::INFO::[newsunpack] Unrar starting on /media/sda/_tmp/xxxxxx
2008-08-31 10:39:25,404::DEBUG::Rar_sets: {'xxxxxx': ['/media/sda/_tmp/xxxxxx/xxxxxx.part01.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part02.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part03.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part04.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part05.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part06.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part07.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part08.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part09.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part10.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part11.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part12.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part13.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part14.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part15.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part16.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part17.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part18.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part19.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part20.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part21.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part22.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part23.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part24.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part25.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part26.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part27.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part28.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part29.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part30.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part31.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part32.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part33.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part34.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part35.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part36.rar', '/media/sda/_tmp/xxxxxx/xxxxxx.part37.rar']}
2008-08-31 10:39:25,406::INFO::[newsunpack] Extracting rarfile /media/sda/_tmp/xxxxxx/xxxxxx.part01.rar (belonging to xxxxxx) to /media/sda/_complete/_UNPACK_xxxxxx
2008-08-31 10:39:25,406::DEBUG::[newsunpack] RAR_Extract(): Extractionpath: /media/sda/_complete/_UNPACK_xxxxxx
2008-08-31 10:39:25,407::ERROR::[newsunpack] Unknown error while running rar_unpack on xxxxxx.nzb

I've used 0.4.2, 0.4.3-rc4 and 0.4.3 final ....

So can anybody helb me? 

[shypike]

SABnzbd uses par2 and unrar, but we have no influence over what is installed.
For the Windows version, binaries of par2 and unrar are included, but we cannot
do this for Linux.
There have been complaints before about incompatible versions of par2 and unrar.

Here's what I am using on Ubuntu 8.04, both coming from the standard apt-get:

par2

Code: Select all

par2cmdline version 0.4, Copyright (C) 2003 Peter Brian Clements.
Modifications for concurrent processing, Unicode support, and hierarchial
directory support are Copyright (c) 2007-2008 Vincent Tan.
Concurrent processing utilises Intel Thread Building Blocks 2.0, Copyright (c) 2007-2008 Intel Corp.
Executing using the 32-bit x86 (IA32) instruction set.

unrar

Code: Select all

UNRAR 3.71 beta 1 freeware      Copyright (c) 1993-2007 Alexander Roshal

For a test you could do a download without post-processing and test if par2 and unrar work from a command shell.

[pmneo]

Hi!

my par2 is the self version, unrar is version 3.71 beta 3!

Just tested manualy par2 and unrar and it works fine ...

Is there any way to get a more detailed Error Information??

Thx

[shypike]

The way errors are handled in the par2 part of SABnzbd is a bit awkward.
Several errors land in just one error report, the one you get.

The only thing I can suggest is to do this:
1. clear the log files (~/.sabnzbd.logs)
2. Download a small job
3. Wait for the problem to occur
4. Zip the following files:
   - sabnzbd.ini (remove passwords!)
   - all log files
   - The NZB you used.
5. mail to bugs at sabnzbd.org, including the URL of this Post.

BTW: where did you get the distribution file from and how did you install it?
Which Python release do you use, which modules did you install?

[pmneo]

Hi Shypike!

Thx 4 your help, i have downloaded a small job and then i wondered because it doesn't failed.

Then i downloaed a big job and it fails!

After this big failed job, the job that has worked before failed too ...

I downloaed the sources from sabnzbd.org / the linked sourceforge page!

I have installed python 2.5.1 (r251:54863) with following modules:

Code: Select all

python                          - An interactive high-level object-oriented
python-central                  - register and build utility for Python pack
python-chardet                  - universal character encoding detector
python-cheetah                  - text-based template engine and Python code
python-cherrypy                 - Python web development framework
python-feedparser               - Universal Feed Parser for Python
python-libxml2                  - Python bindings for the GNOME XML library
python-minimal                  - A minimal subset of the Python language (d
python-pyopenssl                - Python wrapper around the OpenSSL library
python-support                  - automated rebuilding support for python mo
python-yenc                     - yEnc encoding/decoding extension for Pytho
python2.5                       - An interactive high-level object-oriented
python2.5-minimal               - A minimal subset of the Python language (v

Thx

[shypike]

Did the small job have any PAR2 files? If not, it makes sense that it succeeds.
But I must repeat: without the info I asked for, there's nothing we can do.

I don't know all the Python packages you listed, but it looks
unlikely that any interfere with SABnzbd.

[shypike]

Maybe you should try the un-official debian package created by JCFP,
discussed here:

http://forums.sabnzbd.org/http://forums ... 1709#p1709

[pmneo]

Yes the Small Job has par2 files!

I've allready tryed the un-official package ... with the same result ...

I will prepare the log files and send it to you!

Thx

[pmneo]

I think i have located the problem!!

The par2_repair / rar_unpack procedure failes everytime if the logfile is splittet after the max size of 5 MB!

All jobs works fine until the log file is splittet!

But if the log file is splittet the jobs that works before doesn't works anymore!

I have send you the log files, the ini and the nzb i used!

Thx

[shypike]

I doubt the logging has anything to do with it.
The "unknown error" appears about 20 sec before the end the log file.
You never know of course, but it is a standard feature of Python we use...

What I do see is that the job contains "Töchter", so with an accented character.
SABnzbd should support this, but we have had problems in the past with this.

Can you check the failed jobs for accented characters?
See if it makes a difference if you avoid these character.

If what you claim is true, it would make a difference if you increase the logging
size to, say, 20M and reduce the logging level to 0 (only errors/warnings).

Tonight I will look further into this.

[pmneo]

Hey shypike!

Yes it appears about 20 sec before the end of the latest log file, but the first created log file is sabnzbd.log.1!
In this log file the download works fine!

I don't think that the characters have anything to do with it, because i've downloaded the nzb twice!

The firsttime (when the log file was smaller than 5MB it works perfectly)

Then i imported the same nzb file again and it fails!

I will test it again with a larger log file size!

Thx pmneo

[shypike]

There's not much to see in your logfiles.
The error message is just too unspecific.

There is a way generate more diagnostics.
Assuming you run 0.4.3Final, you can make a change to the file newsunpack.py.
This change will remove exception handling for a section so that you see
where the problem actually occurs.
The side-effect is that the post processor will crash and stop doing further processing.
It may be worthwhile to investigate this, if you still have the patience.

The patch:
File sabnzbd/newsunpack.py
On line 605: replace try: with if 1:
On line 685: replace except: with if 0:

Next run the program from a shell (so not in -d mode) and catch the console output
with > or the tee program.

I can email you a modified copy if you feel uncomfortable editing it yourself.

For the next release (0.4.4) I'll try to add some better error reporting.

[pmneo]

Okay, i've found the Problem ...

Code: Select all

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.5/threading.py", line 460, in __bootstrap
    self.run()
  File "/home/pmneo/SABnzbd/sabnzbd/postproc.py", line 109, in run
    need_reAdd, res = par2_repair(parfile_nzf, nzo, workdir, _set)
  File "/home/pmneo/SABnzbd/sabnzbd/newsunpack.py", line 614, in par2_repair
    actionname, joinables)
  File "/home/pmneo/SABnzbd/sabnzbd/newsunpack.py", line 713, in PAR_Verify
    startupinfo=stup, creationflags=creationflags)
  File "/usr/lib/python2.5/subprocess.py", line 593, in __init__
    errread, errwrite)
  File "/usr/lib/python2.5/subprocess.py", line 1061, in _execute_child
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory

Killed

Seems that the memory goes out ...

But why does it work the first time ...

Could it be that the memory is not freed correctly??

Thx pmneo

[shypike]

Thanks for taking the trouble to test.

It looks like the operating system reports a memory problem.
In Python, the interpreter takes care of freeing memory, so unless the program
keeps creating new objects (due to a bug), there should not be a problem.

Is it possible to find out what the memory usage of programs on Linux is?
Maybe you have little memory or some other program consumes too much,
or SABnzbd does.
I'm a bit surprised to see that the Python library uses a fork to launch another
program. Forks are quite expensive for programs that use a lot of memory (like SABnzbd).

How much memory do you have? (My box has 768M and works rock-solid)
I know that people running SABnzbd on small NAS systems run into trouble without
some extra measures.
Do you use a memory cache for SABnzbd and if so, how big is it?

[pmneo]

Whell i use SABnzbd on a 128 MB RAM machine, i know thats not very much ...

But i have a second machine with 128 MB RAM an windows xp running (without swap) and on this machine SABnzbd works fine ...

I'll try to take a look on the memory usage, with top or ps aux ... but i had look this morning and the memory usage was about 30% ....

Thx pmneo