Decoder failure: Out of memory

Report & discuss bugs found in SABnzbd
Forum rules
Help us help you:
  • Are you using the latest stable version of SABnzbd? Downloads page.
  • Tell us what system you run SABnzbd on.
  • Adhere to the forum rules.
  • Do you experience problems during downloading?
    Check your connection in Status and Interface settings window.
    Use Test Server in Config > Servers.
    We will probably ask you to do a test using only basic settings.
  • Do you experience problems during repair or unpacking?
    Enable +Debug logging in the Status and Interface settings window and share the relevant parts of the log here using [ code ] sections.
Puzzled
Full Member
Full Member
Posts: 160
Joined: September 2nd, 2017, 3:02 am

Re: Decoder failure: Out of memory

Post by Puzzled »

I notice it says segment bytes="-21730" for one of the articles, but I don't think that's used in the decoder. It might depend on which usenet server is being used.
loopdemack
Newbie
Newbie
Posts: 49
Joined: April 26th, 2008, 2:22 pm

Re: Decoder failure: Out of memory

Post by loopdemack »

sander wrote: June 20th, 2019, 10:01 am I downloaded that .NZB with SABnzbd ... and no problem at all: a nice *mp4 in the resulting directory, no problems in GUI nor LOG
Yes as I told its a special circumstances its not that easy.
Puzzled
Full Member
Full Member
Posts: 160
Joined: September 2nd, 2017, 3:02 am

Re: Decoder failure: Out of memory

Post by Puzzled »

safihre: Maybe you could add a sanity check for the amount of RAM it tries to malloc and give an error that doesn't pause the downloading if it's clearly wrong (negative or > 15M)? Don't know if that's the issue, but at least you'd find out.
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: Decoder failure: Out of memory

Post by sander »

Puzzled wrote: June 20th, 2019, 12:40 pm I notice it says segment bytes="-21730" for one of the articles, but I don't think that's used in the decoder. It might depend on which usenet server is being used.
Ah ... good catch. I can now reproduce with a NZB with all "segment bytes" parts set to negative value, for example 'segment bytes="-728914"'

Crafted NZB here: https://raw.githubusercontent.com/sande ... estnzb.nzb

Code: Select all

2019-06-21 07:22:37,230::WARNING::[decoder:139] Decoder failure: Out of memory
2019-06-21 07:22:37,232::DEBUG::[nzbstuff:293] Finishing import on reftestnzb 100MB auto 2151de41 2019-06-15 [01/19] - "mytestpost.part01.rar" yEnc (1/15) 10485760
2019-06-21 07:22:37,233::INFO::[decoder:141] Decoder-Queue: 0, Cache: 0, 0, 1003487232
2019-06-21 07:22:37,235::DEBUG::[__init__:924] [sabnzbd.nzbstuff.finish_import] Loading data for SABnzbd_nzf_XX_hp5 from /home/sander/Downloads/incomplete/test-negative-segment-bytes-reftestnzb/__ADMIN__/SABnzbd_nzf_XX_hp5
2019-06-21 07:22:37,235::INFO::[decoder:142] Traceback: 
Traceback (most recent call last):
  File "/usr/share/sabnzbdplus/sabnzbd/decoder.py", line 124, in run
    data = self.decode(article, lines, raw_data)
  File "/usr/share/sabnzbdplus/sabnzbd/decoder.py", line 222, in decode
    decoded_data, output_filename, crc, crc_expected, crc_correct = sabyenc.decode_usenet_chunks(raw_data, article.bytes)
MemoryError
2019-06-21 07:22:37,239::INFO::[downloader:279] Pausing
And Sab says "Decoder failure: Out of memory"

Furthermore, SAB GUI says it has downloaded "0 MB / -113 MB" (so: negative), so also SAB itself (not just sabeync) reads & uses those negative values.
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: Decoder failure: Out of memory

Post by sander »

User avatar
safihre
Administrator
Administrator
Posts: 5338
Joined: April 30th, 2015, 7:35 am
Contact:

Re: Decoder failure: Out of memory

Post by safihre »

Very strange.
We do check for negative numbers.. Why doesn't that work? :O

https://github.com/sabnzbd/sabyenc/blob ... enc.c#L564
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
Puzzled
Full Member
Full Member
Posts: 160
Joined: September 2nd, 2017, 3:02 am

Re: Decoder failure: Out of memory

Post by Puzzled »

Are they using a sabyenc version compiled before this commit? https://github.com/sabnzbd/sabyenc/comm ... 3bbbb0e04b
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: Decoder failure: Out of memory

Post by sander »

Safihre and I found out the cause in the C code of sabyenc. and I've sent a Pull Request https://github.com/sabnzbd/sabyenc/pull/15
Afther that PR, SAB will not give an OoM anymore, and the download will succeed.
If you run from source and know how to compile and install sabyenc (on Linux juist one command, or three command including the git commands), you can use it now. Otherwise wait for a new SAB release.

The real root cause is of course a wrong NZB: a negative article size in the NZB is impossible and illegal. So SABnzbd should give a warning for that, and/or refuse the NZB completely.
loopdemack
Newbie
Newbie
Posts: 49
Joined: April 26th, 2008, 2:22 pm

Re: Decoder failure: Out of memory

Post by loopdemack »

sander wrote: June 21st, 2019, 8:56 am Safihre and I found out the cause in the C code of sabyenc. and I've sent a Pull Request https://github.com/sabnzbd/sabyenc/pull/15
Afther that PR, SAB will not give an OoM anymore, and the download will succeed.
If you run from source and know how to compile and install sabyenc (on Linux juist one command, or three command including the git commands), you can use it now. Otherwise wait for a new SAB release.

The real root cause is of course a wrong NZB: a negative article size in the NZB is impossible and illegal. So SABnzbd should give a warning for that, and/or refuse the NZB completely.
Could you speculate which tool did they use to create those crazy nzb files. I blame myself for not backing up erroneous nzb files which made out of memory bug, but I stumble on almost 450 on nzbs.org which is gone now. I will try to ask poster which tool did he used to create such nzb, we should warn indexing sites.
Last edited by loopdemack on June 22nd, 2019, 2:47 am, edited 1 time in total.
User avatar
safihre
Administrator
Administrator
Posts: 5338
Joined: April 30th, 2015, 7:35 am
Contact:

Re: Decoder failure: Out of memory

Post by safihre »

My fault guys, I just made a plain old coding bug... Thanks for reporting and pushing that it was really a bug and not your system! You were right!
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
User avatar
safihre
Administrator
Administrator
Posts: 5338
Joined: April 30th, 2015, 7:35 am
Contact:

Re: Decoder failure: Out of memory

Post by safihre »

I released a new version of SABYenc, 3.3.6.
How do you use SABnzbd, on which platform? I can create a new release for you.
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
loopdemack
Newbie
Newbie
Posts: 49
Joined: April 26th, 2008, 2:22 pm

Re: Decoder failure: Out of memory

Post by loopdemack »

I'm using ubuntu.
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: Decoder failure: Out of memory

Post by sander »

loopdemack wrote: June 22nd, 2019, 5:46 am I'm using ubuntu.
The commands to get the fixed sabyenc on your Ubuntu:

Code: Select all

cd
git clone https://github.com/sabnzbd/sabyenc.git
cd sabyenc
git checkout sabyenc-python2 
sudo python setup.py install
Done
User avatar
safihre
Administrator
Administrator
Posts: 5338
Joined: April 30th, 2015, 7:35 am
Contact:

Re: Decoder failure: Out of memory

Post by safihre »

(if all the prerequisites are installed, info here how to get those: https://sabnzbd.org/wiki/installation/sabyenc.html )
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: Decoder failure: Out of memory

Post by sander »

It looks like @jcfp has already updated his ppa for python-sabyenc with the newest, fixed sabyenc. See https://launchpad.net/~jcfp/+archive/ubuntu/sab-addons ... version 3.3.6-0ubuntu1~jcfp1

Cool! Thank you, @jcfp

So ... Ubuntu users ... just run the sudo apt-get update / upgrade, and your SABnzbd on Ubuntu is protected against Out-of-Memory.
Post Reply