Backup not used for badly formed yEnc articles

pobox · Post by **pobox** » August 28th, 2008, 5:50 pm

Is there anything I can do to minimize skipped article writes? They can be single articles in a file, or two in succession. In the log entries below, I have replaced non-essential data with ellipses. I am using 0.3.4 with no memory cache. I have plenty of disk space.

SABnzbd+ 0.3.4

cache_limit = 0

[...]::INFO::[downloader] Thread 1@[...]: fetching [...]
[...]::INFO::[downloader] Thread 1@[...]: [...] done
[...]::INFO::[decoder] Decoding
[...]::WARNING::[assembler] missing

There were no log entries for the articles that were not assembled, such as these:

[...]::INFO::[articlecache] Flushing <Article: article=[...] to disk
[...]::INFO::[sabnzbd] Saving data for SABnzbd_article_[...]

Post by **shypike** » August 29th, 2008, 1:39 am

What's the problem?
Does SABnzbd behave incorrectly leading to problems?
Or is it just that you think it logs too many warning messages?

BTW, we have a 0.4.2 release now and 0.4.3 will be released this weekend.

pobox · Post by **pobox** » August 29th, 2008, 6:48 am

The four lines in the log excerpt above are for the same article. The problem is that sometimes SABnzbd downloads articles and then loses them. Why does that happen?

Post by **shypike** » August 29th, 2008, 9:31 am

I'm not sure what happens here.
But one thing I do notice in your log excerpt: it says art_id=None.
Looks more some articles are not downloaded at all.
This suggests a corrupt NZB file.

Can you email the logging and the NZB file to bugs at sabnzbd.org?

pobox · Post by **pobox** » September 14th, 2008, 7:25 pm

I found out why this is happening. There are some malformed yEnc articles being transmitted around Usenet. Some malformed articles have =ybegin but no =yend in the body. SABnzbd downloads, attempts to decode, does not find a =yend and tosses the article. It does this whether or not the use of fill servers on yEnc CRC errors is configured. If SABnzbd find a =ybegin but does not find a =yend I would want it to try a fill server just like for a CRC error.

Post by **shypike** » September 15th, 2008, 2:48 am

I see your point.
Could you send me an NZB that demonstrates this problem? This would make it easier to test a solution.
(mail to bugs at sabnzbd dot org).

I'm not sure if trying a backup server is useful. Most problems are caused by the uploaders.
So an article with a missing =yend would be identical on all servers.

Filed as Ticket #127

pobox · Post by **pobox** » September 21st, 2008, 9:22 pm

You could look at:

Newzbin NB32 ID: dbzka

part014.rar (193/201)

Transit news server corruption: Probably at !xs4all!ramfeed1.eweka.nl!

Bad on newsfarm.atl.highwinds-media.com (us.usenet-news.net)
Bad on news.newsreader.com
...and my ISP server, news.qwest.net
Probably bad on Usenetserver and Newshosting, but only if !xs4all!ramfeed1.eweka.nl! is in the Path header.

Good on newsfarm.ams.highwinds-media.com (eu.usenet-news.net)
Good on both Giganews USA and Giganews Europe.

SABnzbd drops it and doesn't try a backup server. So do you want to bother to write new code because of some rare mangled articles, or let QuickPar repair the damage? The articles that SABnzbd drops on failed decode have no =yend, and garbage at bottom, but still somehow ends with a dot so these get transmitted around Usenet. Only one transit news server is doing this, most of the damaged articles have a =yend from a different article appended. I started a topic on the Eweka board about it but the corruption of articles continues.

switch · Post by **switch** » September 23rd, 2008, 2:41 pm

Currently for an article to be classed as yEnc encoded it requires a ypart and yend. We could change it so it :
a) Only requires a ypart. Will end up complaining about a CRC error and the data will not be used.
b) Check if it only has a ypart and no yend, if so create a new type of error; data will not be used.
c) Only require a ypart, but still try to use what data is there. This would allow to keep the data but would possibly remove some detection of actual currupt downloads, and could possibly lead to other issues.

Since I can't reproduce your error on any of my servers, we might need you to test any changes. What platform are you on, windows?

Post by **shypike** » September 23rd, 2008, 4:18 pm

We will treat a missing =yend as a protocol error and report it as a warning.
If the "Fail on yEnc crc Errors" is set, it will try another server.
The current code just ignores the article, which isn't very helpful.

I can send you a Windows binary if you want to test it yourself.

pobox · Post by **pobox** » September 23rd, 2008, 6:01 pm

Wouldn't any article that is missing a yend also fail CRC? If ypart is found but pcrc32 not found, wouldn't that article fail CRC? If so, what would be the need to check for a yend? Failed CRC would be the error to try a fill server. Currently, if the article has a ypart and yend but fails CRC, SABnzbd still uses the data, or if set to try a fill server and still fails CRC, uses the data from there. I requested that change back when early SABnzbd was tossing all articles that failed CRC (We want to keep them because a large percentage of articles contain parts of two adjacent PAR2 source blocks, and known posting bugs drop only one line of yEnc, or only one byte).

I think the best is:

If fail_on_crc = 0 :
If ypart found and CRC fails use data anyway.

If fail_on_crc = 1 :
If ypart found and CRC fails use fill server, if CRC fails from fill server use data anyway.

Keep it simple, no need to check for yend. Does that make sense?

Yes I have Windows, please send a binary, thanks.

Post by **shypike** » September 24th, 2008, 3:37 pm

A CRC is optional in yEnc.
You're supposed to check for =yend and the validity of the parameters that follow it.
I don't think it's a good idea to start accepting all sorts of malformed yEnc articles.

I'll send you the link tomorrow (don't have much time today).

pobox · Post by **pobox** » September 25th, 2008, 5:32 pm

Thanks. Here is the simplified article trace for :

::INFO::[downloader] Thread [email protected]:119: fetching
::INFO::[downloader] Thread [email protected]:119: done
::INFO::[decoder] Decoding
::WARNING::[decoder] Badly formed yEnc article in
::DEBUG::[trylist] Appending newsfarm.atl.highwinds-media.com:119 to .__try_list
::INFO::[decoder] found at least one untested server
::INFO::[downloader] Thread [email protected]:80: fetching
::INFO::[downloader] Thread [email protected]:80: done
::INFO::[decoder] Decoding
::INFO::[articlecache] Flushing to disk
::INFO::[sabnzbd] Saving data for
::INFO::[sabnzbd] Loading data for
::INFO::[sabnzbd] removed

I also tested with a different backup server sending the bad article, and tested again with 'Fail on yEnc crc Errors = off', and in both of those cases the article having no =yend was discarded, which I guess is what we want to happen.

Post by **shypike** » September 26th, 2008, 1:20 am

OK, that will be the solution then.
Thanks for testing.

Support Forum