Big NZBs make for slow/spikey downloads

[Destroyer]

Why not? It works perfectly well there. Those little servers are great as well

[MonkeyFighter]

I have found the problem.

I have never looked at python source code before, so some of this may sound funny since I don't speak python lingo.


The spikes are caused by the nzo saves in the cache - it appears to grab a lock and all of the download threads pause while the nzo is saved.  On my system, with a recent download, the nzo file was 19MB which was taking roughly 6 seconds to write out.  And, unlike actual article data, the nzo file gets written in chunks of 4K so there are thousands of calls to write() instead of just one big call for all 19MB worth.

I can make the spikes go away by commenting out the line that says:

        if file_done:
            sabnzbd.save_data(nzo, nzo.nzo_id)

in the register_article() function in the nzbqueue.py source file.

I am sure there is a much better way to fix it, I think a periodic timer-based save might be the way to go - you could set it to 20 minutes and that way one spike every 20 minutes would have negligible impact on throughput, unlike one spike every 30 seconds or so as it was doing before.  Or maybe fixing the code to write it out in one big chunk instead of a couple of thousand 4K writes would be enough to speed it up all on its own.

And yes, this problem exists in the 0.5 beta too.

[shypike]

Well, my staring at this piece of (very old) source code was less effective than yours.
This is one of the darker parts (as in our understanding all about it) that we inherited
from the original author.

Actually this part has been modified, because this saving used to an option.
But we had so many corruption reports back then that we decided to
save the internal admin at the completion of each file.
Never really thought about the cost of this, due to the lock (which is required).

I'll look into this further to find an optimal solution.
(The old option is not coming back, it caused too much problems).

Thank you very much for this report!

[cr0ntab]

Hey Shypike,

Would you mind updating this thread if/when you find a solution for this and commit it to the SVN?

I too am suffering from this issue, and MokeyFighters fix worked. However, as he stated, it's probably not a very good permanent fix.

[shypike]

It's already in.
You can set the parameter save_interval to the minimum amount
of seconds between subsequents saves of the internal admin.
A good value would be 600, which is 10 minutes.

The parameter is only in the INI file, not in the web-ui.

[cr0ntab]

Got it! Thanks a bunch 

[diamaunt]

I've run into the same problem with 0.4.12, and what I attributed it to was the behavior of sabnzbd when working with large nzb files... some of the sites allow you to easily click a few checkboxes and download one nzb that ends up being a dozen DVDs worth of data, sabnzbd wanted to download all the pieces before it started working on them, so I ended up with the working directory having over fifty thousand files in it, even on a good unix box, that slows things down, when you cd into the directory and type 'ls' and it sits there for a few seconds, instead of popping back a dir listing, you know there's trouble. *grin*.  All other files access get very slow too.

having it launch the processes to recombine the pieces into the rars, at least, during the downloading would go a long way to alleviating the slowdowns with big nzbs, I've had to resort to breaking up the nzbs into smaller batches.

(twin dual core opterons with 4gig of memory and a 8tb zfs file system)

oh, and btw, props for a great product!

[shypike]

This is solved in 0.5.0 RC1 coming out soon.

The cause is that SABnzbd insists on saving the whole internal
job administration after completion of every file in a job.
Very safe and crash-resistant, but also very expensive
when the admin is large (and it is for a 20G+ job).

The large amount of files is normally not a problem.
Large jobs don't have that many files.
Frequent abortion of jobs can lead to leaks of cached articles.
This is a separate problem that will be addressed later.

[diamaunt]

I beg to differ on your statement about a large number of files not being a problem, disk access slows considerably when windows has more than a few thousand files in a directory, and unix has more than twenty to thirty thousand, any time you have to create a file, any time anything hits the directory, it takes much longer for the OS to return.  I just checked a 7.3 dvd image nzb, and it had 19,787 individual articles to download, to be reassembled into 83 rars and pars, and thence to one file. 

two of those and file access would slow down dramatically, and system cpu time would go way up.

The last time I'd tried to download a large nzb, after tracking down the large number of files, I let it keep going, and it did eventually finish the very slow by that time download, and process all the files, cleaning up the directory, there weren't any files 'leaked' after it was finished, and everything was back to normal. 

I first ran into this problem when managing a backbone news server, which developed a performance problem, and I tracked down to a single directory with close to a hundred thousand article files in it, (some recruiter had decided to spam one of the jobs newsgroups) there were so many files in the directory that a 'rm *' returned 'too many arguments' and wouldn't delete files, had to do a 'find . |xargs rm' to get rid of 'em..

If that's fixed in the .5 rc1, I'll have to give it a shot!

[shypike]

Yes, having a very large number of files is a problem.
But SABnzbd doesn't first write all 20.000 articles to disk.
As soon as all the articles for one file are complete,
they are assembled into one diskfile and immediately deleted.
In your example, the amount of articles will be 20.000/83.
Worst case you're talking about no more than 500 files.
If you have a large memory cache it's even less.

The cache folder can have leaks, though. requiring manual cleanup.
This is one of the first things we'll tackle after 0.5.0Final.

[diamaunt]

Ok, cool, perhaps that behavior was a freak occurance... I'm pretty sure it wasn't leaks, since I hadn't had any abnormal shutdowns, and it finally ended up deleting the files... this was a couple of revs back, I believe. (0.4.something).

[shypike]

If a poster decides to upload an ISO as a single file
of 20.000 articles, then you do have a problem!
Fortunately this is very rare.