Sabnzbd on Unraid, Download speed keeps dropping to zero every 5 seconds

[vinister]

So,

Sabnzbd 0.6.5 installed on Unraid server 5.0B7.

My ISP connection is up to 50Mbps.  If I do a speetest, I max out the 50Mbps, and get a nice big solid chunk of download in my router bandwidth logs.

In Sabnzbd I have 3 servers setup (Astraweb, Giganews, and Usenetserver).  I typically use Astraweb, but am using the other 2 temporarily to troubleshoot.

My download speeds in SAb can reach 5MB/s.  BUT only for short bursts.  In my router logs, I can see consistent short bursts of data, dropping to zero between each.  These bursts last about one second, with about 4-5 seconds in between them.  So it takes about an hour to download a short TV episode...

What could be happening here?  I used to run SAB on a windows server on this same network, and I had consistent speeds, always maxing out my connection.  2 things have happened since then:  1 is a move to UNRAID, 2 is an upgrade to my ISP plan from a 25mbit plan to a 50mbit plan, which means a new modem (same router)

[shypike]

Set an article cache in Config->General.
-1 for "unlimited" memory, which is about 1.5 times the largest file you'll be downloading.
SABnzbd is also rather CPU hungry, so it matters what your Unraid has.

[vinister]

Thanks.

I tried the article cache setting, and report back if it works.

Do you think its because I'm writing directly to the array?  I don't use a cache drive.  I was thinking of adding another drive, external to the array, and keeping the files on that drive until the post-processing step. 

The motherboard is the same that used to run on windows, and the speed difference is roughly 100x, so something else must be amiss.

[vinister]

Ok, I put a -1 in the box, and it seems to be quite a bit slower now!

SABnzbd says the speed is constant at 172KB/s.    However in my router logs, nothing is even coming through.  I am getting random bursts up to about 50KB/s, but 90% of the time its at zero.

hmmmm, I just noticed the server is totally unresponsive now.

Maybe I'll try putting some more ram in there and trying again.

[shypike]

How much RAM do you have?
BTW: writing lots of temporary files to a RAID array is not good idea.
Unraid is not a real RAID, so at least use a section that is not reconstructable.

[vinister]

its got 2GB, should be adequate.

I realize its not the best idea, but there are certainly lots of people doing the same judging from the discussions on unraid forums.  Mounting non-array drives is not a standard feature of unraid, a plugin is needed.  I'll get it done, but I'm still new to linux so its slow going.

I have searched unraid forums, this one, and the xbmc forums looking for people experiencing this issue, and can't find anything that quite matches.  The fact that the identical hardware worked previously, on the same network, leads me to believe its either the transition to unraid, or the upgrades to SAB since I've last used it.  I guess I can try to install sab on my local machine, and set the unraid box as the download location, to see if I am having problems with the array. 

However, I have read of many people writing directly to the array at high speeds without problems.

is there anything I can look for in the logs that could tell me why the download speed is being limited?

[vinister]

Heres' the throughput:




and here's what happens if I hit speedtest.net, from the same network:




its weird, because like I said, I have 3 different working servers entered into the config.  The result is the same with any of the 3 servers enabled, or all of them enabled.

I really think this is showing that SAB is not the root of the issue, since it is 'bursting' into high speeds.  Granted not fully saturating my connection, but I would be satisfied with 2MB/s if I could sustain it. 

[shypike]

Is it possible to run a test with release 0.5.6?
If this makes a difference then at least have some idea where to look.

[vinister]

I'm still running some tests to figure it out, its just difficult to get any computer time this weekend.

On a fresh reboot of the unraid server, as soon as I start SABnzbd, it gets a full 6MB/s. It stays solid for about 15 seconds, then starts to fall off, eventually settling into the bursts speed as above. CPU utilization by python is less than 5%, and ram used is less than 40%. I think it's absolutely a problem with writing to the array. I need to add some disks, and in the process I will add a cache drive which should alleviate the disk system.

Thanks for your hints so far, I will post my results and if I need any more assistance from you!

[vinister]

SOLVED.

I don't know what the exact problem was, I starting having all kinds of problems with the array going unstable. Seemed like it could be a memory leak or something. I couldn't figure it out, so I formatted everything and started over.

Again I installed SAB to 'disk1' as part of the array. This time, no problems, downloads are solid 6MB/s every time, nice and solid. I'm going to run out of space pretty quickly!

I'm going to stick a 640GB WD BLACK hdd in there for cache, and use that for SAB's incoming/complete folders. That should remove the direct writes to the array, correct?

Thanks again for a wonderful program. LOTS of use on my windows boxes, now lots of use on my unraid box too! Your efforts are very much appreciated!

[vinister]

well, maybe not completely solved then!

I came home today, and the server had downloaded a respectable 195GB of data in the last 12 hours, nice!

But, take a look at the current speed, and its at 170KB/s. hmmmm. Then it got a little stuck with 2 different files at 100%, but still in the queue, not un-raring. Checked it awhile later, and both downloads have failed, with the following error:

Error "[Errno 12] Cannot allocate memory" while running rar_unpack on.....

Is 2GB of RAM not enough for an unraid server running SAB? Or is there a deeper issue?

And now again I am seeing 'pauses' in the download stream. Speed has slowed to 2MB/s, and is slowing steadily, with a dip to zero every 10 seconds or so. WTF


Here's what I get when I run 'TOP'

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15206 nobody 20 0 1592m 1.2g 3508 S 91 63.1 594:16.56 python
6790 root 20 0 80956 3232 636 S 17 0.2 92:41.47 shfs

So it seems SAB is taking 63% of the ram, and 91% of the cpu. IF I pause SAB, the cpu usage drops to 5-6%, but the ram stays at 63%.

When I try to "retry" UN-RARing the downloads, the cpu and RAM usage stay the same, the download stays at 100% for awhile, then appears below in a 'failed' state, could not allocate memory.

I tried clearing the history, which didn't change anything.

[vinister]

After searching I see this is not a new issue.

I don't have a SWAP, so the memory is full, plain as english.

Now just have to find out if unraid is supposed to have a swap, or what to do from here. At least memory is cheap, I can always toss a 8GB kit in there and call it a day

[shypike]

When there is a memory leak, then buying extra RAM won't help.
I'd say 2G should be enough, unless you're running something else that eats lots of memory.
There is another thread about SABnzbd's sometimes excessive memory usage on some Unix systems.
Also, some work-arounds are described.
http://forums.sabnzbd.org/viewtopic.php?f=3&t=4713

[vinister]

Darn, doesn't seem to be very conclusive. I thought there were lots of people running sab on unraid. Don't we all have the same issue?

[shypike]

Surprisingly we also get mixed reports from Ubuntu users.
For most everything is OK, some do have these memory issues.