Download slowdown issue

[hjpool]

I'm running SABnzbd in a docker container on an UnRaid installation.
But i'm experiencing some issue's with the download speed.

When the container is freshly started or restarted the download is able to get to about 83 MB/s.
But after about 2,5GB the speed drops to about 30 MB/s, and it won't go any faster than this until you restart the container.

I've checked the debugging logs and i've found the following rule:
"2020-02-12 16:03:22,537::DEBUG::[downloader:548] Downloader-slowdown: 81638"
After that the speed is going down.

What could be the reason that this is happening and is this bug?

For comparison, when i'm running NZBGet on the same machine in a container, the speeds averages at about 83 MB/s.

Other "relevant" information:
SABnzbd version: 2.3.9
UnRaid version: 6.8.2
Docker version: 19.03.5
Machine CPU: Intel i5-3470
Machine Mem: 32 GiB
Storage: docker runs from SSD storage and SABnzbd (in)complete folders are on HDDs

[sander]

I have the same, and I thought it was caused by (fast) RAM (4GB) filling up with the download, and after that writing to the (slow) disk ... resulting in a slower download.

BUT ... your RAM is 32 GiB, so unlikely it's that ... unless your docker container RAM is limited to 4GB or so (#longshot)
Check with "docker stats" in the column "MEM USAGE / LIMIT"!

Oh wait, there is another thing: SABnzbd marks the first x% (10% or so) or so a downloaded immediately (the par2 files??), and I *think* it counts that as download thus adding to the download speed (#evenlongershot)

[hjpool]

docker stats gives the following response:

Code: Select all

CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
1cbcbcc465f5        binhex-sabnzbd      71.51%              778.8MiB / 31.12GiB   2.44%               2.55GB / 28.9MB     1.02MB / 209kB      60

So the full 32GiB is available.

I've traces the log entry back to some code in the downloader py(thon) file at line L533-L548
But my knowledge of python isn't good enough to know what is exactly going on.

Apparently i'm not allowed to post a link... (or a file with extension?)

[sander]

As a newbie you cannot post links. Workaround: put spaces in it.

[Puzzled]

CPU usage seems high. Is it like that all the time, and does it have access to all 4 cores? Do you know if it's 71% of 1 core (so that 400% would be max) or of all the cores combined (so that 100% is max)? Is download throttling enabled, and if so, what happens if you disable it?

[hjpool]

CPU usage is high because i was running a download through SABnzbd and the load is the load of one core, so yeah 400% is the max.
Also the container has access to all the available cores, so this should not be a problem.

What do you mean with "download throttling", i haven't got any resctictions setup in SABnzbd about the"Maximum line speed" or "Percentage of line speed"
Or is there another throttling setting?

[sander]

AFAIK:
a Python program (as SABnzbd) can use one core
a spawned process (like unrar, par2) can use more & other cores

[hjpool]

OK,

I've changed some code in the downloader.py file and now i'm able to download at the original 80MB/s speed.
On line 535 i've changed the time.sleep(0.01) to time.sleep(0.001) and there doesn't seem to go anything wrong.

I'm wondering what the idea behind this line of code is and what the consequences are if i'm using this value?

[sander]

ah ... interesting.

pinging @safihre as he knows those things.

[safihre]

Are you using a VPN?

In the Downloader we loop over all the ongoing connections and use a system level function to check if the connection is ready to output data. Sometimes all the connections are still receiving data and can't be processed yet, so the loop starts again really quickly. In those cases the looping can get out of control and uses huge amounts of CPU just to loop, without even doing something.
So that's why that code is there.
However, the system level function to check if a connection is ready to read doesn't work well on Linux systems that use a VPN and in this case maybe the Docker.
So the code above there will start downloading without this special slowdown, if will then test the effect of adding the slowdown, if it speed goes down it should disable the slowdown.
This has proven to be reliable, but maybe not in your case. The check if downloading can be slowed is only ran on the start of downloading, so requires a restart of Sabnzbd to be triggered again.

[hjpool]

safihre thanks for your response.

No, i'm not using any VPN and there doesn't seem to be any extra CPU load while downloading.
So I guess it works for me? :-)

[sander]

hjpool, now I'm curious: can you run SABnzbde outside the docker straight on the OS (on the same machine) to see if the docker does matter.
A docker does do NAT ... so it might have network effect like VPN has.

[Puzzled]

I don't understand why the delay is triggered if download throttling (the settings you found) is disabled. However, if it adds a 0.01 seconds delay for each iteration in the data polling loop then that will result in a maximum download speed of 1600 KB/s for each connection. In practice it will probably be lower. That's probably why setting it to 0.001 helps.

I don't understand this part of the code very well, but judging by the debug message it seems like the calculated download speed is 81 MB/s. Is there a limit of 80 MB/s anywhere that is inactive? If so you could try setting it to 100 MB/s. Adding a 0.001 delay for each poll can be a good idea for reducing cpu load, though, as discussed in this thread: viewtopic.php?f=11&t=24206

[hjpool]

hjpool, now I'm curious: can you run SABnzbde outside the docker straight on the OS (on the same machine) to see if the docker does matter.
A docker does do NAT ... so it might have network effect like VPN has.

Sander,
Sorry, i'm not quickly able to run SABnzbd outside the docker straight on de machine. (at this time)
But as a comparison, NZBGet is also running inside a container and doesn't have any problems with downloading at the +80MB/s speeds.

But i would like to use SABnzbd and for now my problem seems to have been solved, the downloads are nice and quick.

[hjpool]

I don't understand why the delay is triggered if download throttling (the settings you found) is disabled. However, if it adds a 0.01 seconds delay for each iteration in the data polling loop then that will result in a maximum download speed of 1600 KB/s for each connection. In practice it will probably be lower. That's probably why setting it to 0.001 helps.

I don't understand this part of the code very well, but judging by the debug message it seems like the calculated download speed is 81 MB/s. Is there a limit of 80 MB/s anywhere that is inactive? If so you could try setting it to 100 MB/s. Adding a 0.001 delay for each poll can be a good idea for reducing cpu load, though, as discussed in this thread: viewtopic.php?f=11&t=24206

There aren't any limits set, everything is default and empty, i've tried settings limits @ 1000MB/s but about 81MB/s is the fastest that my connection is able to go.
Through that thread i've found the delay