Large queue slows downloads; interface unusable

[feld]

Hi guys,

When I have a large queue the interface becomes pretty much unusable and I've also noticed that the download speed drops from 2+MB/s down to about 1MB/s. This is extremely strange. It often gets so bad that I can't even get it to shutdown or restart; it simply refuses to do anything and I have to kill -9 it. Also, it's not chewing up unusual amounts of memory and I've never hit swap on this machine.

Thoughts?


Version: 0.5.4 (final)
OS: FreeBSD 8.1 (Atom CPU, 4GB ram)
Install-type: ports
Skin : smpl
Firewall Software: pf
Are you using IPV6? yes
Is the issue reproducible? yes, just drop about 50 items in your queue. helps if it's going to be a lot. Recently the queue was 300GB.

[shypike]

Set the refresh rate to a higher value or use the Plush skin.
When your queue is empty, start up once with the --clean parameter
to remove all debris from the cache folder.

[feld]

Thanks for the reply. I also rebuilt python2.6 with HUGE_STACK_SIZE and PTH (gnu pth) support. My research shows they both should give a big increase in performance -- especially the stack size because FreeBSD's default stack size is pretty small. This seems to have lowered my CPU usage a lot for both Sabnzbd and Sickbeard. The web interface is much more responsive now. I'll try out your tweaks as well.

[shypike]

I learn everyday!
I wasn't aware that this was possible at all.
Do you have some more details, so that we can add the info to our Wiki?

[feld]

I learn everyday!
I wasn't aware that this was possible at all.
Do you have some more details, so that we can add the info to our Wiki?

They're just standard options when you build Python on FreeBSD right now. The stack size increase is because on FreeBSD the default stack size is really small (I don't know the details), and the GNU Portable Thread Library (pth) is because the python developers discovered something is wrong with the way threads work on FreeBSD and how Python works, and they think it's a bug in the OS. Telling it to use the well-known GNU Portable Thread Library is a workaround and helps to mitigate the problems. They're working towards a fix in the OS so this isn't necessary.