Freezing while loading NZB

[selstam]

Hi!

Whenever I load an NZB (no matter what the size) sabnzbd freezes and the CPU usage goes up to 100%. I can't access the web interface and the downloading stops.

It does not matter wether I drop it in the watchfolder structure or use an URL.

AFAIK this issue was not present in 0.5.0B3 and 0.5.0B4 but manifested itself in 0.5.0RC6 (and also in 0.5.0 FINAL).

Anyone got any ideas?

Version: 0.5.0 FINAL
OS: Debian Lenny (2.6.26-2-amd64-kernel)
Install-type: Python source
Skin (if applicable): Default
Firewall Software: None, HW firewall
Are you using IPV6? No
Is the issue reproducible? Yes

[shypike]

How long does the "freeze" take?
It will be unresponsive for some short time, because parsing the NZB
is a relatively costly thing which goes at the expense of other tasks.

[selstam]

We're talking about 60 seconds for a 63kB large NZB-file. During this time cpu usage goes through the roof ...

Code: Select all

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 402 selstam   20   0 1247m 915m 3268 S 99.5 45.4 119:05.95 SABnzbd.py

This is a 2.4GHz machine with 2GB's of memory ... :-)

[selstam]

Might have an update on this ... things seems to have returned to normal when my download backlog eased up a bit.

I now only have four items in my queue for a total of 95GB ... a week ago that was about 5-6 times as much. For the moment response-times seems much, much better and loading an NZB-file for 42GB takes 4-5 seconds (which sounds pretty normal to me).

[shypike]

We are looking at improving queue behavior for a later release.
Still, I cannot explain why it takes longer to add new NZB files to a large queue.
I cannot reproduce that problem.

[wodix]

Hi,

i have the same problem in two different enviroments (Win2003 Server & Qnap). If is the Queue larger (for example 100 GB) then every new action with the queue will be very slowly.

If i look into the cache folder, i will see that the still exists SABnzbd_nzo_* files will recreate ever and ever again. I think this will produce probably the performance problem.

In the next days, if my queue is empty, i will temp goes back to 4.12 and will see how is this a difference.

I can not say why not more other people have this problem. Maybe not everybody deals with big nzb files and queues  

[shypike]

Make sure you do not use the Classic template with large queues.

[wodix]

thanks for the template hint, i changed now to the plush template. I put now on an empty Sabnzbd+ Queue ~30 nzb files (~ 700 GB) in the folder scan directory. The Queue is filled up very quick, till to the last 2 nzb. Now the bad performance is comming back. Any Action like delete, reload, etc hang. The CPU has 100 % for the process (on a very strong system) for a very long time (5 minutes) and to do anything else is not possible. After this Sabnzbd+ works normal (download, refresh, config parts,..). But if i delete now 1 item in the queue, it will take 2-3 minutes and has the same bad performance.
As i wrote earlier in this thread, i could see that the SABnzbd_nzo_* files in the cache folder are recreated multiple times. After the queue is smaller, everthing works normal.

I could this reproduce on different systems with the final version 0.5, so i think this is not an eviroment problem.

[shypike]

On Linux, is there a number of files in a folder, beyond which file access slows down?
I know there is on Windows.
Currently there is a huge number of files in the cache folder
and maybe this limit is crossed for large queues.
For 0.6.0 we are working on storing all temporary files in the
"incomplete" folders instead of in "cache".

[wodix]

I can update my last post now. I make now a new fresh Windows Installation with the Sabnzbd+ Version 4.12, and make the same things (~30 nzb files, ~ 850 GB). Anything goes on quick and clean. The CPU of the System never hangs at 100 %. A item deleting in the queue takes a second, also with the classic template.

The other thing is, that sabnzbd+ takes much much less memory (~130 MB) in this situartion, as my 0.5 installations (> 700 MB).

For me it looks like that it is a good idea, to switch back for a while to 4.12 with my primary system. I will use the lovely new 0.5 feature (for example better RSS Support), with my second Installtion an System, witch will not so hard used in the daily work.

If i could test or describe anything else, please let me know. I will be happy to support the development to get fix this.

[wodix]

On Linux, is there a number of files in a folder, beyond which file access slows down?
I know there is on Windows.
Currently there is a huge number of files in the cache folder
and maybe this limit is crossed for large queues.
For 0.6.0 we are working on storing all temporary files in the
"incomplete" folders instead of in "cache".

I got the problem on windows and linux systems. I use also a big artice cache (512M), that prevent the big amounts of files in the cache folder.
On my last descripted test, with the Windows 4.12 Installation, i have no cache, and the cache folder takes > 13000 files without any problem. So something is realy going very different between version 0.4x and 0.5x.

[shypike]

I found that the full queue is saved to disk after a new NZB enters the queue.
Not strictly necessary and I will optimized this.
Saving the queue locks it (for good reasons) and so blocks both downloading and the UI.

I haven't looked at the difference in memory usage yet.

[wodix]

greate to read, that you found something that can be the reason for the described problem(s), and you can and/or will fix this.

For more details i will once more look, into the memory difference. I will setup in the moment two VMware Client (one 0.4.12 and one 0.5.0 on my ESX Server, with the same sabnzb+ settings. Then later maybe i can better describe what is the watched difference...

[shypike]

For some reason, after some time 0.5.1 starts keeping the full admin
for every NZB in memory, unlike 0.4.12 which keeps most on disk.
This explains the excessive memory usage.

[wodix]

thanks for the clarification on this issue!