0.5.2RC1: NOT Fixed: Large queue caused very slow UI and high memory usage

[OhNoes!]

Version: 0.5.2 RC1
OS: Windows 7
Install-type: Windows Installer
Skin (if applicable): Default
Firewall Software: Kaspersky Internet Security 2010
Are you using IPV6? No
Is the issue reproducible? Yes

The changelog of version 0.5.2RC1 reports that 'Large queue caused very slow UI and high memory usage' has been fixed.
I have had this issue for a long time so I decided to upgrade to version 0.5.2RC1 but sadly I have to report that this issue still hasn't been fixed. Actually, the loading time of the UI has even increased after upgrading to 0.5.2RC1 from 0.5.0 Final.
Momentarily, my queue is 352GB (Sick, I know) and the UI takes about 8 seconds to load on a Intel Core i5, 4GB DDR2 (Dell Studio 17 notebook) using FireFox...

[shypike]

It will not become better until the current jobs are out of the queue.

[OhNoes!]

It will not become better until the current jobs are out of the queue.

I did a clean install of 0.5.2RC1 with an empty queue before I started adding the 352GB queue and started downloading.

[shypike]

I really have no idea. I tested this with a 1T queue without any problems.
What the average size of the SABnzbd_nzo_* files in the cache folder?

A few checks:
How large is the cache folder altogether?
How many files? (at some point Windows gets very slow).
Do you use the "Classic" skin?  Performance of smpl and Plush is better.
Is the virus scanner active in the "cache" and "incomplete" folders?

[OhNoes!]

Answers in bold below.

What the average size of the SABnzbd_nzo_* files in the cache folder? 20-21KB

A few checks:
How large is the cache folder altogether? 489MB
How many files? (at some point Windows gets very slow). 6643 files
Do you use the "Classic" skin?  Performance of smpl and Plush is better. Yes, classic
Is the virus scanner active in the "cache" and "incomplete" folders? Yes

[shypike]

Don't let the virus scanner scan cache and incomplete.
It's pointless, you should only scan the final folders.

I suspect the cache is rather large, larger than it should be.
SABnzbd suffers from file leakage in the cache (will be solved in a later release).
Was the cache folder empty when you started adding jobs?
Maybe the file dates will tell you.

The next time your queue is empty, remove all SABnzbd_* files from the cache.

[ErikBrown]

Version: 0.5.2 final
OS: Windows XP pro SP3
Install-type: Windows Installer
Skin (if applicable): SMPL black
Firewall Software: none
Are you using IPV6? No
Is the issue reproducible? Yes

Shypike,

You mentioned that 489 MB or 6643 files in the cache directory is larger than it should be. I have 540 MB or 48502 files in the cache directory. Just like the original writer of this thread, my queue is very large and will not get empty anywhere soon. Is there a way to safely remove files from the cache folder without loosing the queue?

Regards,
Erik

[shypike]

Is there a way to safely remove files from the cache folder without loosing the queue?

To some extent.
First check if there are many SABnzbd_article_* files in your cache (I mean hundreds).
If so:

Pause all the jobs in the queue individually, except the top one.
Let the top job finish completely.
Remove all files of this pattern: SABnzbd_article_*
Unpause the jobs.

Alternatively:
Add one NZB you don't need to the queue.
Make it the second job.
Wait until the top job is finished.
Pause downloading.
Remove all files of this pattern: SABnzbd_article_*
Remove the junk job.
Resume downloading.


We're working on a better solution for the next major release.
That should avoid the whole issue.

[OhNoes!]

Don't let the virus scanner scan cache and incomplete.
It's pointless, you should only scan the final folders.

Kaspersky won't let me... Aargh I hate it. I even paid money for it.

I suspect the cache is rather large, larger than it should be.
SABnzbd suffers from file leakage in the cache (will be solved in a later release).
Was the cache folder empty when you started adding jobs?
Maybe the file dates will tell you.

Apparantly not. I uninstalled SABNZBD and removed all old files and folders and then reinstalled ver 0.5.2. The loading times seem to have decreased but still take quite long to load.

The next time your queue is empty, remove all SABnzbd_* files from the cache.

We're working on a better solution for the next major release.
That should avoid the whole issue.

Looking forward to it.

[shypike]

Kaspersky won't let me...

Does Kapersky scan every file or just the 'known" extensions?
In the latter case there wouldn't be a problem anyway.
Also I'm not claiming it's necessarily a problem.
It's just that I have seen some virus scanners perform very badly
when they do on-access scans on every single file.
I've see McAfee choke on files that were written to in increments
and it trying to re-scan the the full file every time.

Eric Brown has a problem with his almost 50,000 files, Windows doesn't like that.

The next major release will just store all temp files in the download folder of the NZB.
This prevents oversized folders and makes sure that all junk is removed
when the job has been post-processed.

I

[OhNoes!]

Kaspersky won't let me...

Just the 'known" extensions?

So it shouldn't be a problem. I also tried disabling Kaspersky, it makes no difference at all, so I think Kaspersky isn't causing the problem

[ErikBrown]

Shypike,

I have removed all the SABnzbd_article_* files. But that were only 198 files. The majority of the files are SABnzbd_nzf_* files and some others are SABnzbd_nzo_* files. What are these nzf and nzo files? Can they be removed also?

I selected all the SABnzbd_nzf_* files and with rightclick I tried to see the properties to check how many files there were. The PC did not respond for over 30 minutes till I stopped the process manually. So it appears that Windows has indeed a serious problem with that many files.

Regards,
Erik

[shypike]

Each SABnzbd_nzo file represents one NZB, there should be as many as in your queue.
Each SABnzbd_nzf represents one file to be downloaded.
So it could be that there are 50,000 files to be downloaded but it sounds like an awful lot.

Cleaning them out is very tricky, undoable actually.
There's no option but to let the queue run empty and delete what remains behind.