High Memory Usage / Potential Memory Leaks Investigation Thread

[inpheaux]

I wish to start this thread because I think its time that we start finding solution for the bug which is eating my box free memory.

To locate the problem, I'm calling only users who have 100mbit internet connections on the Linux machines, (Ubuntu, Debian)

You would need to have true premium usenet access with granted 100mbit or more download speed.

To explain the problem:

After fresh start of SABNZBD the res mem usage would be like 10-20mb.

Start hard usage by downloading huge HD materials

for example
http://nzbmatrix.com/nzb-download.php?i ... p=1&cat=50

its a 46GB rls with files size of 488281K

Now tell me which cache size were you using, and please start monitoring the memory usage with for example HTOP

my cache_limit = 50M

Here is the tempo of leaking:

[16/07/10 21:25:22] I have now crazy mem usage like 465mb
[16/07/10 21:37:42] http://img688.imageshack.us/img688/6336/sabusage.png
[16/07/10 21:45:48] usage 526mb
[16/07/10 21:48:14] http://img130.imageshack.us/img130/6556/sabusage2.png
[16/07/10 22:07:50] sab res mem usage 659mb
[16/07/10 22:07:52] http://img713.imageshack.us/img713/2686/sabusage3.png
[16/07/10 22:18:26] usage 788mb
http://img689.imageshack.us/img689/33/sabusage4.png
[16/07/10 22:55:33] usage 885mb
http://img824.imageshack.us/img824/460/sabusage5.png
[16/07/10 23:20:23] usage 930mb
http://img718.imageshack.us/img718/1451/sabusage6.png
...

Download is still not completed I will report in this post how much res mem would sab use after the download is completed will it return to 10-20 or 60mb or will it leak in idle state to 200-300 or more mb.

I wish to raise the question, is there some incompatibility with the kernel or with the  OS which I'm using Debian and Ubuntu server, maybe its some system settings which is causing this issue.
Maybe the 100mbit is too fast for sab engine.

[shypike]

Good initiative.
Start by setting the logging to a minimum (errors/warnings only).
Logging is quite a heavy load too.
We are working on reducing useless logging.

[loopdemack]

The conclusion:


[17/07/10 00:32:41] To conclude the situation, after download is finished and completed, the two BR HD movies around 90GB file size 488281K sabnzbd usage in idle without anything is now 979mb
[17/07/10 00:34:49] http://img28.imageshack.us/img28/4849/s ... pleted.png

[rascalli]

I have 100mbit server, I can download with 100Mbit.
And I have no problems , mem usage stays pretty low & when it goes up, after it is finished it goes down again

[loopdemack]

I have 100mbit server, I can download with 100Mbit.
And I have no problems , mem usage stays pretty low & when it goes up, after it is finished it goes down again

Please state your OS, kernel,  cache size and proof when downloading exact nzb, I posted a link in the first post.

Please take a screenshot from the htop, like I did.

[sander]

Interesting subject.

Question: is it bad that a process uses a lot of (virtual) memory? I thought Linux processes would use as much memory as they could, and as long as they could?

Using a one-liner to report the SABnzbd memory usage, I've created a MRTG real time overview of my sabmemusage:

http://wattcher.015.info/mrtg/sabmemusage.html

Blue = real MEM, Green = VIRT mem.


And, no, I haven't got a 100 Mbps connection, and neither 40 GB downloads, just a 11 Mbps DSL with 5 GB downloads ...

One-liner

Code: Select all

sander@athlon64:~$ cat ./sabnzbd-memory-usage.sh
top -bn1 | grep -i sabnzbd | awk '{ print $5 "\n" $6 "\n\n" }' | sed -e s/m//g

sander@athlon64:~$

[inpheaux]

Its important that we test with huge downloads, because of a few reasons, file size of each rar should be big, in the case of nzb I linked it has 480mb files,
which means that we will make sab in environment which is not below the normal cache size of sab engine.

The second thing is that I saw very unstable sab engine under fast connections, seems its start going crazy because the code and engine is perfected for the slower connections, and than joining threads is done inside the limits of a cache on the slower connections.


Please even with your slower connection could you start for the test the nzb
[redacted]
If you cant download this nzb I can upload it to the rapidshare or some other hosting site

I'm calling again all Linux users with 100mbit or more to join in this test.

State your OS, your kernel, your sabnzbd settings, and webgui used for sabnzbd.

[Admin: if you're going to link to nzbs, find content that's not under copyright]

[inpheaux]

I've still never seen this happen, but I made your thread title a little more descriptive so more people might find it.

[loopdemack]

Ok

[sander]

[redacted]
[Admin: if you're going to link to nzbs, find content that's not under copyright]

[inpheaux]

[redacted]
[Admin: if you're going to link to nzbs, find content that's not under copyright]

[loopdemack]

You are absolutely right

[sander]

On my 11 Mbps links, I'm now downloading a 47 GB file. SAB's memory usage is here: http://wattcher.015.info/mrtg/sabmemusage.html

OS: Ubuntu 10.04 64-bit, 4GB RAM, SAB 0.5.3

Code: Select all

sander@athlon64:~$ free
             total       used       free     shared    buffers     cached
Mem:       3992028    3551908     440120          0     896432    1553536
-/+ buffers/cache:    1101940    2890088
Swap:      2048276       7428    2040848
sander@athlon64:~$

[loopdemack]

On my 11 Mbps links, I'm now downloading a 47 GB file. SAB's memory usage is here: http://wattcher.015.info/mrtg/sabmemusage.html

OS: Ubuntu 10.04 64-bit, 4GB RAM, SAB 0.5.3

Code: Select all

sander@athlon64:~$ free
             total       used       free     shared    buffers     cached
Mem:       3992028    3551908     440120          0     896432    1553536
-/+ buffers/cache:    1101940    2890088
Swap:      2048276       7428    2040848
sander@athlon64:~$

Which version of kernel did you used, are you using python 2.4, 2.5 or some other, were you using pre-compiled version of yEnc, were you using some cache_limit= in sabnzbd.ini or you disabled it with the 0?

As I see your machine started leaking , we should see how much will it leak, and will it released the memory after the download is completed.

As we saw leaking is 5 time smaller and slower without cache enabled, but it still leaks.

[sander]

I'm using a plain, non-tweaked OS and SAB. See below. Let me know if you need more info.

Code: Select all

sander@athlon64:~$ uname -a
Linux athlon64 2.6.31-21-generic #59-Ubuntu SMP Wed Mar 24 07:28:27 UTC 2010 x86_64 GNU/Linux
sander@athlon64:~$

sander@athlon64:~$ python
Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
sander@athlon64:~$

sander@athlon64:~$ grep -i cache .sabnzbd/sabnzbd.ini
cache_dir = /media/5E0C8BBD0C8B8F29/downloads/cache
cache_limit = ""
sander@athlon64:~$

sander@athlon64:~$ locate -i yenc
/usr/lib/perl5/Tk/DummyEncode.pm
/usr/lib/pymodules/python2.6/_yenc.so
/usr/lib/pymodules/python2.6/yenc-0.3.egg-info
/usr/lib/pymodules/python2.6/yenc.py
/usr/lib/pymodules/python2.6/yenc.pyc
/usr/lib/python-support/python-yenc
/usr/lib/python-support/python-yenc/python2.5
/usr/lib/python-support/python-yenc/python2.6
/usr/lib/python-support/python-yenc/python2.5/_yenc.so
/usr/lib/python-support/python-yenc/python2.6/_yenc.so
/usr/share/doc/python-yenc
/usr/share/doc/python-yenc/README
/usr/share/doc/python-yenc/TODO
/usr/share/doc/python-yenc/changelog.Debian.gz
/usr/share/doc/python-yenc/changelog.gz
/usr/share/doc/python-yenc/copyright
/usr/share/doc/python-yenc/examples
/usr/share/doc/python-yenc/yenc-draft.1.3.txt.gz
/usr/share/doc/python-yenc/examples/test.py
/usr/share/python-support/python-yenc
/usr/share/python-support/python-yenc/yenc-0.3.egg-info
/usr/share/python-support/python-yenc/yenc.py
/var/lib/dpkg/info/python-yenc.list
/var/lib/dpkg/info/python-yenc.md5sums
/var/lib/dpkg/info/python-yenc.postinst
/var/lib/dpkg/info/python-yenc.prerm
sander@athlon64:~$