High Memory Usage / Potential Memory Leaks Investigation Thread

[sander]

BTW: should I close the other apps on my 4GB system to see if SAB is going to eat all the memory?

[loopdemack]

Please just take a look at sabnzbd.ini what do you got for "cache_limit="

[sander]

Please just take a look at sabnzbd.ini what do you got for "cache_limit="

Like I posted: nothing filled out. It's empty, both in the GUI and the .ini

[sander]

FYI: I juist closed some other apps on my system, and memory usage is going up.

I am not an expert, but to me, this is still just a signal of *memory usage*, not of a *memory leak*. The real question is IMHO: is SABnzbd able to release memory (without restarting) to the OS and other apps. If so, I don't see a memory leak problem.

[sander]

SAB's mem usage is still going up (now at 1370 MB, see attachment), but alas one parameters (REAL mem) is missing (or: at 1 MB). I haven't got remote access to my machine at home, so I can't check right now what's going on.

[loopdemack]

SAB's mem usage is still going up (now at 1370 MB, see attachment), but alas one parameters (REAL mem) is missing (or: at 1 MB). I haven't got remote access to my machine at home, so I can't check right now what's going on.

I'm sure its leaking and not usage, also in 4 of 13 sab will broke or freeze the process due to very unstable process because of his internal environment which didnt predicted how to work in leaked state,
also in your case you didn't used caching and your leaking was not so aggressive, if you would enable cache leaking would be faster and more extreme.
Without caching its very hard to get full speed, its only possible if sab has dedicated machine just for his downloading.
If you are using torrents, ftp-server, web-server and other things, than caching the sab is the most important factor, of course on machines with faster internet like 100mbit or 1gbit.

And in my testing 13 out of 13 I newer saw sab releasing the memory back after completed download.

Also don't forget this, if you would continue using it for a few days more, if its not broke or freeze you would completely kill the machine, because the load would rise to astronomical levels.

[sander]

FYI: This just arrived in my mailbox:

Code: Select all

SABnzbd has downloaded "blabla"

Finished at 2010-07-19 13:35:59
Downloaded 44.33 GB

Results of the job:

Stage Download
   Downloaded in 12 hours 46 minutes 40 seconds at an average of 1010.45 KB/s

Stage Repair
   [HDC-blabla] Quick Check OK

Stage Unpack
   [HDC-blabla] Unpacked 850 files/folders in 2 hours 57 mins 42 seconds

[sander]

Ah, I found the reason my tool was reporting "1 MB" RES memory usage: it was reported as 1.0g, of which only the "1.0" was seen. :-(

Anyway: I've now started a lot of programs (chrome, firefox, mplayer, virtualbox), but it (indeed) seems SABnzbd is not releasing any memory ...

Code: Select all

sander@athlon64:~$ ./sabnzbd-memory-usage.sh 
1348
1.0g


sander@athlon64:~$ top -bn1 | grep -i sab
 7804 sander    20   0 1348m 1.0g 3196 S    0 26.4 206:25.05 SABnzbd.py                                                                                      
sander@athlon64:~$ 

[sander]

Update: SAB *is* releasing memory: RES has gone down from 1000 MB to 544 MB:

Code: Select all

sander@athlon64:~$ ./sabnzbd-memory-usage.sh 
1348
544


sander@athlon64:~$

[sander]

why is it using 544MB memory though, shouldn't it be more like 20MB

Good question! ;-)

Usage is now at 496 MB, and still going down, see http://wattcher.015.info/mrtg/sabmemusage.html

Maybe a python memory profiler could help: http://www.velocityreviews.com/forums/t ... filer.html so heapy?

[loopdemack]

@sander, could you test the same nzb but with caching enabled in the sabnzbd.ini, it would help a lot in finding the reasons behind the leakage, would you experience the same leak or will it be much faster, higher and more extreme.

Use cache_limit=150M or cache_limit=50M

Also it would be good that we found how to examine with python memory profiler whats going on.

[sander]

I've started a new download ... and Ouch: "Sysload 16.54, 16.47, 11.85", so the system is now extremely unresponsive. I'll close virtualbox and some other apps.

[sander]

@sander, could you test the same nzb but with caching enabled in the sabnzbd.ini, it would help a lot in finding the reasons behind the leakage, would you experience the same leak or will it be much faster, higher and more extreme.

Use cache_limit=150M or cache_limit=50M

Also it would be good that we found how to examine with python memory profiler whats going on.

OK, limit is now on 50M and SAB has been restarted.

Code: Select all

sander@athlon64:~$ grep -i cache_limit .sabnzbd/sabnzbd.ini 
cache_limit = 50M
sander@athlon64:~$


sander@athlon64:~$ ./sabnzbd-memory-usage.sh 
298
57


sander@athlon64:~$

[sander]

Test complete: with 50 MB cache_limit, the download succeeds with a little bit less memory usage. Memory is still not automagically released afterwards, but I have to little Linux knowledge to decide whether this is a bug (or a feature).

I'll now wait for the guru's (a.k.a. SAB devs) and their analysis ...

[sander]

I've played with heapy.py in a small script creating 100.000 entries of type (long?) integer in a list, and heapy seems to do what it should do: show memory usage. Result below.

Question to SAB devs: If I want to print the heap in SAB eacht x minutes (or after each download phase), how can I do that?


The script:

Code: Select all

# with some help from the Python Phrasebook: Essential Code and Commands

import random
# On Ubuntu: sudo apt-get install python-guppy
from guppy import hpy; hp=hpy()

def dumpheap() :

	mislukt = 0
	gelukt = False
	while not gelukt:
		try:
			print hp.heap()
			gelukt = True
		except:
			print "niet gelukt"
			mislukt += 1
	print "Gelukt, na x pogingen:",mislukt
	return

dumpheap()
list = []
for i in range(1, 100000):
	list.append(random.randint(100,10000000000000))
dumpheap()

The outputt:

Code: Select all

sander@quirinius:~/heapy-exp$ 
sander@quirinius:~/heapy-exp$ time python testje.py 
Partition of a set of 24474 objects. Total size = 1796772 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0  10881  44   707552  39    707552  39 str
     1   5599  23   203984  11    911536  51 tuple
     2    344   1   109376   6   1020912  57 dict (no owner)
     3   1583   6   107644   6   1128556  63 types.CodeType
     4     70   0   107056   6   1235612  69 dict of module
     5    180   1    95904   5   1331516  74 dict of type
     6    200   1    88716   5   1420232  79 type
     7   1507   6    84392   5   1504624  84 function
     8    124   1    67552   4   1572176  88 dict of class
     9   1027   4    36972   2   1609148  90 __builtin__.wrapper_descriptor
<89 more rows. Type e.g. '_.more' to view.>
Gelukt, na x pogingen: 0
niet gelukt
Partition of a set of 124483 objects. Total size = 4207376 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0  99998  80  1999936  48   1999936  48 long
     1  10883   9   707648  17   2707584  64 str
     2    131   0   420064  10   3127648  74 list
     3   5598   4   203956   5   3331604  79 tuple
     4    350   0   110192   3   3441796  82 dict (no owner)
     5   1583   1   107644   3   3549440  84 types.CodeType
     6     70   0   107056   3   3656496  87 dict of module
     7    180   0    95904   2   3752400  89 dict of type
     8    200   0    88716   2   3841116  91 type
     9   1506   1    84336   2   3925452  93 function
<89 more rows. Type e.g. '_.more' to view.>
Gelukt, na x pogingen: 1

real	0m4.482s
user	0m4.264s
sys	0m0.060s
sander@quirinius:~/heapy-exp$