Memory leak?

Get help with all aspects of SABnzbd
Forum rules
Help us help you:
  • Are you using the latest stable version of SABnzbd? Downloads page.
  • Tell us what system you run SABnzbd on.
  • Adhere to the forum rules.
  • Do you experience problems during downloading?
    Check your connection in Status and Interface settings window.
    Use Test Server in Config > Servers.
    We will probably ask you to do a test using only basic settings.
  • Do you experience problems during repair or unpacking?
    Enable +Debug logging in the Status and Interface settings window and share the relevant parts of the log here using [ code ] sections.
finalius
Newbie
Newbie
Posts: 25
Joined: October 30th, 2011, 9:33 am

Memory leak?

Post by finalius »

When adding many large jobs to the queue, SAB seems to encounter some kind of memory leak. It ends up using about 50% of my system memory (1GB) and never releases that again, even after all downloads are finished. Also, downloading and post-processing gets slower over time.

I've paused all my downloads, waited for SAB to settle and then took a bunch of screenshots. IMHO the memory consumption in this scenario should go back to almost zero, because:
- nothing in Cache
- nothing to do (no de-yencing, no repairing, no unraring, no downloading, no anything)

Image
as you can see, SAB is sitting around on its ass lazily (which is ok)

Image
Webinterface tells the same story - nothing to do. But you can see that the time between finished jobs suddenly jumps at around 4am in the history queue. before, it took SAB ~20 minutes per job, then it's sometimes well over the one hour mark.

Image
Memory usage is 47% of 2GB, even though there is nothing to do.
User avatar
shypike
Administrator
Administrator
Posts: 19774
Joined: January 18th, 2008, 12:49 pm

Re: Memory leak?

Post by shypike »

Some Linux ports of Python refuse to return memory to the OS
or are at least very reluctant to do so.
A proper OS will swap out unused memory anyway, so it shouldn't hurt much.
finalius
Newbie
Newbie
Posts: 25
Joined: October 30th, 2011, 9:33 am

Re: Memory leak?

Post by finalius »

I only have 200MB Swap because my OS is on a 2GB USB Stick :) the OS is debian. Are you sure this couldn't be handled via code?
User avatar
sander
Release Testers
Release Testers
Posts: 8830
Joined: January 22nd, 2008, 2:22 pm

Re: Memory leak?

Post by sander »

First of all:
SABnzbd loves memory, and loves using it. (If you want a NZB-downloader with a small footprint, consider nzbget. See http://ipv6-or-no-ipv6.blogspot.com/201 ... -1110.html for an instruction).
SABnzbd uses more memory with large (> 10GB) downloads
In general, it's my experience on Linux, that the OS will fill up all memory. The reclaiming of memory will only happen when needed by other applications.

Now about the memory leak: IMHO (and confirmed by http://en.wikipedia.org/wiki/Memory_leak), something is only a memory leak if a program uses a lot of memory when not needed AND it cannot be reclaimed by the OS / other programs. So, what happens when SABnzbd is idle, has nothing in the queue, and you start up other memory-loving programs? Does SABnzbd's memory usage go down then?
exussum
Jr. Member
Jr. Member
Posts: 79
Joined: November 18th, 2010, 2:51 am

Re: Memory leak?

Post by exussum »

Usually that number is the amount requested by the processes, Which isnt equal to the amount used. It can also show weirdly for shared libraries as you have sickbeard running too that may be a cause.

You also have paused jobs, so the memory for downloading them is still "active" though 1GB is still a large number
finalius
Newbie
Newbie
Posts: 25
Joined: October 30th, 2011, 9:33 am

Re: Memory leak?

Post by finalius »

I don't have an issue with SAB using 1GB of memory while actually doing stuff, but "stockpiling" that memory seems like a bad approach, especially since there can be absolutely nothing meaningful in there once the queue is paused/empty and all yencs are decoded - or am I wrong?

I will check whether that memory can be reclaimed by the OS/other processes and report back here.
finalius
Newbie
Newbie
Posts: 25
Joined: October 30th, 2011, 9:33 am

Re: Memory leak?

Post by finalius »

Okay, so I've thrown in a bunch of large downloads yesterday night and went to bed. This morning, things looked like this:
Image
It's currently roughly 9am, so SAB hasn't been doing anything for six hours.

Console shows the following:
Image
Not quite as bad as yesterday, but still SAB uses a solid 768MB or RAM for doing nothing.

So, how do I efficiently use up all the memory in my system to see whether SAB gives it back to me? My idea was tmpfs (or Ramdisk for Windows users). I create a ramdisk with size 2GB and start filling it, then monitor what the memory consumption of SAB does. Note that Linux does not allocate the memory upon creation of the tmpfs (opposed to Windows' Ramdisk if I'm not mistaken). The memory only gets used once you start filling the tmpfs. So I can easily create a 2GB tmpfs, even if that would consume all my system's memory if I fill it up to the last bit.
Image
After 400M, pretty much all the free memory should have been used up. Of course, much of the used memory was "cached" by the kernel and other system thingies, which will release the memory if things get a bit uncomfortable. I'll skip the 500M screen and go right to the 1000M screen, as things look pretty much the same. Even after 500M, the system found some other ram to give up. SAB didn't give in a bit of it's memory so far.

So, "let's fry the fucker" I figured and made a 1000M file. This would get the system seriously into trouble. As you can see, my three python daemons claim >40% of memory, and I'm sure the kernel would like a couple of MBs too. So at 1000M, things will get tight.
Image
Note the speed. I'm copying from /dev/zero (a virtual device creating only zeros) to a ramdisk, and the speed is 20mb/s !!! why? Because the system has to swap out everything to the slow USB drive. Sab gave me 1% of RAM back, still consuming shitloads.

And here I encountered a surprise. When I came back to my consoles from making my screenshots, SAB was gone.
Image
Now, thinking that this is a coincidence would be a bit much, right? SAB has never crashed for me before. So I assume it crashed because it was out of memory. This means that although SAB allocates 40% of the memory of my system, it can't even use it itsself anymore. Now this seems like a proper memory leak to me.

Just to finish the story, here's how things look after unmounting the tmpfs.
Image
Lots of free memory, and an overall happy system.

Cheers
User avatar
jcfp
Release Testers
Release Testers
Posts: 989
Joined: February 7th, 2008, 12:45 pm

Re: Memory leak?

Post by jcfp »

Now, thinking that this is a coincidence would be a bit much, right? SAB has never crashed for me before. So I assume it crashed because it was out of memory.
Program crash or the Linux oom-killer in action? Check system logs.
finalius
Newbie
Newbie
Posts: 25
Joined: October 30th, 2011, 9:33 am

Re: Memory leak?

Post by finalius »

Yup, you're right. It was the oom-killer. The fact stands that SAB rather gets killed than giving up some of its allocated memory.
exussum
Jr. Member
Jr. Member
Posts: 79
Joined: November 18th, 2010, 2:51 am

Re: Memory leak?

Post by exussum »

Hmm currently im downloading a file with sab and its sitting around 112mb of memory with a Article Cache Limit of 128M. Total memory on the box is 192Mb (VPS) though mine has not ran as long as yours, Constant start up and shut downs to free memory for other applications on there.

Ill try leaveing it open see if it starts consuming more. What OS and what version of sab ? (And python) ?
finalius
Newbie
Newbie
Posts: 25
Joined: October 30th, 2011, 9:33 am

Re: Memory leak?

Post by finalius »

Debian wheezy amd64
Python 2.7.2-9 (debian wheezy package)
Sab 0.6.10
User avatar
sander
Release Testers
Release Testers
Posts: 8830
Joined: January 22nd, 2008, 2:22 pm

Re: Memory leak?

Post by sander »

Is the tool 'pdb' useful for finding a memory leak? I tried to following the instructions on http://www.lshift.net/blog/2008/11/14/t ... mory-leaks , but after typing 'r' to run SABnzbd.py, it ran, but a CTRL-C stopped it all completely. So I was not sent back to the pdb shell ... :-(

And if a quick workaround is needed, an hourly call to API-restart (http://wiki.sabnzbd.org/api#toc21) could help.

FWIW: One ore two years ago I also researched memory usage by SAB, and how to limit it. It must be somewhere in the forum.
User avatar
sander
Release Testers
Release Testers
Posts: 8830
Joined: January 22nd, 2008, 2:22 pm

Re: Memory leak?

Post by sander »

Ah, found it:

http://forums.sabnzbd.org/viewtopic.php?t=4713

and "ulimit -d 500111 -m 500111 -v 500111" to limit SAB's mem usage to 500 MB. I don't know what happens if you do a 50GB download with those settings.
exussum
Jr. Member
Jr. Member
Posts: 79
Joined: November 18th, 2010, 2:51 am

Re: Memory leak?

Post by exussum »

hmm im using

Debain lenny 32 bit and thats the only difference

Wonder if theres a problem with 64 bit python ?
User avatar
shypike
Administrator
Administrator
Posts: 19774
Joined: January 18th, 2008, 12:49 pm

Re: Memory leak?

Post by shypike »

Except on OSX we don't support 64bit Python.
And even on OSX there might be some issues, which we are looking at.
64bit may work, it's just that we don't test with it.
Post Reply