Hi,
I am a new user of Sabnzbd on the Iomega Storcentre IX2and I am very happy with all it's features and compatibility with other applications such as Sickbeard and Couchpotatoe.
However I have noticed 2 or 3 times now that my downloads get stuck about half way..
The first time it occurred I was downloading a complete season of a tv-series which was about 40 GB.. I figured maybe somehow this was too big for Sabnzbd to handle.. I tried restarting sabnzbd, my nas, nothing happening..
I confirmed my newsprovider (astraweb) was working properly after testing that from my PC.
Only after removal of the nzb and restart of NAS it would work again for new downloads...
After that I had many successful downloads.. until todat when I tried to download a 15GB movie and again it got stuck about half way (8 GB)...
I look in the details and I see until file 50 has been downloaded, however files 51-99 are stuck.
In the logs the last few entries I see are:
2012-09-23 00:40:40,907::INFO::[__init__:884] /mnt/pools/A/A0/Downloads/_incomplete/zed17485.nfo.1/__ADMIN__/SABnzbd_nzf_jCQQaV removed
2012-09-23 00:40:40,984::INFO::[__init__:884] /mnt/pools/A/A0/Downloads/_incomplete/zed17485.nfo.1/__ADMIN__/SABnzbd_nzf_S8YTaD removed
2012-09-23 00:40:40,987::INFO::[__init__:884] /mnt/pools/A/A0/Downloads/_incomplete/zed17485.nfo.1/__ADMIN__/SABnzbd_nzf_s6vZ2V removed
2012-09-23 00:40:40,989::INFO::[__init__:884] /mnt/pools/A/A0/Downloads/_incomplete/zed17485.nfo.1/__ADMIN__/SABnzbd_nzf_gxolgb removed
2012-09-23 00:40:40,991::INFO::[__init__:884] /mnt/pools/A/A0/Downloads/_incomplete/zed17485.nfo.1/__ADMIN__/SABnzbd_nzf_RgKgCB removed
2012-09-23 00:40:41,076::INFO::[nzbqueue:176] Saving queue
2012-09-23 00:40:41,079::INFO::[__init__:895] Saving data for queue9.sab in /mnt/pools/A/A0/Media/.Apps/sabnzbd/admin/queue9.sa
This is about 2.5 hours ago (at the time of writing it's 2:11) and nothing happened since, except a reboot which didn't fix it.
Hope somebody can help as this is a bit annoying if I can't rely on downloads to complete..
Many thanks!
Vincent
Downloads get stuck about half way
Forum rules
Help us help you:
Help us help you:
- Are you using the latest stable version of SABnzbd? Downloads page.
- Tell us what system you run SABnzbd on.
- Adhere to the forum rules.
- Do you experience problems during downloading?
Check your connection in Status and Interface settings window.
Use Test Server in Config > Servers.
We will probably ask you to do a test using only basic settings. - Do you experience problems during repair or unpacking?
Enable +Debug logging in the Status and Interface settings window and share the relevant parts of the log here using [ code ] sections.
-
- Newbie
- Posts: 10
- Joined: September 22nd, 2012, 6:59 pm
Re: Downloads get stuck about half way
How much memory does this box have?
-
- Newbie
- Posts: 10
- Joined: September 22nd, 2012, 6:59 pm
Re: Downloads get stuck about half way
Hi Shypike,
Thanks for looking into this..
It's the latest model in the IX2 series but I see it only has 256 MB of DDR3 ram (CPU 1.6 GHz), is this sufficient?
Thruth be told, when I was doing the download yesterday at the same time I was doing an rsync...
Cheers,
Vincent
Thanks for looking into this..
It's the latest model in the IX2 series but I see it only has 256 MB of DDR3 ram (CPU 1.6 GHz), is this sufficient?
Thruth be told, when I was doing the download yesterday at the same time I was doing an rsync...
Cheers,
Vincent
Re: Downloads get stuck about half way
256M is rather small.
SABnzbd being written in Python without much effort spent on saving memory,
such an amount is a challenge.
More so when more software is competing for memory.
It's important not to have an article cache in Config->General (set it to 0).
I believe there are methods to set an upper limit on memory consumption for a specific app
in Linux, but I don't know the details.
You can ask in out Ubuntu/Debian forum.
SABnzbd being written in Python without much effort spent on saving memory,
such an amount is a challenge.
More so when more software is competing for memory.
It's important not to have an article cache in Config->General (set it to 0).
I believe there are methods to set an upper limit on memory consumption for a specific app
in Linux, but I don't know the details.
You can ask in out Ubuntu/Debian forum.
Re: Downloads get stuck about half way
See https://forums.sabnzbd.org/viewtopic.ph ... mit#p57630 for memory limiting: ulimit is your friend.
However, I don't know what happens when you hard-limit memory for the SABnzbd process. Maybe hard crashes and hard "out of memory" messages.
@OP: on your Iomega Storcenter IX2, can you enable swap space? I'm assuming your Storcenter has no swap enabled ... check with "free -m". I have a system with 256MB RAM and 512 MB RAM, and SABnzbd can download 5GB posts. I can't test bigger downloads because my harddisk space is limited on that system.
The import numbers are:
106: 106 MB free memory)
511 or 456: the MB's from swap
EDIT:
Another useful memory detection command:
However, I don't know what happens when you hard-limit memory for the SABnzbd process. Maybe hard crashes and hard "out of memory" messages.
@OP: on your Iomega Storcenter IX2, can you enable swap space? I'm assuming your Storcenter has no swap enabled ... check with "free -m". I have a system with 256MB RAM and 512 MB RAM, and SABnzbd can download 5GB posts. I can't test bigger downloads because my harddisk space is limited on that system.
Code: Select all
sander@toverdoos:~$ free -m
total used free shared buffers cached
Mem: 237 219 18 0 54 33
-/+ buffers/cache: 130 106
Swap: 511 55 456
sander@toverdoos:~$
106: 106 MB free memory)
511 or 456: the MB's from swap
EDIT:
Another useful memory detection command:
Code: Select all
sander@toverdoos:~$ cat /proc/meminfo | grep -i -e ^Mem -e Swap | grep -i -e total -e free
MemTotal: 243264 kB
MemFree: 23200 kB
SwapTotal: 524284 kB
SwapFree: 467392 kB
sander@toverdoos:~$
-
- Newbie
- Posts: 10
- Joined: September 22nd, 2012, 6:59 pm
Re: Downloads get stuck about half way
Hi Shypike and Sander,
Thanks for the tips, they are very much appreciated.. I disabled the article cache as suggested..
Sander, please see output below on my IX2, I guess this means swapping is enabled?
I'll look into the ulimit and do some more testing..
In the meantime is there a way to get more detailed logging to see what happens when I have this issue the next time?
Thanks,
Vincent
Thanks for the tips, they are very much appreciated.. I disabled the article cache as suggested..
Sander, please see output below on my IX2, I guess this means swapping is enabled?
Code: Select all
root@ix2:/# free -m
total used free shared buffers cached
Mem: 241 217 24 0 0 34
-/+ buffers/cache: 181 59
Swap: 511 112 399
root@ix2:/#
Code: Select all
root@ix2:/# cat /proc/meminfo | grep -i -e ^Mem -e Swap | grep -i -e total -e free
MemTotal: 247164 kB
MemFree: 24200 kB
SwapTotal: 524280 kB
SwapFree: 408896 kB
root@ix2:/#
In the meantime is there a way to get more detailed logging to see what happens when I have this issue the next time?
Thanks,
Vincent
Re: Downloads get stuck about half way
You have swap ... that's good.
Set logging to debug: http://localhost:8080/status/ -> set logging to "+Debug"
You could monitor the memory usage by SABnzbd:
The VIRTual memory is large, but less relevant. I believe the more important is RESident size: "The non-swapped physical memory a task has used.". So monitor that ... does it go up considerably during the download of a big post?
Set logging to debug: http://localhost:8080/status/ -> set logging to "+Debug"
You could monitor the memory usage by SABnzbd:
Code: Select all
sander@R540:~$ top -bn1 | grep -i -e sab -e VIRT
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4395 sander 20 0 2524m 57m 11m S 0 1.5 0:08.65 SABnzbd.py
sander@R540:~$
-
- Newbie
- Posts: 10
- Joined: September 22nd, 2012, 6:59 pm
Re: Downloads get stuck about half way
Hi Sander,
Here's a top during a download of the same movie as before (15GB) (will monitor using your command as well)..
So far so good..
Seems another process for the Twonky Media server uses up a lot of CPU but not so much memory. Thinking of disabling it anyway, it was installed standard but it uses up huge chunk of CPU..
Here's a top during a download of the same movie as before (15GB) (will monitor using your command as well)..
So far so good..
Seems another process for the Twonky Media server uses up a lot of CPU but not so much memory. Thinking of disabling it anyway, it was installed standard but it uses up huge chunk of CPU..
Code: Select all
root@ix2:/# top
top - 17:26:59 up 11 min, 1 user, load average: 3.41, 1.74, 0.84
Tasks: 116 total, 1 running, 115 sleeping, 0 stopped, 0 zombie
Cpu(s): 87.1%us, 10.6%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 2.3%si, 0.0%st
Mem: 247164k total, 226072k used, 21092k free, 668k buffers
Swap: 524280k total, 6520k used, 517760k free, 89812k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3173 root 20 0 334m 33m 4404 S 54.6 14.1 1:41.82 python2.6
2485 root 20 0 17848 8440 1272 S 43.2 3.4 1:43.58 twonkymediaserv
155 root 20 0 0 0 0 S 0.7 0.0 0:00.48 pdflush
3222 root 20 0 199m 26m 4212 S 0.7 10.9 0:09.27 python2.6
3668 root 20 0 2240 1108 872 R 0.7 0.4 0:00.55 top
1399 root 20 0 302m 6576 4264 S 0.3 2.7 0:05.26 svcd
1 root 20 0 1976 596 568 S 0.0 0.2 0:02.58 init
2 root 15 -5 0 0 0 S 0.0 0.0 0:00.01 kthreadd
3 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
-
- Newbie
- Posts: 10
- Joined: September 22nd, 2012, 6:59 pm
Re: Downloads get stuck about half way
Update:
(still downloading)
How do I identify which python process is sabnzbd?
(still downloading)
How do I identify which python process is sabnzbd?
Code: Select all
root@ix2:/# free -m
total used free shared buffers cached
Mem: 241 220 20 0 0 74
-/+ buffers/cache: 145 95
Swap: 511 6 505
Code: Select all
root@ix2:/# top
top - 17:34:59 up 19 min, 1 user, load average: 5.85, 4.79, 2.72
Tasks: 116 total, 2 running, 114 sleeping, 0 stopped, 0 zombie
Cpu(s): 84.2%us, 12.2%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 3.6%si, 0.0%st
Mem: 247164k total, 222412k used, 24752k free, 444k buffers
Swap: 524280k total, 7884k used, 516396k free, 85628k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2485 root 20 0 17848 8948 1268 S 49.2 3.6 5:14.27 twonkymediaserv
3173 root 20 0 336m 35m 4000 S 49.2 14.7 5:50.73 python2.6
156 root 15 -5 0 0 0 S 0.7 0.0 0:02.14 kswapd0
1399 root 20 0 302m 6136 3960 S 0.3 2.5 0:06.83 svcd
3222 root 20 0 199m 25m 3692 S 0.3 10.6 0:11.66 python2.6
Re: Downloads get stuck about half way
With "-c" you should see the complete command line commands:
Code: Select all
sander@R540:~$ top -bn1 -c | grep -i sabnzbd
3227 sander 20 0 97428 948 828 S 0 0.0 0:00.00 grep --color=auto -i sabnzbd
4395 sander 20 0 2524m 62m 11m S 0 1.7 1:57.91 /usr/bin/python -OO ./SABnzbd.py
sander@R540:~$
-
- Newbie
- Posts: 10
- Joined: September 22nd, 2012, 6:59 pm
Re: Downloads get stuck about half way
Thanks, here's the output (without the grep as I wanted to see other processes as well):
Code: Select all
root@ix2:/# top -bn1 -c
top - 17:50:14 up 35 min, 1 user, load average: 5.64, 5.60, 4.53
Tasks: 117 total, 2 running, 115 sleeping, 0 stopped, 0 zombie
Cpu(s): 66.1%us, 11.1%sy, 0.2%ni, 19.6%id, 0.6%wa, 0.0%hi, 2.4%si, 0.0%st
Mem: 247164k total, 226696k used, 20468k free, 628k buffers
Swap: 524280k total, 9180k used, 515100k free, 83160k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3173 root 20 0 340m 38m 2932 S 47.0 15.8 13:24.40 /opt/bin/python2.6 /mnt/pools/A/A0/Media/.Apps/sabnzbd/SABnzbd.py --server 192.168.1.39:9111 --d
Re: Downloads get stuck about half way
I've written a patch for SABnzbd (sabnzbd/misc.py) to show the VIRT and RES memory usage (Linux only). Example screendump:
I downloaded a 7GB post on my 256 MB RAM / 512 MB swap, and I did *not* see the memory go up.
If you want to use it, get the file http://www.appelboor.com/dump/misc.py and copy it over sabnzbd/misc.py
I downloaded a 7GB post on my 256 MB RAM / 512 MB swap, and I did *not* see the memory go up.
If you want to use it, get the file http://www.appelboor.com/dump/misc.py and copy it over sabnzbd/misc.py
-
- Newbie
- Posts: 10
- Joined: September 22nd, 2012, 6:59 pm
Re: Downloads get stuck about half way
Hi Sander,
That's great, very useful!
Will try it tonight when I'm home.
Cheers,
Vincent
P. S. I like your domain name (I'm Dutch too)!
That's great, very useful!
Will try it tonight when I'm home.
Cheers,
Vincent
P. S. I like your domain name (I'm Dutch too)!
Re: Downloads get stuck about half way
Sander: I integrated your code into 0.7.4.
Thanks!
Thanks!
Re: Downloads get stuck about half way
For what it's worth, I'm running sabnzbd (and sickbeard) on a Seagate Dockstar, which has only 128 MB of RAM, and it runs fine. It boots Debian squeeze from a usb flash drive and it has a swap partition on the flash drive. I have downloaded things as large as 15 to 20 GB, no problem.
Here's what the output of htop looks like with the machine idling:
Here's what the output of htop looks like with the machine idling:
Code: Select all
CPU[|| 3.9%] Tasks: 70 total, 1 running
Mem[||||||||||||102/122MB] Load average: 0.09 0.06 0.06
Swp[| 28/999MB] Uptime: 4 days, 18:16:29
S CPU% MEM% TIME+ Command
R 2.0 0.5 27:01.50 htop