Cannot allocate memory while running rar_unpack

[snorg]

Hi there,

I'm having the most annoying trouble with this error and I can't for the life of me figure out why.

Error "[Errno 12] Cannot allocate memory" while running rar_unpack on <filename>

Obviously it would indicate a memory error, however the machine has 1GB RAM and the sysload reads
Sysload: 0.01 | 0.03 | 0.00 | V=2615M R=90M


Here's my top output

Mem: 1048576k total, 737320k used, 311256k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 622192k cached

So, what could be causing this? I'm so perplexed and I've spent hours trawling the forums for answers to no avail

[sander]

What kind of system?

Does it happen all the time? Also with small downloads?

Can you log onto the system (ssh), and try a manual "unrar x ..."?

[snorg]

Hi there,

Thanks for the reply

Ubuntu 12.04 is the OS

uname -a
Linux 2.6.32-042stab053.5 #1 SMP Tue Mar 27 11:42:17 MSD 2012 x86_64 x86_64 x86_64 GNU/Linux

Yes, it happens with all downloads even very small ones.

When I login and do a manual unrar x it works completely fine with no problems.

Any other suggestions or information I could check to help troubleshoot?

[sander]

Can you post the output of:

Code: Select all

top -bn1 | head -7

and

Code: Select all

unrar | head -5

[jcfp]

Ubuntu 12.04 is the OS

Linux 2.6.32-042stab053.5

These two don't match. A newish Ubuntu version with a several years old non-standard kernel? So what system is this really?

[snorg]

Not sure what to tell you, it was the most recent image on the OS install list offered by my VPS provider
Is the version of unrar correct? I'm using the latest 64-bit release direct from rarlabs, not the unrar-free package since I know it has problems

================================================

root@vps:~# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04 LTS"
root@vps:~# uname -a
Linux vps 2.6.32-042stab053.5 #1 SMP Tue Mar 27 11:42:17 MSD 2012 x86_64 x86_64 x86_64 GNU/Linux

================================================

root@vps:~/rar# top -bn1 | head -7
top - 17:01:04 up 19:57, 1 user, load average: 0.00, 0.00, 0.00
Tasks: 28 total, 1 running, 27 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.2%us, 0.1%sy, 0.0%ni, 99.6%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 1048576k total, 737792k used, 310784k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 154028k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

================================================

root@vps:~/rar# unrar | head -5

UNRAR 4.20 freeware Copyright (c) 1993-2012 Alexander Roshal

Usage: unrar <command> -<switch 1> -<switch N> <archive> <files...>
<@listfiles...> <path_to_extract\>

================================================

[sander]

You have no swap. That causes very hard memory limits, and AFAIK leading to problems like you describe.

However ... I would expect you could successfully download and unrar a 50MB post. Can you try that?

[snorg]

Just tried a 7MB post that was zipped, and the unzip function failed for the same reason (Error "[Errno 12] Cannot allocate memory" while running unzip() on <filename>)

I figured with 1GB RAM that swap space wouldn't be required, and I'm not sure if OpenVZ even supports swap

[jcfp]

Maybe limits imposed by the vps? Quota? Is all of that memory yours to use, or is this shared with others?
Because this is certainly enough, 1 GB of memory with 80% or more free in total shouldn't limit sabnzbd at all.

[sander]

Maybe limits imposed by the vps? Quota? Is all of that memory yours to use, or is this shared with others?
Because this is certainly enough, 1 GB of memory with 80% or more free in total shouldn't limit sabnzbd at all.

But the OP has 73% of his 1GB in use:

Mem: 1048576k total, 737792k used, 310784k free

[sander]

Just tried a 7MB post that was zipped, and the unzip function failed for the same reason (Error "[Errno 12] Cannot allocate memory" while running unzip() on <filename>)

Can you run a manual "unrar x ..." on the download stuff and post the full output here?

[snorg]

That's the strange part, manual unrar e x works completely fine -- I've put the info at the bottom.

You pointed out something good though - I'm using 73% of my memory, which is shockingly high.

I rebooted my VPS and after about 5 minutes it stabilized at 18MB usage.
Once I launched SAB, it immediately jumped to 550MB! However, when I run top it shows that SAB is only using 10% of my memory (110M)

What the !? This is the most strange behaviour - I've done it three times now to verify it wasn't a random occurrance
Even my top output shows that a majority of the memory is used, however in the process list nothing is using the memory!

================
top - 20:33:16 up 33 min, 1 user, load average: 0.00, 0.05, 0.01
Tasks: 23 total, 1 running, 22 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.1%us, 0.1%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 1048576k total, 734272k used, 314304k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 734272k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
507 root 20 0 2615m 110m 5164 S 0 10.8 1:57.12 SABnzbd.py
1 root 20 0 24000 1988 1340 S 0 0.2 0:00.06 init
2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd/22000
3 root 20 0 0 0 0 S 0 0.0 0:00.00 khelper/22000
101 root 20 0 17180 724 516 S 0 0.1 0:00.01 upstart-udev-br
116 root 20 0 21284 1080 772 S 0 0.1 0:00.00 udevd
179 root 20 0 49904 2800 2208 S 0 0.3 0:00.00 sshd
223 root 20 0 21280 644 344 S 0 0.1 0:00.00 udevd
226 root 20 0 21280 656 348 S 0 0.1 0:00.00 udevd
253 root 20 0 15136 560 364 S 0 0.1 0:00.00 upstart-socket-
264 root 20 0 14916 1116 920 S 0 0.1 0:00.00 xinetd
267 root 20 0 19060 1048 812 S 0 0.1 0:00.00 cron
301 syslog 20 0 12700 840 656 S 0 0.1 0:00.00 syslogd
334 root 20 0 78612 1120 424 S 0 0.1 0:00.00 saslauthd
335 root 20 0 78612 740 44 S 0 0.1 0:00.00 saslauthd
462 root 20 0 73308 3584 2768 S 0 0.3 0:00.06 sshd
474 root 20 0 18072 2092 1548 S 0 0.2 0:00.01 bash
506 root 20 0 24228 1496 924 S 0 0.1 0:00.00 screen
582 root 20 0 89032 2280 592 S 0 0.2 0:00.02 sendmail-mta
621 root 20 0 69720 2896 1288 S 0 0.3 0:00.00 apache2
625 www-data 20 0 69744 2196 576 S 0 0.2 0:00.00 apache2
647 root 20 0 83076 21m 1688 S 0 2.1 0:00.08 miniserv.pl
1152 root 20 0 17156 1316 1060 R 0 0.1 0:00.04 top

======================================================

2012-10-16 20:29:06,059 ERROR: Error "[Errno 12] Cannot allocate memory" while running rar_unpack on <filename>
=============
root@vps:/tmp/incomplete/folder# unrar e filename.rar

UNRAR 4.20 freeware Copyright (c) 1993-2012 Alexander Roshal


Extracting from filename.rar

Extracting filename.mp4 4%

Extracting from filename.r00

... filename.mp4 8%

Extracting from filename.r01

... filename.mp4 12%

Extracting from filename.r02

... filename.mp4 16%

Extracting from filename.r03

... filename.mp4 20%

Extracting from filename.r04

... filename.mp4 24%

Extracting from filename.r05

... filename.mp4 28%

Extracting from filename.r06

... filename.mp4 32%

Extracting from filename.r07

... filename.mp4 36%

Extracting from filename.r08

... filename.mp4 40%

Extracting from filename.r09

... filename.mp4 44%

Extracting from filename.r10

... filename.mp4 48%

Extracting from filename.r11

... filename.mp4 52%

Extracting from filename.r12

... filename.mp4 57%

Extracting from filename.r13

... filename.mp4 61%

Extracting from filename.r14

... filename.mp4 65%

Extracting from filename.r15

... filename.mp4 69%

Extracting from filename.r16

... filename.mp4 73%

Extracting from filename.r17

... filename.mp4 77%

Extracting from filename.r18

... filename.mp4 81%

Extracting from filename.r19

... filename.mp4 85%

Extracting from filename.r20

... filename.mp4 89%

Extracting from filename.r21

... filename.mp4 93%

Extracting from filename.r22

... filename.mp4 97%

Extracting from filename.r23

... filename.mp4 OK
All OK
root@vps:/tmp/incomplete/folder#

[jcfp]

But the OP has 73% of his 1GB in use

Nope, most of that is cached or buffers, see the first post. That stuff can be cleared at any time by the kernel if need be, effectively making it free memory.

[snorg]

Is there some way I can force the information to be cleared and free the memory?

I actually purchased an increase in the RAM on my server so that it now has an additional 512MB. At first I had no troubles with unrar, however after a few files I'm back to square 1. Sab + Python are using 15% of my RAM, and I'm not really running anything else, so I have no idea where the rest of the RAM is being used..

root@vps:/tmp# free -m
total used free shared buffers cached
Mem: 1536 1013 522 0 0 1013
-/+ buffers/cache: 0 1536
Swap: 0 0 0



=============================

EDIT: I gave up and installed Ubuntu 10.04 32-bit, everything seems to be working normally now. If the issue re-occurs I'll let you know, otherwise... thanks for all the help everyone