I put in my own monitoring script to check what is going on. It finally caught the offending process which turned out to be sabnzb realated. Here are my run stats before no more output was generating due to my server being toast. You can see the CPU load build up in the very last run.
Anyone have a clue what may be going on here, choking on a rar file?
Code: Select all
Fri Dec 4 01:40:01 CST 2015
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
nobody 28851 7.6 4.0 2283148 207248 ? Sl Nov28 610:47 | \_ /usr/bin/python /usr/bin/sabnzbdplus --config-file /config
nobody 29581 4.1 9.6 2453300 492780 ? Sl Nov28 335:31 | \_ /usr/sbin/python2 /opt/sickrage/SickBeard.py --config /config/config.ini --datadir /config
nobody 29276 3.2 1.1 549648 57512 ? Sl Nov28 260:06 | \_ /opt/plexmediaserver/Plex Media Server
nobody 29539 2.4 3.4 1613068 173256 ? Sl Nov28 197:36 | | \_ /usr/bin/ghb
root 28410 1.5 0.2 1104412 10196 ? Ssl Nov28 124:54 /usr/local/sbin/shfs /mnt/user -disks 15 2048000000 -o noatime,big_writes,allow_other -o remember=330
root 28253 1.3 0.0 89100 3468 ? Sl Nov28 111:20 /usr/local/sbin/emhttp
nobody 30206 0.6 1.9 1786388 97108 ? Sl Nov28 53:17 | \_ /usr/bin/python /opt/plexpy/PlexPy.py --datadir /config
root 1720 0.4 0.3 293828 17608 ? S Nov28 32:00 \_ /usr/sbin/smbd -D
root 28337 0.3 0.0 0 0 ? S Nov28 26:20 \_ [unraidd]
nobody 19193 0.2 0.8 884588 42004 ? Sl 01:24 0:02 | \_ Plex Plug-in [com.plexapp.agents.localmedia] /opt/plexmediaserver/Resources/Plug-ins-a10e3c2/Framework.bundle/Contents/Resources/Versions/2/Python/bootstrap.pyc --server-version 0.9.14.4.1556-a10e3c2 /opt/plexmediaserver/Resources/Plug-ins-a10e3c2/LocalMedia.bundle
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 0 0 35288 151488 2255844 0 0 685 473 6 11 12 2 84 3
total used free shared buffers cached
Mem: 4965 4930 34 0 147 2202
-/+ buffers/cache: 2579 2385
Swap: 0 0 0
Code: Select all
APP=COMMAND: PID=PID;Memory%=%MEM;VirtualMemory=VSZ;RealMemoryInRam=RSS
APP=/config: PID=29581;Memory%=9.6;VirtualMemory=2453300;RealMemoryInRam=492780
APP=/config: PID=28851;Memory%=4.0;VirtualMemory=2283148;RealMemoryInRam=207248
APP=start: PID=29747;Memory%=4.0;VirtualMemory=2180116;RealMemoryInRam=205476
APP=start: PID=29329;Memory%=3.7;VirtualMemory=2583612;RealMemoryInRam=189352
APP=/config: PID=29861;Memory%=3.6;VirtualMemory=2275232;RealMemoryInRam=187136
APP=start: PID=29027;Memory%=3.6;VirtualMemory=2180116;RealMemoryInRam=186704
APP=/usr/bin/ghb: PID=29539;Memory%=3.4;VirtualMemory=1613068;RealMemoryInRam=173256
APP=/opt/plexmediaserver/Resources/Plug-ins-a10e3c2/System.bundle: PID=29682;Memory%=2.6;VirtualMemory=1778156;RealMemoryInRam=133992
APP=/config: PID=30206;Memory%=1.9;VirtualMemory=1786388;RealMemoryInRam=97108
APP=Server: PID=29276;Memory%=1.1;VirtualMemory=549648;RealMemoryInRam=57512
APP=/opt/plexmediaserver/Resources/Plug-ins-a10e3c2/LocalMedia.bundle: PID=19193;Memory%=0.8;VirtualMemory=884588;RealMemoryInRam=42004
APP=/opt/plexmediaserver/Resources/Plug-ins-a10e3c2/TheTVDB.bundle: PID=19155;Memory%=0.8;VirtualMemory=1025100;RealMemoryInRam=40792
APP=server.js: PID=30287;Memory%=0.6;VirtualMemory=695140;RealMemoryInRam=32732
APP=96: PID=29320;Memory%=0.4;VirtualMemory=83084;RealMemoryInRam=23252
APP=--storage-driver=btrfs: PID=28556;Memory%=0.4;VirtualMemory=1449768;RealMemoryInRam=22636
APP=/opt/yify-pop: PID=29847;Memory%=0.3;VirtualMemory=675252;RealMemoryInRam=19412
APP=-D: PID=1720;Memory%=0.3;VirtualMemory=293828;RealMemoryInRam=17608
APP=/usr/local/emhttp/plugins/dynamix/scripts/localmaster: PID=22015;Memory%=0.2;VirtualMemory=105268;RealMemoryInRam=15160
APP=8889: PID=29268;Memory%=0.2;VirtualMemory=108344;RealMemoryInRam=14904
Code: Select all
Fri Dec 4 02:00:01 CST 2015
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
[b]nobody 24392 200 0.1 234848 7648 ? Rl 01:53 13:13 | \_ /usr/local/bin/par2 r /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e01-1080p-proof.par2 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e01-1080p.par2 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e02-1080p-proof.par2 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r00 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r01 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r02 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r03 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r04 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r05 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r06 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r07 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r08 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r09 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r10 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r11 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r12 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r13 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r14 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r15 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r16 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r17 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r18 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r19 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r20 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r21 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r22 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r23 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r24 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r25 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r26 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r27 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r28 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r29 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r30 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r31 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r32 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r33 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-FLHD/flhd-outlanders01e05-1080p.r34 /downloads/Download/incomplete/Outlander.S01.Vol.1.1080p.BluRay.x264-[/b]
nobody 28851 7.6 4.1 2282636 212928 ? Sl Nov28 611:53 | \_ /usr/bin/python /usr/bin/sabnzbdplus --config-file /config
nobody 29581 4.2 9.6 2453300 492780 ? Sl Nov28 337:00 | \_ /usr/sbin/python2 /opt/sickrage/SickBeard.py --config /config/config.ini --datadir /config
nobody 29276 3.2 1.1 547596 58936 ? Sl Nov28 261:20 | \_ /opt/plexmediaserver/Plex Media Server
nobody 29539 2.4 3.3 1613068 171580 ? Sl Nov28 197:49 | | \_ /usr/bin/ghb
root 28410 1.5 0.2 1104496 10528 ? Ssl Nov28 125:50 /usr/local/sbin/shfs /mnt/user -disks 15 2048000000 -o noatime,big_writes,allow_other -o remember=330
root 28253 1.3 0.0 89100 3468 ? Sl Nov28 111:21 /usr/local/sbin/emhttp
nobody 30206 0.6 1.8 1786388 93172 ? Sl Nov28 53:32 | \_ /usr/bin/python /opt/plexpy/PlexPy.py --datadir /config
nobody 1720 0.4 0.3 293828 17608 ? R Nov28 32:03 \_ /usr/sbin/smbd -D
root 28337 0.3 0.0 0 0 ? S Nov28 26:35 \_ [unraidd]
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
7 2 0 55976 104160 2284184 0 0 701 473 8 14 12 2 84 3
Code: Select all
total used free shared buffers cached
Mem: 4965 4910 54 0 101 2230
-/+ buffers/cache: 2578 2386
Swap: 0 0 0
APP=COMMAND: PID=PID;Memory%=%MEM;VirtualMemory=VSZ;RealMemoryInRam=RSS
APP=/config: PID=29581;Memory%=9.6;VirtualMemory=2453300;RealMemoryInRam=492780
APP=/config: PID=28851;Memory%=4.1;VirtualMemory=2282636;RealMemoryInRam=212928
APP=start: PID=29747;Memory%=4.0;VirtualMemory=2180116;RealMemoryInRam=205476
APP=start: PID=29329;Memory%=3.7;VirtualMemory=2583612;RealMemoryInRam=189352
APP=start: PID=29027;Memory%=3.6;VirtualMemory=2180116;RealMemoryInRam=186704
APP=/config: PID=29861;Memory%=3.6;VirtualMemory=2275232;RealMemoryInRam=185812
APP=/usr/bin/ghb: PID=29539;Memory%=3.3;VirtualMemory=1613068;RealMemoryInRam=171580
APP=/opt/plexmediaserver/Resources/Plug-ins-a10e3c2/System.bundle: PID=29682;Memory%=2.5;VirtualMemory=1778156;RealMemoryInRam=130168
APP=/config: PID=30206;Memory%=1.8;VirtualMemory=1786388;RealMemoryInRam=93172
APP=Server: PID=29276;Memory%=1.1;VirtualMemory=547596;RealMemoryInRam=58936
APP=server.js: PID=30287;Memory%=0.6;VirtualMemory=695140;RealMemoryInRam=32732
APP=/opt/plexmediaserver/Resources/Plug-ins-a10e3c2/LocalMedia.bundle: PID=19193;Memory%=0.6;VirtualMemory=884588;RealMemoryInRam=32616
APP=/opt/plexmediaserver/Resources/Plug-ins-a10e3c2/TheTVDB.bundle: PID=19155;Memory%=0.6;VirtualMemory=1092688;RealMemoryInRam=32172
APP=/opt/plexmediaserver/Resources/Plug-ins-a10e3c2/PlexThemeMusic.bundle: PID=24482;Memory%=0.4;VirtualMemory=949988;RealMemoryInRam=24088
APP=96: PID=29320;Memory%=0.4;VirtualMemory=83084;RealMemoryInRam=22752
APP=--storage-driver=btrfs: PID=28556;Memory%=0.4;VirtualMemory=1449768;RealMemoryInRam=22636
APP=/opt/yify-pop: PID=29847;Memory%=0.3;VirtualMemory=675252;RealMemoryInRam=19412
APP=-D: PID=1720;Memory%=0.3;VirtualMemory=293828;RealMemoryInRam=17608
APP=/usr/local/emhttp/plugins/dynamix/scripts/localmaster: PID=25860;Memory%=0.2;VirtualMemory=105268;RealMemoryInRam=15168