SABYenc found but still using python to repair?

[garyleecn]

I just installed&upgraded SABYenc for SABnzbd on my Synology (DSM 6.2 + sabnzbd 2.1 from synocommunity), I can confirm that SAB does find SABYenc
and I got this log

Code: Select all

2017-07-22 02:33:44,311::INFO::[SABnzbd:407] SABYenc module (v3.0.2)... found!

however, when I check the CPU usage during repair, SABnzbd is still using python. I can see 1 python process, and 3 par2 process during repair, each taking ~10-15% CPU to a total of 40-60%. I'm using a quad core xeon so 40-60% seems a little to high.

so, how do I know that SAB is using SABYenc instead of python? and how do I force SAB to use SABYenc?

many thanks

[safihre]

Sabyenc is used during the decoding of articles from usenet, this is before they get combined into the files and written to disk.
Then after all the files are written to disk we use par2 to verify if all files were correct. This is the Repair process and it's completely seperate from the downloading. It also requires lots of CPU power indeed to verify and repair files.

So if you get the Sabyenc message, it's used always.

[sander]

In addition to Safihre's comment, here are my two cents:

however, when I check the CPU usage during repair, SABnzbd is still using python.

Of course; SABnzbd is a python program. So as long as SAB is running, python is running.

I can see 1 python process, and 3 par2 process during repair, each taking ~10-15% CPU to a total of 40-60%. I'm using a quad core xeon so 40-60% seems a little to high.

Why is that too high? As long as there is work to do, CPU cycles will be used. And I think par2 would try to eat 100% CPU cycle if your HDD/SSD (with the par files on it) would have a speed of a million MB/s ... which your HDD/SSD hasn't, so therefor your CPU is not fully used.

Some more technical stuff: SAByenc is a library, which can be used by any program, so for example from SABnzbd/python.
It's a library, so you won't see it with "ps".
But to proof the library is used (apart from SAB's log line), on Linux/Unix/MacOS, you can do something like this:

With SABnzbd running, find the PID of SABnzbd:

Code: Select all

$ pgrep -i sabnzbd
25284

Use 'lsof' (list open files) of that process, and grep for sabyenc:

Code: Select all

$ lsof -p 25284 | grep sabyenc
SABnzbd.p 25284 sander  mem    REG              179,2    14192 1062045 /usr/lib/python2.7/dist-packages/sabyenc.x86_64-linux-gnu.so

So, there you have it: the sabyenc.x86_64-linux-gnu.so library. That is the proof that SABnzbd has the sabyenc library as open file.

FWIW: if I want to see all libraries files used by my running SAB with PID 25284:

Code: Select all

$ lsof -p 25284 | grep "\.so$"
SABnzbd.p 25284 sander  mem    REG              179,2    47608 1713582 /lib/x86_64-linux-gnu/libnss_files-2.24.so
SABnzbd.p 25284 sander  mem    REG              179,2    29392 1184920 /usr/lib/python2.7/lib-dynload/_multiprocessing.x86_64-linux-gnu.so
SABnzbd.p 25284 sander  mem    REG              179,2    31712 1713596 /lib/x86_64-linux-gnu/librt-2.24.so
SABnzbd.p 25284 sander  mem    REG              179,2   163128 1061795 /usr/lib/python2.7/dist-packages/_dbus_bindings.x86_64-linux-gnu.so
SABnzbd.p 25284 sander  mem    REG              179,2    46848 1184925 /usr/lib/python2.7/lib-dynload/bz2.x86_64-linux-gnu.so
SABnzbd.p 25284 sander  mem    REG              179,2    77408 1184935 /usr/lib/python2.7/lib-dynload/pyexpat.x86_64-linux-gnu.so
SABnzbd.p 25284 sander  mem    REG              179,2    14848 1061715 /usr/lib/python2.7/dist-packages/Cheetah/_namemapper.x86_64-linux-gnu.so
SABnzbd.p 25284 sander  mem    REG              179,2    14192 1062045 /usr/lib/python2.7/dist-packages/sabyenc.x86_64-linux-gnu.so
SABnzbd.p 25284 sander  mem    REG              179,2    14576 1061803 /usr/lib/python2.7/dist-packages/_yenc.x86_64-linux-gnu.so
SABnzbd.p 25284 sander  mem    REG              179,2   929168 1321061 /usr/lib/python2.7/dist-packages/cryptography/hazmat/bindings/_openssl.x86_64-linux-gnu.so
SABnzbd.p 25284 sander  mem    REG              179,2   160896 1061445 /usr/lib/python2.7/dist-packages/_cffi_backend.x86_64-linux-gnu.so
SABnzbd.p 25284 sander  mem    REG              179,2    10104 1321060 /usr/lib/python2.7/dist-packages/cryptography/hazmat/bindings/_constant_time.x86_64-linux-gnu.so
SABnzbd.p 25284 sander  mem    REG              179,2    96024 1184921 /usr/lib/python2.7/lib-dynload/_sqlite3.x86_64-linux-gnu.so
SABnzbd.p 25284 sander  mem    REG              179,2   148376 1184910 /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so
SABnzbd.p 25284 sander  mem    REG              179,2    70536 1184917 /usr/lib/python2.7/lib-dynload/_json.x86_64-linux-gnu.so
SABnzbd.p 25284 sander  mem    REG              179,2    29384 1184915 /usr/lib/python2.7/lib-dynload/_hashlib.x86_64-linux-gnu.so
SABnzbd.p 25284 sander  mem    REG              179,2   110728 1184922 /usr/lib/python2.7/lib-dynload/_ssl.x86_64-linux-gnu.so
SABnzbd.p 25284 sander  mem    REG              179,2  1088952 1713570 /lib/x86_64-linux-gnu/libm-2.24.so
SABnzbd.p 25284 sander  mem    REG              179,2    10656 1713598 /lib/x86_64-linux-gnu/libutil-2.24.so
SABnzbd.p 25284 sander  mem    REG              179,2    14608 1713568 /lib/x86_64-linux-gnu/libdl-2.24.so
SABnzbd.p 25284 sander  mem    REG              179,2  1856752 1713562 /lib/x86_64-linux-gnu/libc-2.24.so
SABnzbd.p 25284 sander  mem    REG              179,2   142400 1713592 /lib/x86_64-linux-gnu/libpthread-2.24.so
SABnzbd.p 25284 sander  mem    REG              179,2   162608 1710547 /lib/x86_64-linux-gnu/ld-2.24.so

So ... some nice technical details to enjoy ...

[sander]

As a test, I remove sabyenc from my system, and SABnzbd notices that:

Code: Select all

2017-07-22 16:07:44,281::WARNING::[SABnzbd:413] SABYenc module... NOT found! Expecting v3.0.2 - https://sabnzbd.org/sabyenc
2017-07-22 16:07:44,282::INFO::[SABnzbd:417] _yenc module... found!

Note that the old library " _yenc" is still found and used.

So, which library is used:

Code: Select all

sander@Stream-13:~$ pgrep -i sabnzbd
805

sander@Stream-13:~$ lsof -p 805 | grep -i yenc
SABnzbd.p 805 sander  mem    REG              179,2    14576 1061803 /usr/lib/python2.7/dist-packages/_yenc.x86_64-linux-gnu.so

Ah: "_yenc.x86_64-linux-gnu.so", which is the old _yenc. Not the new sabyenc.

QED