Page 1 of 2

How to use Deobfuscate feature in 3.1.0Beta1 ?

Posted: September 4th, 2020, 11:09 pm
by sj50
I have installed 3.1.0Beta1 [f0d31e0] and enabled "Deobfuscate final filenames" but I still see obfuscate file name.
Is there anything else I have to do to make it work?

Re: How to use Deobfuscate feature in 3.1.0Beta1 ?

Posted: September 5th, 2020, 12:54 am
by sander
Can you give examples of obfuscated file names that are still there?

Re: How to use Deobfuscate feature in 3.1.0Beta1 ?

Posted: September 5th, 2020, 2:04 am
by sj50
this is what I got from one of the nzb after unpack.
/sabnzbd/completed/Movies_UHD/Sunshine (dot) 2007 (dot) PROPER (dot) BluRay (dot) 1080p (dot) DTS-HD (dot) MA (dot) 5 (dot) 1 (dot) AVC (dot) REMUX-FraMeSToR /9y5gNEsQaxB6H (dot) mkv

edit - replace " (dot) " with "." as I am not allowed to post link

Re: How to use Deobfuscate feature in 3.1.0Beta1 ?

Posted: September 5th, 2020, 1:16 pm
by sander
So: 9y5gNEsQaxB6H.mkv is the file name.

You and I clearly see that that is obfuscated ... but how? Because there are no spaces and dots? What would you say about that? (see https://github.com/sabnzbd/sabnzbd/blob ... y#L99-L130 for the current criteria)

But let's what the function says:

Reference:

Code: Select all

[email protected]:~/git/sabnzbd$ python3 -c "from sabnzbd.deobfuscate_filenames import *; print(is_probably_obfuscated('599c1c9e2bdfb5114044bf25152b7eaa.mkv')) " 
True

Your filename:

Code: Select all

[email protected]:~/git/sabnzbd$ python3 -c "from sabnzbd.deobfuscate_filenames import *; print(is_probably_obfuscated('9y5gNEsQaxB6H.mkv')) " 
False
Let's turn on logging:

Code: Select all

[email protected]:~/git/sabnzbd$ python3 -c "import logging, sys; logging.basicConfig(stream=sys.stdout, level=logging.DEBUG); from sabnzbd.deobfuscate_filenames import * ; print(is_probably_obfuscated('9y5gNEsQaxB6H.mkv'))" 
DEBUG:root:Not obfuscated (default)
False
"Not obfuscated (default)" ... so not any trigger kicks in to say it's obfuscated

I find this one hard to detect. Capitals and small letters and numbers are a good sign: "My Movie 2020.mkv" and "MyMovie2020,mkv" are not obfuscated ... so how to detect your filename 9y5gNEsQaxB6H.mkv is obfuscated?

Re: How to use Deobfuscate feature in 3.1.0Beta1 ?

Posted: September 5th, 2020, 1:45 pm
by sander

Code: Select all

>>> filebasename = "9y5gNEsQaxB6H"

>>> sum(1 for c in filebasename if c.isupper())
5
>>> sum(1 for c in filebasename if c.islower())
5
>>> sum(1 for c in filebasename if c.isnumeric())
3
>>> sum(1 for c in filebasename if c == " " or c == ".")
0
>>> len(filebasename)
13
So ... no spaces/dots, and number of capitals is the same as number of small letters ... that is a sign of obfuscation ... ?

Or number of capitals and number of smaller letters and number of numerics are all about the same ... ?

Re: How to use Deobfuscate feature in 3.1.0Beta1 ?

Posted: September 5th, 2020, 9:22 pm
by sj50
"If there is no "rename.par2" available, it will rename large, not-excluded files to the job-name in the queue if the filename looks obfuscated"
There is no rename.par2 it should rename it job name, But here script does not consider file name as obfuscated right?
May be add a comparing file name length with job name length as sign of obfuscation?
I have noticed text file which contains some random string, is this string key to deobfuscation ?

Re: How to use Deobfuscate feature in 3.1.0Beta1 ?

Posted: September 11th, 2020, 2:46 pm
by sander
Hi sj50,

Solved in the upcoming 3.1.0 Beta2 ... thanks to your report!!

Re: How to use Deobfuscate feature in 3.1.0Beta1 ?

Posted: September 12th, 2020, 2:37 pm
by Epedemic
I have a different problem with the deobfuscate function, which was not a problem using the old deobfuscate py :

When downloading a blu-ray, it will take some, but not all, of the numbered 000** *dot* m2ts files from the bdmv stream folder and move them to the root "job name" folder and rename them "job name *dot* 1 *dot* m2ts", 2 , 3 etc seemingly randomly numbered, so they are very difficult to restore unless you compare original names and sizes etc.... This is present in Beta2 as well.

Only the Rars are obfuscated, but the folders inside look good when i look at the source.

Sorry about the *dot* things, but as a new user i am not allowed to post links... and apparently the forum equates file names with dots to links. Hope the post is understandable anyway ;)

Re: How to use Deobfuscate feature in 3.1.0Beta1 ?

Posted: September 12th, 2020, 2:44 pm
by sander
Epedemic wrote:
September 12th, 2020, 2:37 pm
When downloading a blu-ray, it will take some, but not all, of the numbered 000** *dot* m2ts files from the bdmv stream folder and move them to the root "job name" folder and rename them "job name *dot* 1 *dot* m2ts", 2 , 3 etc seemingly randomly numbered, so they are very difficult to restore unless you compare original names and sizes etc.... This is present in Beta2 as well.
So ... those file have extension ".m2ts"? And one blu-ray download has multiple files with that extension?
It's easy for me to exclude them in the deobfuscate function. Would that be better, so not rename them at all?

If possible, can you share such a download with me (via PM and/or pastebin)?

Re: How to use Deobfuscate feature in 3.1.0Beta1 ?

Posted: September 12th, 2020, 3:15 pm
by safihre
Deobfuscate.py would only rename files if there was just 1 large file. If there were multiple, it wouldn't do anything. To indeed prevent this or prevent for example a job that has multiple episodes of a series inside to be rename all the same.

Re: How to use Deobfuscate feature in 3.1.0Beta1 ?

Posted: September 12th, 2020, 4:11 pm
by Epedemic
sander wrote:
September 12th, 2020, 2:44 pm

So ... those file have extension ".m2ts"? And one blu-ray download has multiple files with that extension?
It's easy for me to exclude them in the deobfuscate function. Would that be better, so not rename them at all?

If possible, can you share such a download with me (via PM and/or pastebin)?
PM sent... I hope... Still a bit challenged about the "new user" thing, so i hope it is correctly sent even if it is in "outbox" and not in "sent messages". Otherwise let me know, and we'll figure out something else :)

Re: How to use Deobfuscate feature in 3.1.0Beta1 ?

Posted: September 13th, 2020, 12:55 pm
by sander
Hi Epedemic: I received your PM, and replied to it ... did you receive it?

I'm curious about the multiple file .m2ts ...

Re: How to use Deobfuscate feature in 3.1.0Beta1 ?

Posted: September 18th, 2020, 4:54 am
by sander
OK, I downloaded it (65 GB download) with sabnzbdplus-3.1.0Beta2:

In between result (near the end but not yet finished, so before renaming):

Code: Select all

-rwxrwxrwx 1 root root 58156695552 sep 11 19:45 00000.m2ts
-rwxrwxrwx 1 root root  1162223616 sep 11 19:42 00001.m2ts
-rwxrwxrwx 1 root root     2850816 sep 11 19:31 00002.m2ts
-rwxrwxrwx 1 root root   200871936 sep 11 19:36 00003.m2ts
-rwxrwxrwx 1 root root  2231027712 sep 11 19:32 00004.m2ts
-rwxrwxrwx 1 root root   355411968 sep 11 19:17 00005.m2ts
-rwxrwxrwx 1 root root  1393446912 sep 11 19:21 00006.m2ts
-rwxrwxrwx 1 root root   145752064 sep 18 11:18 00007.m2ts
-rwxrwxrwx 1 root root       36864 sep 11 18:18 00012.m2ts
-rwxrwxrwx 1 root root    89751552 sep 11 18:18 00013.m2ts
-rwxrwxrwx 1 root root    17418240 sep 11 18:18 00014.m2ts
-rwxrwxrwx 1 root root   114726912 sep 11 18:18 00015.m2ts
-rwxrwxrwx 1 root root     2912256 sep 11 18:18 00016.m2ts
-rwxrwxrwx 1 root root  1097736192 sep 11 19:23 00017.m2ts
Finally:

Code: Select all

-rwxrwxrwx 1 root root 58156695552 sep 11 19:45 The.Something.1981-(USA-UHD).m2ts*
-rwxrwxrwx 1 root root  1162223616 sep 11 19:42 The.Something.1981-(USA-UHD).1.m2ts*
-rwxrwxrwx 1 root root   200871936 sep 11 19:36 The.Something.1981-(USA-UHD).2.m2ts*
-rwxrwxrwx 1 root root  2231027712 sep 11 19:32 The.Something.1981-(USA-UHD).3.m2ts*
-rwxrwxrwx 1 root root   355411968 sep 11 19:17 The.Something.1981-(USA-UHD).4.m2ts*
-rwxrwxrwx 1 root root  1393446912 sep 11 19:21 The.Something.1981-(USA-UHD).5.m2ts*
-rwxrwxrwx 1 root root   396656640 sep 11 18:18 The.Something.1981-(USA-UHD).6.m2ts*
-rwxrwxrwx 1 root root    89751552 sep 11 18:18 The.Something.1981-(USA-UHD).7.m2ts*
-rwxrwxrwx 1 root root    17418240 sep 11 18:18 The.Something.1981-(USA-UHD).8.m2ts*
-rwxrwxrwx 1 root root   114726912 sep 11 18:18 The.Something.1981-(USA-UHD).9.m2ts*
-rwxrwxrwx 1 root root  1097736192 sep 11 19:23 The.Something.1981-(USA-UHD).10.m2ts*


I do understand that result: only big files (>10MB) are renamed, and the renaming is OK ... files still in order. Question @epedemic:
1) is that OK for you,
2) or should SAB just exclude .m2ts extensions,
3) or rename with the long number in place (so: The.Something.1981-(USA-UHD).00001.m2ts


BUT ... completely missing are the smaller files, like the 00002.m2ts! That is bad. I have to find out how that can happen.

Re: How to use Deobfuscate feature in 3.1.0Beta1 ?

Posted: September 18th, 2020, 8:37 am
by sander
Based on your report, I found a bug in 3.1.0 Beta2 (causing the disappearing file), which was already solved in the git version, so in 3.1.0 RC1

And in the git version, I now excluded the .m2ts (and other BR) files: no more renaming. Also in the upcoming 3.1.0 RC1.

Re: How to use Deobfuscate feature in 3.1.0Beta1 ?

Posted: September 19th, 2020, 2:41 pm
by Epedemic
sander wrote:
September 18th, 2020, 4:54 am
I do understand that result: only big files (>10MB) are renamed, and the renaming is OK ... files still in order. Question @epedemic:
1) is that OK for you,
2) or should SAB just exclude .m2ts extensions,
3) or rename with the long number in place (so: The.Something.1981-(USA-UHD).00001.m2ts


BUT ... completely missing are the smaller files, like the 00002.m2ts! That is bad. I have to find out how that can happen.
I did some testing with several downloads on RC1, and can confirm everything looks good now. Thanks!

However, that seems to be in conflict with what you wrote. So to sum it up: m2ts files in a blu-ray/UHD blu-ray file structure (in bdmv\stream should NEVER be renamed). I can think of situations though where a m2ts should be renamed (eg in a remux to a single m2ts instead of mkv, but that is another case, and not really anything i dabble in)