How to use Deobfuscate feature in 3.1.0Beta1 ?

Questions and bug reports for Beta releases should be posted here.
Forum rules
Help us help you:
  • Tell us what system you run SABnzbd on.
  • Adhere to the forum rules.
  • Do you experience problems during downloading?
    Check your connection in Status and Interface settings window.
    Use Test Server in Config > Servers.
    We will probably ask you to do a test using only basic settings.
  • Do you experience problems during repair or unpacking?
    Enable +Debug logging in the Status and Interface settings window and share the relevant parts of the log here using [ code ] sections.
sj50
Newbie
Newbie
Posts: 3
Joined: September 4th, 2020, 11:01 pm

How to use Deobfuscate feature in 3.1.0Beta1 ?

Post by sj50 »

I have installed 3.1.0Beta1 [f0d31e0] and enabled "Deobfuscate final filenames" but I still see obfuscate file name.
Is there anything else I have to do to make it work?
User avatar
sander
Release Testers
Release Testers
Posts: 8812
Joined: January 22nd, 2008, 2:22 pm

Re: How to use Deobfuscate feature in 3.1.0Beta1 ?

Post by sander »

Can you give examples of obfuscated file names that are still there?
sj50
Newbie
Newbie
Posts: 3
Joined: September 4th, 2020, 11:01 pm

Re: How to use Deobfuscate feature in 3.1.0Beta1 ?

Post by sj50 »

this is what I got from one of the nzb after unpack.
/sabnzbd/completed/Movies_UHD/Sunshine (dot) 2007 (dot) PROPER (dot) BluRay (dot) 1080p (dot) DTS-HD (dot) MA (dot) 5 (dot) 1 (dot) AVC (dot) REMUX-FraMeSToR /9y5gNEsQaxB6H (dot) mkv

edit - replace " (dot) " with "." as I am not allowed to post link
User avatar
sander
Release Testers
Release Testers
Posts: 8812
Joined: January 22nd, 2008, 2:22 pm

Re: How to use Deobfuscate feature in 3.1.0Beta1 ?

Post by sander »

So: 9y5gNEsQaxB6H.mkv is the file name.

You and I clearly see that that is obfuscated ... but how? Because there are no spaces and dots? What would you say about that? (see https://github.com/sabnzbd/sabnzbd/blob ... y#L99-L130 for the current criteria)

But let's what the function says:

Reference:

Code: Select all

sander@witte2004:~/git/sabnzbd$ python3 -c "from sabnzbd.deobfuscate_filenames import *; print(is_probably_obfuscated('599c1c9e2bdfb5114044bf25152b7eaa.mkv')) " 
True

Your filename:

Code: Select all

sander@witte2004:~/git/sabnzbd$ python3 -c "from sabnzbd.deobfuscate_filenames import *; print(is_probably_obfuscated('9y5gNEsQaxB6H.mkv')) " 
False
Let's turn on logging:

Code: Select all

sander@witte2004:~/git/sabnzbd$ python3 -c "import logging, sys; logging.basicConfig(stream=sys.stdout, level=logging.DEBUG); from sabnzbd.deobfuscate_filenames import * ; print(is_probably_obfuscated('9y5gNEsQaxB6H.mkv'))" 
DEBUG:root:Not obfuscated (default)
False
"Not obfuscated (default)" ... so not any trigger kicks in to say it's obfuscated

I find this one hard to detect. Capitals and small letters and numbers are a good sign: "My Movie 2020.mkv" and "MyMovie2020,mkv" are not obfuscated ... so how to detect your filename 9y5gNEsQaxB6H.mkv is obfuscated?
User avatar
sander
Release Testers
Release Testers
Posts: 8812
Joined: January 22nd, 2008, 2:22 pm

Re: How to use Deobfuscate feature in 3.1.0Beta1 ?

Post by sander »

Code: Select all

>>> filebasename = "9y5gNEsQaxB6H"

>>> sum(1 for c in filebasename if c.isupper())
5
>>> sum(1 for c in filebasename if c.islower())
5
>>> sum(1 for c in filebasename if c.isnumeric())
3
>>> sum(1 for c in filebasename if c == " " or c == ".")
0
>>> len(filebasename)
13
So ... no spaces/dots, and number of capitals is the same as number of small letters ... that is a sign of obfuscation ... ?

Or number of capitals and number of smaller letters and number of numerics are all about the same ... ?
sj50
Newbie
Newbie
Posts: 3
Joined: September 4th, 2020, 11:01 pm

Re: How to use Deobfuscate feature in 3.1.0Beta1 ?

Post by sj50 »

"If there is no "rename.par2" available, it will rename large, not-excluded files to the job-name in the queue if the filename looks obfuscated"
There is no rename.par2 it should rename it job name, But here script does not consider file name as obfuscated right?
May be add a comparing file name length with job name length as sign of obfuscation?
I have noticed text file which contains some random string, is this string key to deobfuscation ?
User avatar
sander
Release Testers
Release Testers
Posts: 8812
Joined: January 22nd, 2008, 2:22 pm

Re: How to use Deobfuscate feature in 3.1.0Beta1 ?

Post by sander »

Hi sj50,

Solved in the upcoming 3.1.0 Beta2 ... thanks to your report!!
Epedemic
Newbie
Newbie
Posts: 4
Joined: September 12th, 2020, 2:21 pm

Re: How to use Deobfuscate feature in 3.1.0Beta1 ?

Post by Epedemic »

I have a different problem with the deobfuscate function, which was not a problem using the old deobfuscate py :

When downloading a blu-ray, it will take some, but not all, of the numbered 000** *dot* m2ts files from the bdmv stream folder and move them to the root "job name" folder and rename them "job name *dot* 1 *dot* m2ts", 2 , 3 etc seemingly randomly numbered, so they are very difficult to restore unless you compare original names and sizes etc.... This is present in Beta2 as well.

Only the Rars are obfuscated, but the folders inside look good when i look at the source.

Sorry about the *dot* things, but as a new user i am not allowed to post links... and apparently the forum equates file names with dots to links. Hope the post is understandable anyway ;)
User avatar
sander
Release Testers
Release Testers
Posts: 8812
Joined: January 22nd, 2008, 2:22 pm

Re: How to use Deobfuscate feature in 3.1.0Beta1 ?

Post by sander »

Epedemic wrote: September 12th, 2020, 2:37 pm When downloading a blu-ray, it will take some, but not all, of the numbered 000** *dot* m2ts files from the bdmv stream folder and move them to the root "job name" folder and rename them "job name *dot* 1 *dot* m2ts", 2 , 3 etc seemingly randomly numbered, so they are very difficult to restore unless you compare original names and sizes etc.... This is present in Beta2 as well.
So ... those file have extension ".m2ts"? And one blu-ray download has multiple files with that extension?
It's easy for me to exclude them in the deobfuscate function. Would that be better, so not rename them at all?

If possible, can you share such a download with me (via PM and/or pastebin)?
User avatar
safihre
Administrator
Administrator
Posts: 5338
Joined: April 30th, 2015, 7:35 am
Contact:

Re: How to use Deobfuscate feature in 3.1.0Beta1 ?

Post by safihre »

Deobfuscate.py would only rename files if there was just 1 large file. If there were multiple, it wouldn't do anything. To indeed prevent this or prevent for example a job that has multiple episodes of a series inside to be rename all the same.
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
Epedemic
Newbie
Newbie
Posts: 4
Joined: September 12th, 2020, 2:21 pm

Re: How to use Deobfuscate feature in 3.1.0Beta1 ?

Post by Epedemic »

sander wrote: September 12th, 2020, 2:44 pm
So ... those file have extension ".m2ts"? And one blu-ray download has multiple files with that extension?
It's easy for me to exclude them in the deobfuscate function. Would that be better, so not rename them at all?

If possible, can you share such a download with me (via PM and/or pastebin)?
PM sent... I hope... Still a bit challenged about the "new user" thing, so i hope it is correctly sent even if it is in "outbox" and not in "sent messages". Otherwise let me know, and we'll figure out something else :)
User avatar
sander
Release Testers
Release Testers
Posts: 8812
Joined: January 22nd, 2008, 2:22 pm

Re: How to use Deobfuscate feature in 3.1.0Beta1 ?

Post by sander »

Hi Epedemic: I received your PM, and replied to it ... did you receive it?

I'm curious about the multiple file .m2ts ...
User avatar
sander
Release Testers
Release Testers
Posts: 8812
Joined: January 22nd, 2008, 2:22 pm

Re: How to use Deobfuscate feature in 3.1.0Beta1 ?

Post by sander »

OK, I downloaded it (65 GB download) with sabnzbdplus-3.1.0Beta2:

In between result (near the end but not yet finished, so before renaming):

Code: Select all

-rwxrwxrwx 1 root root 58156695552 sep 11 19:45 00000.m2ts
-rwxrwxrwx 1 root root  1162223616 sep 11 19:42 00001.m2ts
-rwxrwxrwx 1 root root     2850816 sep 11 19:31 00002.m2ts
-rwxrwxrwx 1 root root   200871936 sep 11 19:36 00003.m2ts
-rwxrwxrwx 1 root root  2231027712 sep 11 19:32 00004.m2ts
-rwxrwxrwx 1 root root   355411968 sep 11 19:17 00005.m2ts
-rwxrwxrwx 1 root root  1393446912 sep 11 19:21 00006.m2ts
-rwxrwxrwx 1 root root   145752064 sep 18 11:18 00007.m2ts
-rwxrwxrwx 1 root root       36864 sep 11 18:18 00012.m2ts
-rwxrwxrwx 1 root root    89751552 sep 11 18:18 00013.m2ts
-rwxrwxrwx 1 root root    17418240 sep 11 18:18 00014.m2ts
-rwxrwxrwx 1 root root   114726912 sep 11 18:18 00015.m2ts
-rwxrwxrwx 1 root root     2912256 sep 11 18:18 00016.m2ts
-rwxrwxrwx 1 root root  1097736192 sep 11 19:23 00017.m2ts
Finally:

Code: Select all

-rwxrwxrwx 1 root root 58156695552 sep 11 19:45 The.Something.1981-(USA-UHD).m2ts*
-rwxrwxrwx 1 root root  1162223616 sep 11 19:42 The.Something.1981-(USA-UHD).1.m2ts*
-rwxrwxrwx 1 root root   200871936 sep 11 19:36 The.Something.1981-(USA-UHD).2.m2ts*
-rwxrwxrwx 1 root root  2231027712 sep 11 19:32 The.Something.1981-(USA-UHD).3.m2ts*
-rwxrwxrwx 1 root root   355411968 sep 11 19:17 The.Something.1981-(USA-UHD).4.m2ts*
-rwxrwxrwx 1 root root  1393446912 sep 11 19:21 The.Something.1981-(USA-UHD).5.m2ts*
-rwxrwxrwx 1 root root   396656640 sep 11 18:18 The.Something.1981-(USA-UHD).6.m2ts*
-rwxrwxrwx 1 root root    89751552 sep 11 18:18 The.Something.1981-(USA-UHD).7.m2ts*
-rwxrwxrwx 1 root root    17418240 sep 11 18:18 The.Something.1981-(USA-UHD).8.m2ts*
-rwxrwxrwx 1 root root   114726912 sep 11 18:18 The.Something.1981-(USA-UHD).9.m2ts*
-rwxrwxrwx 1 root root  1097736192 sep 11 19:23 The.Something.1981-(USA-UHD).10.m2ts*


I do understand that result: only big files (>10MB) are renamed, and the renaming is OK ... files still in order. Question @epedemic:
1) is that OK for you,
2) or should SAB just exclude .m2ts extensions,
3) or rename with the long number in place (so: The.Something.1981-(USA-UHD).00001.m2ts


BUT ... completely missing are the smaller files, like the 00002.m2ts! That is bad. I have to find out how that can happen.
User avatar
sander
Release Testers
Release Testers
Posts: 8812
Joined: January 22nd, 2008, 2:22 pm

Re: How to use Deobfuscate feature in 3.1.0Beta1 ?

Post by sander »

Based on your report, I found a bug in 3.1.0 Beta2 (causing the disappearing file), which was already solved in the git version, so in 3.1.0 RC1

And in the git version, I now excluded the .m2ts (and other BR) files: no more renaming. Also in the upcoming 3.1.0 RC1.
Epedemic
Newbie
Newbie
Posts: 4
Joined: September 12th, 2020, 2:21 pm

Re: How to use Deobfuscate feature in 3.1.0Beta1 ?

Post by Epedemic »

sander wrote: September 18th, 2020, 4:54 am I do understand that result: only big files (>10MB) are renamed, and the renaming is OK ... files still in order. Question @epedemic:
1) is that OK for you,
2) or should SAB just exclude .m2ts extensions,
3) or rename with the long number in place (so: The.Something.1981-(USA-UHD).00001.m2ts


BUT ... completely missing are the smaller files, like the 00002.m2ts! That is bad. I have to find out how that can happen.
I did some testing with several downloads on RC1, and can confirm everything looks good now. Thanks!

However, that seems to be in conflict with what you wrote. So to sum it up: m2ts files in a blu-ray/UHD blu-ray file structure (in bdmv\stream should NEVER be renamed). I can think of situations though where a m2ts should be renamed (eg in a remux to a single m2ts instead of mkv, but that is another case, and not really anything i dabble in)
Post Reply