Obfuscated files without PARs

[trinitytek]

Is there any work being done to implement de-obfuscation for files without pars? If you are interested in adding this to your app, please contact me, I do this using powershell, it is pretty simple.

[sander]

Can you post the powershell script, and a link to such a NZB?

[trinitytek]

I am sure you can find one with obfuscated files, but here is the script:

$FileList = Get-ChildItem -Path X:\Downloads\Folder

foreach ($File in $FileList)
{
$test = C:\WinRAR\rar l $file.FullName
$option = [System.StringSplitOptions]::RemoveEmptyEntries
$newvar = ($test[11].split(' ',$option)).foreach{$_}
$number = $newvar[2].PadLeft(4,'0')
$newname = $file.DirectoryName + '\' + 'recovered.' + 'part' + $number + '.rar'

ren $file.FullName $newname
}

[trinitytek]

one caveat, run the files through a renamer first or do it via powershell... the files will need an RAR extension for this script to function

[safihre]

There is a way to indeed extract this from the header of the rar-files.
Maybe you have some coding skills and can help us out with this? I have some basic documentation!

[sander]

Some questions:

- can you put / name the rar files in random order?
- what if there is more than 1 file? The "rar l ..." will list more than 1 file, and will pick ... which name? See example below. Does your script work then (I can only read & understand vaguely the powershell commands)?

Code: Select all

$ rar l 69a04687-a31d-42c0-80d0-06ceeb074336 

RAR 5.50   Copyright (c) 1993-2017 Alexander Roshal   11 Aug 2017
Trial version             Type 'rar -?' for help

Archive: 69a04687-a31d-42c0-80d0-06ceeb074336
Details: RAR 5, volume 1

 Attributes      Size     Date    Time   Name
----------- ---------  ---------- -----  ----
 -rw-r--r--   1048576  2019-07-26 15:33  Anothername.bin
 -rw-r--r-- 104857600  2019-07-26 15:32  Thisissomename.bin
----------- ---------  ---------- -----  ----
            105906176                    2

[sander]

Ah: the order / index is in the "Details: RAR 5, volume 2" ?

Code: Select all

[email protected]:~/post_this/my_obfuscation/bla$ rar l 3b21332d-f6f7-4d67-9b84-1b63e37d7e11

RAR 5.50   Copyright (c) 1993-2017 Alexander Roshal   11 Aug 2017
Trial version             Type 'rar -?' for help

Archive: 3b21332d-f6f7-4d67-9b84-1b63e37d7e11
Details: RAR 5, volume 5

 Attributes      Size     Date    Time   Name
----------- ---------  ---------- -----  ----
 -rw-r--r-- 104857600  2019-07-26 15:32  Thisissomename.bin
----------- ---------  ---------- -----  ----
                    0                    0

[email protected]:~/post_this/my_obfuscation/bla$ rar l f1530e06-cd88-47fe-b500-a6cb36636003

RAR 5.50   Copyright (c) 1993-2017 Alexander Roshal   11 Aug 2017
Trial version             Type 'rar -?' for help

Archive: f1530e06-cd88-47fe-b500-a6cb36636003
Details: RAR 5, volume 2

 Attributes      Size     Date    Time   Name
----------- ---------  ---------- -----  ----
 -rw-r--r-- 104857600  2019-07-26 15:32  Thisissomename.bin
----------- ---------  ---------- -----  ----
                    0                    0

Let's try:

Code: Select all

$ for f in *; do echo "Processing $f file.." ; rar l $f | grep Details ; done
Processing 3b21332d-f6f7-4d67-9b84-1b63e37d7e11 file..
Details: RAR 5, volume 5
Processing 448e77e7-f685-42a2-81a6-1581c3d37979 file..
Details: RAR 5, volume 3
Processing 69a04687-a31d-42c0-80d0-06ceeb074336 file..
Details: RAR 5, volume 1
Processing 9bf8f3c6-672a-41e6-ae86-073afc9c3cb0 file..
Details: RAR 5, volume 4
Processing f1530e06-cd88-47fe-b500-a6cb36636003 file..
Details: RAR 5, volume 2

Ah, nice

[sander]

OK, bingo:

Start: fully obfuscated:

Code: Select all

for f in *.rar; do echo "Processing $f file.."; mv $f `uuidgen` ; done

Code: Select all

[email protected]:~/post_this/my_obfuscation/bla$ ls -al
total 103436
drwxr-xr-x 2 sander sander     4096 jul 26 15:48 .
drwxr-xr-x 3 sander sander     4096 jul 26 15:40 ..
-rw-r--r-- 1 sander sander 13632578 jul 26 15:48 3b21332d-f6f7-4d67-9b84-1b63e37d7e11
-rw-r--r-- 1 sander sander 23068672 jul 26 15:48 448e77e7-f685-42a2-81a6-1581c3d37979
-rw-r--r-- 1 sander sander 23068672 jul 26 15:48 69a04687-a31d-42c0-80d0-06ceeb074336
-rw-r--r-- 1 sander sander 23068672 jul 26 15:48 9bf8f3c6-672a-41e6-ae86-073afc9c3cb0
-rw-r--r-- 1 sander sander 23068672 jul 26 15:48 f1530e06-cd88-47fe-b500-a6cb36636003

Rename with correct order number (but random file name):

Code: Select all

$ for f in *; do echo "Processing $f file.." ; ordernumber=`rar l $f | grep Details | awk '{ print $NF }'` ; echo $ordernumber ; mv $f blabla.part$ordernumber.rar ; done
Processing 3b21332d-f6f7-4d67-9b84-1b63e37d7e11 file..
5
Processing 448e77e7-f685-42a2-81a6-1581c3d37979 file..
3
Processing 69a04687-a31d-42c0-80d0-06ceeb074336 file..
1
Processing 9bf8f3c6-672a-41e6-ae86-073afc9c3cb0 file..
4
Processing f1530e06-cd88-47fe-b500-a6cb36636003 file..
2

Nice result so far:

Code: Select all

[email protected]:~/post_this/my_obfuscation/bla$ ll
total 103436
drwxr-xr-x 2 sander sander     4096 jul 26 15:53 ./
drwxr-xr-x 3 sander sander     4096 jul 26 15:40 ../
-rw-r--r-- 1 sander sander 23068672 jul 26 15:48 blabla.part1.rar
-rw-r--r-- 1 sander sander 23068672 jul 26 15:48 blabla.part2.rar
-rw-r--r-- 1 sander sander 23068672 jul 26 15:48 blabla.part3.rar
-rw-r--r-- 1 sander sander 23068672 jul 26 15:48 blabla.part4.rar
-rw-r--r-- 1 sander sander 13632578 jul 26 15:48 blabla.part5.rar

Let unrar work on it:

Code: Select all

[email protected]:~/post_this/my_obfuscation/bla$ unrar x *part1.rar

UNRAR 5.71 freeware      Copyright (c) 1993-2019 Alexander Roshal


Extracting from blabla.part1.rar

Extracting  Anothername.bin                                           OK 
Extracting  Thisissomename.bin                                        21%

Extracting from blabla.part2.rar

...         Thisissomename.bin                                        43%

Extracting from blabla.part3.rar

...         Thisissomename.bin                                        65%

Extracting from blabla.part4.rar

...         Thisissomename.bin                                        87%

Extracting from blabla.part5.rar

...         Thisissomename.bin                                        OK 
All OK

Check:

Code: Select all

[email protected]:~/post_this/my_obfuscation/bla$ ll
total 206860
drwxr-xr-x 2 sander sander      4096 jul 26 15:54 ./
drwxr-xr-x 3 sander sander      4096 jul 26 15:40 ../
-rw-r--r-- 1 sander sander   1048576 jul 26 15:33 Anothername.bin
-rw-r--r-- 1 sander sander  23068672 jul 26 15:48 blabla.part1.rar
-rw-r--r-- 1 sander sander  23068672 jul 26 15:48 blabla.part2.rar
-rw-r--r-- 1 sander sander  23068672 jul 26 15:48 blabla.part3.rar
-rw-r--r-- 1 sander sander  23068672 jul 26 15:48 blabla.part4.rar
-rw-r--r-- 1 sander sander  13632578 jul 26 15:48 blabla.part5.rar
-rw-r--r-- 1 sander sander 104857600 jul 26 15:32 Thisissomename.bin
[email protected]:~/post_this/my_obfuscation/bla$ 

Done!

To do: checking the file is really a rar file before renaming.

[sander]

Python script is working

Code: Select all

$ python deobfuscate.py 
deobfuscate.py
Skipping (as not a rar file): deobfuscate.py
3b21332d-f6f7-4d67-9b84-1b63e37d7e11
Renaming 3b21332d-f6f7-4d67-9b84-1b63e37d7e11 to blablabla.part5.rar
notarar-blablabla
Skipping (as not a rar file): notarar-blablabla
448e77e7-f685-42a2-81a6-1581c3d37979
Renaming 448e77e7-f685-42a2-81a6-1581c3d37979 to blablabla.part3.rar
f1530e06-cd88-47fe-b500-a6cb36636003
Renaming f1530e06-cd88-47fe-b500-a6cb36636003 to blablabla.part2.rar
9bf8f3c6-672a-41e6-ae86-073afc9c3cb0
Renaming 9bf8f3c6-672a-41e6-ae86-073afc9c3cb0 to blablabla.part4.rar
69a04687-a31d-42c0-80d0-06ceeb074336
Renaming 69a04687-a31d-42c0-80d0-06ceeb074336 to blablabla.part1.rar

Shypike ... now what? Where/how to put this into SABnzbd?


Edit
reference NZB:
https://raw.githubusercontent.com/sande ... 9c3cb0.nzb

Current SAB does some strange renaming :

Code: Select all

2019-07-26 17:08:42,163::INFO::[postproc:376] Finished unpack_magic on blabla123
2019-07-26 17:08:42,164::DEBUG::[filesystem:625] Moving (overwrite: 0) /home/sander/Downloads/incomplete/blabla123/reftestnzb obfuscated rar files no par 073afc9c3cb0 [2+5] - "448e77e7-f685-42a2-81a6-1581c3d37979" yEnc (1+33) 23068672 => /home/sander/Downloads/complete/_UNPACK_blabla123/reftestnzb obfuscated rar files no par 073afc9c3cb0 [2+5] - "448e77e7-f685-42a2-81a6-1581c3d37979" yEnc (1+33) 23068672
2019-07-26 17:08:42,165::DEBUG::[filesystem:718] Renaming "/home/sander/Downloads/incomplete/blabla123/reftestnzb obfuscated rar files no par 073afc9c3cb0 [2+5] - "448e77e7-f685-42a2-81a6-1581c3d37979" yEnc (1+33) 23068672" to "/home/sander/Downloads/complete/_UNPACK_blabla123/reftestnzb obfuscated rar files no par 073afc9c3cb0 [2+5] - "448e77e7-f685-42a2-81a6-1581c3d37979" yEnc (1+33) 23068672"
2019-07-26 17:08:42,166::DEBUG::[filesystem:625] Moving (overwrite: 0) /home/sander/Downloads/incomplete/blabla123/reftestnzb obfuscated rar files no par 073afc9c3cb0 [1+5] - "3b21332d-f6f7-4d67-9b84-1b63e37d7e11" yEnc (1+20) 13632578 => /home/sander/Downloads/complete/_UNPACK_blabla123/reftestnzb obfuscated rar files no par 073afc9c3cb0 [1+5] - "3b21332d-f6f7-4d67-9b84-1b63e37d7e11" yEnc (1+20) 13632578
2019-07-26 17:08:42,167::DEBUG::[filesystem:718] Renaming "/home/sander/Downloads/incomplete/blabla123/reftestnzb obfuscated rar files no par 073afc9c3cb0 [1+5] - "3b21332d-f6f7-4d67-9b84-1b63e37d7e11" yEnc (1+20) 13632578" to "/home/sander/Downloads/complete/_UNPACK_blabla123/reftestnzb obfuscated rar files no par 073afc9c3cb0 [1+5] - "3b21332d-f6f7-4d67-9b84-1b63e37d7e11" yEnc (1+20) 13632578"

[safihre]

While we could do it, we should do this with the header (like Unrar does). Needs a seperate version for rar3 and rar5.

[hypersucker]

Has this been implemented? Currently on the latest beta and I have one where SAB downloads the files just fine, uses the par2 to repair them but then I'm left with the obfuscated filenames. Those are all rars but how do I figure out which one is which in what order? I read over at NZBGet that they implemented something to read the rar file header and rename files according to this but I cant test that in Get since Get refuses to download this nzb (for health reasons) but SAB downloads it just fine.

[sander]

SABnzbd has now built-in deobfuscation of obfuscated rar files, and more deobfuscation.

Did you turn on Debofuscate Final Filenames? See http://127.0.0.1:8080/sabnzbd/config/sw ... _filenames

If it doesn't work, share your NZB via pastebin.com, or [email protected]

PS: there is one thing that SAB (nor any other too) cannot do: a whole season, fulle obfuscated, and then determine which file is which ... Impossible ...

[hypersucker]

[email protected] does not work, I sent it to [email protected] instead. I turned on the above mentioned switch and SAB did rename the obfuscated files now but I think still not properly as far as I can tell. It's a single episode btw.

[sander]

Thanks for the NZB.
SAB downloaded it, and indeed the result is obfuscated files, which are rar files.
SAB can handle obfuscated rar files. I don't know why SAB didn't do that on these files. First guess: double obfuscated, and SAB only does one run / one layer of deobfuscation.
I'm currently working on another thing for SAB, so no time yet to investigate this.

One question to you: is it just this NZB, or do have more of these problematics posts?

[hypersucker]

it is just this nzb for now. don't worry. I'll get it reupped with proper renaming.