pre-check on contents of post for unwanted content

[sander]

Hi,

I have more and more nonsense downloads with a fake *.avi and a Codec/Setup.exe, which exe contains a virus like "Packed.Win32.TDSS.2!O, Artemis!33B01D7F9BF5, PUP.Optional.OptimumInstaller.A"

My feature request: do a pre-check on the first rar (so *01.rar) for the contents of the rar file. If there is 'forbidden' file extension (like .exe), pause or abort the download. That would save time and bandwidth.
So this looks like the current function 'Action when encrypted RAR is downloaded' in Config -> Switches.

Below is an example test on a *.part01.rar file

Code: Select all

sander@flappie:~/Downloads/complete$ unrar l someDownload.part01.rar 

UNRAR 5.00 beta 8 freeware      Copyright (c) 1993-2013 Alexander Roshal

Archive: someDownload.part01.rar
Details: RAR 4, volume

 Attributes      Size    Date   Time   Name
----------- ---------  -------- -----  ----
    ..A....    220448  14-03-14 14:56  Blabla.XviD-Ecf/Codec/Setup.exe
    ..A....       113  14-03-14 00:22  Blabla.XviD-Ecf/If_you_get_error.txt
    ..A.... 736100352  19-03-14 11:51  Blabla.XviD-Ecf/Blabla.XviD-Ecf.avi
----------- ---------  -------- -----  ----
            736320913  volume 1        3

sander@flappie:~/Downloads/complete$ 

[sander]

OK, proof of concept written & working: The download pauses after the first 'offending' rar file is in.

In the SAB Warning page:

2014-03-30 20:27:17,702 WARNING: WARNING:: Paused job "blabla.XviD-Ecf.part13" because of unwanted content in RAR file SJ

I've put the code into assembler.py, and I've copy-pasted from the functionality and use of check_encrypted_rar():

Three new lines at the end of this block:

Code: Select all

                    if check_encrypted_rar(nzo, filepath):
                        if cfg.pause_on_pwrar() == 1:
                            logging.warning(Ta('WARNING: Paused job "%s" because of encrypted RAR file'), latin1(nzo.final_name))
                            nzo.pause()
                        else:
                            logging.warning(Ta('WARNING: Aborted job "%s" because of encrypted RAR file'), latin1(nzo.final_name))
                            nzo.fail_msg = T('Aborted, encryption detected')
                            import sabnzbd.nzbqueue
                            sabnzbd.nzbqueue.NzbQueue.do.end_job(nzo)

                    if rar_contains_unwanted_file(nzo, filepath):
                            logging.warning(Ta('WARNING: Paused job "%s" because of unwanted content in RAR file SJ'), latin1(nzo.final_name))
                            nzo.pause()

and a new function (with 'exe' hardcoded for the time being):

Code: Select all

def rar_contains_unwanted_file(nzo, filepath):
	unwanted = False
	if is_rarfile(filepath):
		try:
			zf = RarFile(filepath, all_names=True)
			logging.debug('files in rar file SJ: %s', zf.namelist())
			for somefile in zf.namelist() :
				logging.debug('file in rar file SJ: %s', somefile)
				if somefile.lower().endswith('exe'):
					unwanted = True
			zf.close()
		except:
			logging.debug('RAR file %s cannot be inspected SJ', filepath)
	return unwanted

This will save me a lot of time & bandwidth.

[sander]

PS:

Some stuff from sabnzbd.log:

Code: Select all

2014-03-30 20:55:10,222::DEBUG::[assembler:332] rar file to check SJ: /home/sander/Downloads/incomplete/Blabla.XviD-Ecf.part13/Blabla.XviD-Ecf.part01.rar
2014-03-30 20:55:10,222::DEBUG::[assembler:335] files in rar file SJ: ['Blabla.XviD-Ecf/Codec/Setup.exe', 'Blabla.XviD-Ecf/If_you_get_error.txt', 'Blabla.XviD-Ecf/Blabla.XviD-Ecf.avi']
2014-03-30 20:55:10,223::DEBUG::[assembler:337] file in rar file SJ: Blabla.XviD-Ecf/Codec/Setup.exe
2014-03-30 20:55:10,223::DEBUG::[assembler:337] file in rar file SJ: Blabla.XviD-Ecf/If_you_get_error.txt
2014-03-30 20:55:10,223::DEBUG::[assembler:337] file in rar file SJ: Blabla.XviD-Ecf/Blabla.XviD-Ecf.avi
2014-03-30 20:55:10,223::WARNING::[assembler:125] WARNING: Paused job "Blabla.XviD-Ecf.part13" because of unwanted content in RAR file SJ

EDIT:

Visible warning is now

2014-03-30 21:35:04,079 WARNING: WARNING:: Paused job "blabla" because of unwanted content in RAR file SJ. Unwanted file is Blabla.DVDRip.XviD-LEd/Codec/Setup.exe

[shypike]

Interesting idea, I will look into it.

[sander]

I've uploaded my modded SABnzbd to my own SABnzbd github fork.
See git diff here: https://github.com/sanderjo/sabnzbd/com ... c7defdb324

For those who want to test before I send a request to Shypike's git, follow these git instructions:

Code: Select all

git clone https://github.com/sanderjo/sabnzbd.git
cd sabnzbd/
git checkout 0.7.x
grep -i unwanted *

That last command should give a few lines.

Stop SABnzbd, open sabnzbd.ini for editing, and in the [misc]-part insert:

Code: Select all

unwanted_extensions = exe, bla
pause_on_unwanted_extensions = 1

Meaning of the value of pause_on_unwanted_extensions:
0 = just warn
1 = pause
2 = abort


Then start SAB the normal way:

Code: Select all

./SABnzbd.py

Example of sabnzbd.log info:

Code: Select all

2014-04-04 20:49:51,697::DEBUG::[SABnzbd:1335] Unwanted extensions are ['exe', 'bla']
2014-04-04 20:50:40,040::DEBUG::[assembler:355] Unwanted file blablabla.LiMiTED.DVDRip.XviD-Ecf/Codec/Setup.exe
2014-04-04 20:50:40,040::WARNING::[assembler:127] WARNING: In "blablabla.LiMiTED.DVDRip.XviD-Ecf.part13" unwanted extension in RAR file. Unwanted file is blablabla.LiMiTED.DVDRip.XviD-Ecf/Codec/Setup.exe 
2014-04-04 20:50:40,040::DEBUG::[assembler:129] Unwanted extension ... pausing

Please post back your feedback

[TonioRoffo]

OK, I just did the git clone in a new folder to check this build.

Changed sabnzbd.ini and added the said lines:

under [misc]
unwanted_extensions = wmv
pause_on_unwanted_extensions = 2

I start SABnzbd.py, then exit after a while

I check my sabnzbd.ini file and my lines are gone. Where should they go, I'm obviously putting them in a wrong place.

Thanks

[sander]

Oops: the options must indeed be put into the [misc] part. I'll change that in my post.

Now your problem: did you stop SABnzbd first, and then manually edit sabnzbd.ini, then save & exit ?

[Eejit]

A very interesting idea. I search for exe files AFTER it's all downloaded and unpacked in my post processing script for movies.
How does your mod effect applications being downloaded? Is it category dependant?

[sander]

A very interesting idea. I search for exe files AFTER it's all downloaded and unpacked in my post processing script for movies.
How does your mod effect applications being downloaded? Is it category dependant?

Well, I'm on Linux, so I never download applications from newsgroups, and thus I've not included any category dependency.

The good news: if SAB pauses the download (so: option 1), you can Resume it again, and the download continues and you'll get your full download anyway.

[sander]

FYI:

The SAB mod is working very well for me; it seems more and more posts contain fake content combined with a file "Codec/Setup.exe" (which contains malware)

With the mod, SAB will detect it very early and pause the download with a Warning. Screenshot:



And the Warning page:

[TonioRoffo]

my bad, it works for me as well. I probably didn't stop the sabnzbd process first time I edited the .ini file.

I hope this makes it to the code.

[sander]

FYI: I've sent a Pull Request to Shypike: https://github.com/sabnzbd/sabnzbd/pull/147

[sander]

Update:

As of the git version as of today 2014-04-13, you need a new format in sabnzbd.ini, still in the [misc] part:

Code: Select all

unwanted_extensions = .exe, .bla
action_on_unwanted_extensions = 1

Changes:
The extensions MUST be in lower case and with a leading dot.
The option is now 'action_on...' (not 'pause_on...')

Logging has changed a little bit:

Code: Select all

2014-04-13 09:36:35,973::DEBUG::[SABnzbd:1327] Unwanted extensions are ... ['.exe', '.bla']
2014-04-13 09:41:44,717::DEBUG::[assembler:355] Unwanted file blabla.2013.iNTERNAL.DVDRip.XviD-nEo/Codec/CG_Xvid_Codecs_Setup.exe
2014-04-13 09:41:44,717::WARNING::[assembler:127] WARNING: In "blabla 2013 (3)" unwanted extension in RAR file. Unwanted file is blabla.2013.iNTERNAL.DVDRip.XviD-nEo/Codec/CG_Xvid_Codecs_Setup.exe 
2014-04-13 09:41:44,718::DEBUG::[assembler:129] Unwanted extension ... pausing

HTH

[andymac]

This is great!

I was looking into ways to fix newznab as well as sab to take care of this same issue and got directed here. For myself, I prefer to simply have the job aborted so couch can go find something new. I'll pull down your clone later, but for now, thanks for posting the functions in the beginning. I just changed them to abort the job.

So I'll check it out when I look at the clone, but my only addition to what you've obviously added would be an option to abort rather than just pause.

Otherwise, I'll try to keep an eye to see if / when this is pulled into the main!

Great help, man! Now if only newznab / nZedb could index these and remove them right away. Harder for them as they'd have to pull each rar to do that (the file doesn't show up in the main rar). Pipe dream, probably .

Thanks!

[sander]

my only addition to what you've obviously added would be an option to abort rather than just pause.

Meaning of the value of action_on_unwanted_extensions:
0 = just warn
1 = pause
2 = abort

... so Abort is already there.