test nzb before download

Get help with all aspects of SABnzbd
Forum rules
Help us help you:
  • Are you using the latest stable version of SABnzbd? Downloads page.
  • Tell us what system you run SABnzbd on.
  • Adhere to the forum rules.
  • Do you experience problems during downloading?
    Check your connection in Status and Interface settings window.
    Use Test Server in Config > Servers.
    We will probably ask you to do a test using only basic settings.
  • Do you experience problems during repair or unpacking?
    Enable +Debug logging in the Status and Interface settings window and share the relevant parts of the log here using [ code ] sections.
sailmakered
Release Testers
Release Testers
Posts: 9
Joined: September 16th, 2010, 8:50 pm

test nzb before download

Post by sailmakered »

Version: 0.5.0 Final
OS: Vista
Install-type: Windows Installer
Skin (if applicable): Plush
Firewall Software: Vista
Are you using IPV6? no
Is the issue reproducible? n/a


Great program!!

Anyone use anything to test nzb file part completion before download?  Usenetserver.com claims 600 day retention but I seem to get 400 day files with a lot of the files missing 1 or 2 parts.  Too many for the pars to fix.  Any ideas?
User avatar
shypike
Administrator
Administrator
Posts: 19774
Joined: January 18th, 2008, 12:49 pm

Re: test nzb before download

Post by shypike »

You can try http://www.newsgroupdownloaders.com/nzbval/
I'm not familiar with it and you should be able to google some more.
We do have plans to add pre-download verification at some time in the future.
sailmakered
Release Testers
Release Testers
Posts: 9
Joined: September 16th, 2010, 8:50 pm

Re: test nzb before download

Post by sailmakered »

Thanks

Tried it....really ssslllloooowwwww.

Sounds like a great thing to add to sab.
User avatar
shypike
Administrator
Administrator
Posts: 19774
Joined: January 18th, 2008, 12:49 pm

Re: test nzb before download

Post by shypike »

Slowness can be caused by the server as well.
Did you check an NZB with lots of missing articles?
sailmakered
Release Testers
Release Testers
Posts: 9
Joined: September 16th, 2010, 8:50 pm

Re: test nzb before download

Post by sailmakered »

I did.

Just tried an nzb I knew was complete.  8.839GB (150 rar x 66 parts) in about 10 minutes.  Program didn't max out my connection or cpu.  Seemed slower than downloading headers in xnews but sometimes those headers remain but the file doesn't.
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: test nzb before download

Post by sander »

shypike wrote:
We do have plans to add pre-download verification at some time in the future.
I've written a python script that does pre-download verification by STAT-ing all articles in a given NZB. It will tell the complete-ness of the post on your server.
It works, but can be a bit slow (about 10 minutes for the check of a 4.5 GB post), probably because the script is non/single-threaded: to check a 4.5GB post, this means around 13.000 stat-send/expect commands in sequential order. Apparently, one thread can do 20 STAT commands per second.
So I guess multi-threading can bring speed improvement.

The script is here: http://www.appelboor.com/sander/nzb-checker.py

Code: Select all

Usage:
nzb-checker.py <NZBfilename> <newsserver>
nzb-checker.py <NZBfilename> <newsserver> <newsserver-username> <newsserver-pass>

So, for example:
python nzb-checker.py toy71MB.nzb newszilla6.xs4all.nl
python nzb-checker.py toy71MB.nzb newsreader3.eweka.nl myname secretpass


Code: Select all

ubuntu@ubuntu:~/nzb-checker$ time python nzb-checker.py toy.nzb newszilla6.xs4all.nl
number of arguments: 2
toy.nzb newszilla6.xs4all.nl  
201 dreader33.news.xs4all.nl NNRP Service Ready - [email protected] (no posting).

NNTP connection OK

Let's start checking ...
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 

total artciles in NZB 200 , total available on newsserver 133
Completeness on server is 66.50 %
size of (original) post is 71 Mbytes
age of post (in days): 3

real	0m7.453s
user	0m0.112s
sys	0m0.056s
ubuntu@ubuntu:~/nzb-checker$ 
Please don't send me unrequested PM's; the forum is the best way to communicate.
If someone helps you, please reply to that help.
f you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: test nzb before download

Post by sander »

Followup:

I've now written a mutli-threaded nzb-checker: checking a NZB for a 4.5GB post takes around 25 seconds, when using 30 threads (=connections to the newsserver)

Let me know if you're interested in the python script.

Code: Select all

ubuntu@ubuntu:~/nzb-checker$ time python mijn-nzb-draadje.py  --nzb=young-sept17-4.5GB.nzb --server=reader.ipv6.xsnews.nl --thread=30 --user=ipv6-blabla --pass=blabla
Start reading NZB file ...
... reading NZB file completed
Wait until threads are done ...
... threads are done


Articles: 12874
OK: 12874
Not OK: 0
Connection error: 0
Completeness on server is 100.00 %

real	0m25.179s
user	0m5.100s
sys	0m1.164s
ubuntu@ubuntu:~/nzb-checker$
Please don't send me unrequested PM's; the forum is the best way to communicate.
If someone helps you, please reply to that help.
f you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
User avatar
inpheaux
Administrator
Administrator
Posts: 563
Joined: January 16th, 2008, 9:14 pm

Re: test nzb before download

Post by inpheaux »

Holy crap great job Sander!

I wonder how hard this would be to implement as a pre-processing script?
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: test nzb before download

Post by sander »

inpheaux wrote: Holy crap great job Sander!
LOL!
inpheaux wrote:
I wonder how hard this would be to implement as a pre-processing script?
I guess not too hard; it should use the SABnzbd parameters (newsserver etc), and it could best be run when no other downloads are running as it uses the same newsserver connections.

It's here http://www.appelboor.com/sander/nzb-checker-threaded.py

The script works great for sunny day scenario's and it does some checks on successful connections, but it is not yet good at handling error situations / raindy day scenario's.
Please don't send me unrequested PM's; the forum is the best way to communicate.
If someone helps you, please reply to that help.
f you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
phyber
Newbie
Newbie
Posts: 5
Joined: February 10th, 2011, 9:12 am

Re: test nzb before download

Post by phyber »

Here, have a small hack to make this support SSL connections.
Works fine for me in the few tests I've done. YMMV.

Code: Select all

--- nzb-checker-threaded.py	2011-09-19 21:54:22.000000000 +0100
+++ ssl-checker.py	2011-02-10 14:13:36.000000000 +0000
@@ -1,3 +1,4 @@
+#!/usr/bin/python
 import Queue
 from threading import Thread
 import time
@@ -36,7 +37,18 @@
 	# logon to Newsserver
 	if debug: print "trying to connect to newsserver ..."
 	try:
-		tn = telnetlib.Telnet(newsserver,119,5)
+		if newsssl == True:
+			import socket
+			import ssl
+			tn = telnetlib.Telnet()
+			tn.host = newsserver
+			tn.port = 563
+			tn.timeout = 5
+			tn.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+			tn.sock = ssl.wrap_socket(tn.sock)
+			tn.sock.connect((tn.host, tn.port))
+		else:
+			tn = telnetlib.Telnet(newsserver,119,5)
 		waarde = tn.read_until("\n",60)
 
 	except:
@@ -101,7 +113,7 @@
 
 
 def printUsage():
-	print "Usage: python usage-checker.py  --nzb=thisismynzb.nzb --server=newszilla6.xs4all.nl [--thread=4] [--user=newsservername] [--pass=newsserverpassword] [--debug=1]"
+	print "Usage: python usage-checker.py  --nzb=thisismynzb.nzb --server=newszilla6.xs4all.nl [--thread=4] [--user=newsservername] [--pass=newsserverpassword] [--ssl=1] [--debug=1]"
 
 ###########################################################################################################################
 ###########################################################################################################################
@@ -111,7 +123,7 @@
 
 
 try:
-        opts, args = getopt.getopt(sys.argv[1:], "n:s:u:p:t:d",['nzb=','server=','user=','pass=','threads=','debug='])
+	opts, args = getopt.getopt(sys.argv[1:], "n:s:u:p:t:dS",['nzb=','server=','user=','pass=','threads=','debug=','ssl='])
 except:
 	printUsage()
 	sys.exit()	
@@ -119,6 +131,7 @@
 
 # defaults:
 nzbfilename = newsserver = newsname = newspass = ''
+newsssl = False
 debug = False
 num_worker_threads = 1
 
@@ -133,6 +146,8 @@
 		newspass = arg
 	elif opt in ('-t', '--threads'):
 		num_worker_threads = int(arg)
+	elif opt in ('-S', '--ssl'):
+		newsssl = True
 	elif opt in ('-d', '--debug'):
 		if int(arg)>0:
 			debug = True
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: test nzb before download

Post by sander »

@phyper: do you use my script? Does it work?
Please don't send me unrequested PM's; the forum is the best way to communicate.
If someone helps you, please reply to that help.
f you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
phyber
Newbie
Newbie
Posts: 5
Joined: February 10th, 2011, 9:12 am

Re: test nzb before download

Post by phyber »

Yes, this is a patch to your script. It should apply cleanly to the download you have at http://www.appelboor.com/sander/nzb-checker-threaded.py
I tested it on a few NZBs I had around and they all seemed to check out just fine. There is a slight delay after logging in with SSL mode compared to the non-SSL mode, but that might just be giganews being weird. I don't have another SSL news server to test it against.
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: test nzb before download

Post by sander »

The script is now unstoppable: CTRL-C does not work. Do you know how to get that working? I tried the stuff below, but that doesn't work :-(


try:
workqueue.join()
except KeyboardInterrupt:
raise
except:
print "this is the except"
Please don't send me unrequested PM's; the forum is the best way to communicate.
If someone helps you, please reply to that help.
f you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
phyber
Newbie
Newbie
Posts: 5
Joined: February 10th, 2011, 9:12 am

Re: test nzb before download

Post by phyber »

It stops responding to ^C since it seems there is a deadlock somewhere, which is a shame. I couldn't work out where it was so I wrote a new script that does exactly what this script does, except it supports SSL and doesn't hang when using it. It also isn't threaded yet, so it's quite a bit slower.
It still needs a bit of cleaning up, but I'll hopefully be back to leave it here for people to use soon.
warthog
Newbie
Newbie
Posts: 7
Joined: March 15th, 2010, 3:27 pm

Re: test nzb before download

Post by warthog »

Regarding phyber's patch, I have some familiarity with python, but I'm now a little lost. Is there a way to apply the patch directly with python or some other utility in a *nix environment or does the code need to be added manually to the script from sander?

Figured it out. For anyone else who may be curious take phyber's code and paste into a file named yourfilename.diff (I used checker-patch.diff)

Code: Select all

patch nzb-checker-threaded.py checker-patch.diff
Last edited by warthog on April 12th, 2011, 12:17 pm, edited 1 time in total.
Locked