Can't fetch NZB from my indexer on specific SABnzbd

Get help with all aspects of SABnzbd
Forum rules
Help us help you:
  • Are you using the latest stable version of SABnzbd? Downloads page.
  • Tell us what system you run SABnzbd on.
  • Adhere to the forum rules.
  • Do you experience problems during downloading?
    Check your connection in Status and Interface settings window.
    Use Test Server in Config > Servers.
    We will probably ask you to do a test using only basic settings.
  • Do you experience problems during repair or unpacking?
    Enable +Debug logging in the Status and Interface settings window and share the relevant parts of the log here using [ code ] sections.
josh4trunks
Newbie
Newbie
Posts: 30
Joined: August 25th, 2011, 10:32 am

Can't fetch NZB from my indexer on specific SABnzbd

Post by josh4trunks »

Hi everyone, I'm trying to grab an NZB from my nZEDb (newznab compatible) install. I'm trying to figure out how I can further debug this and get to the bottom of this issue. Thanks!

PROBLEM
The link i want to work is..
https://www.jruehlig.com/nzedb/getnzb/fe1a1474a5026b74b5d63b78634f6b055add149a&i=3&r=de390776ff47336d50e5585876aa1704
The log shows..

Code: Select all

Grabbing URL https://www.jruehlig.com/nzedb/getnzb/fe1a1474a5026b74b5d63b78634f6b055add149a&i=3&r=de390776ff47336d50e5585876aa1704
2014-12-24 09:34:16,288::DEBUG::[urlgrabber:372] No response from indexer, retry after 60 sec
2014-12-24 09:34:16,289::INFO::[urlgrabber:199] Retry URL https://www.jruehlig.com/nzedb/getnzb/fe1a1474a5026b74b5d63b78634f6b055add149a&i=3&r=de390776ff47336d50e5585876aa1704
SPECS
OS = FreeBSD v9.3
SABnzbd = v0.7.20
Python = v2.7.8

TESTS
This link works, I uploaded the same nzb to my server
https://www.jruehlig.com/nzedb/test.nzb

This link doesn't work, I temporarily disabled https on my server
http://www.jruehlig.com/nzedb/getnzb/fe1a1474a5026b74b5d63b78634f6b055add149a&i=3&r=de390776ff47336d50e5585876aa1704

Grabbing an NZB from OZnzb's API, which is also https and newznab based, works.

On a different SABnzbd instance the original link works
OS = FreeBSD v8.3
SABnzbd = v0.7.20
Python = v2.7.2

My tests lead me to believe it is something with the newer Python and my nZEDb api?
Last edited by josh4trunks on January 3rd, 2015, 9:05 pm, edited 2 times in total.
User avatar
shypike
Administrator
Administrator
Posts: 19774
Joined: January 18th, 2008, 12:49 pm

Re: Can't fetch NZB from my indexer on specific SABnzbd inst

Post by shypike »

We normally test on Python 2.7, I'm not sure about the last digit.
Do you have a possibility to test on another Python version?
josh4trunks
Newbie
Newbie
Posts: 30
Joined: August 25th, 2011, 10:32 am

Re: Can't fetch NZB from my indexer on specific SABnzbd inst

Post by josh4trunks »

shypike wrote:We normally test on Python 2.7, I'm not sure about the last digit.
Do you have a possibility to test on another Python version?
I see FreeBSD just added 2.7.9. I'll update and see if that makes a difference.
EDIT
Still doesn't work.
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: Can't fetch NZB from my indexer on specific SABnzbd inst

Post by sander »

When I fill out the URL, I get the same error on my system.

So I added a debug line (logging.debug("SJ: I got in the except due to %s", sys.exc_info()[0])) to my sabnzbd/urlgrabber.py, and the URL results in:

Code: Select all

2014-12-25 10:07:49,039::INFO::[urlgrabber:78] URLGrabber starting up
2014-12-25 10:07:54,044::INFO::[urlgrabber:117] Grabbing URL https://www.jruehlig.com/nzedb/getnzb/fe1a1474a5026b74b5d63b78634f6b055add149a&i=3&r=de390776ff47336d50e5585876aa1704
2014-12-25 10:07:59,012::DEBUG::[urlgrabber:136] SJ: I got in the except due to <class 'urllib.ContentTooShortError'>
2014-12-25 10:07:59,012::DEBUG::[urlgrabber:376] No response from indexer, retry after 60 sec
So: urllib.ContentTooShortError
exception urllib.error.ContentTooShortError(msg, content)
This exception is raised when the urlretrieve() function detects that the amount of the downloaded data is less than the expected amount (given by the Content-Length header). The content attribute stores the downloaded (and supposedly truncated) data.
So a problem with the Content-Lenth.
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: Can't fetch NZB from my indexer on specific SABnzbd inst

Post by sander »

Ah, this is interesting: I can reproduce an error in a few lines of code:

Code: Select all

$ python
Python 2.7.6 (default, Mar 22 2014, 22:59:56) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib
>>> opener = urllib.FancyURLopener({})
>>> url = 'https://www.jruehlig.com/nzedb/getnzb/fe1a1474a5026b74b5d63b78634f6b055add149a&i=3&r=de390776ff47336d50e5585876aa1704'
>>> fn, header = opener.retrieve(url)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/urllib.py", line 264, in retrieve
    size = int(headers["Content-Length"])
ValueError: invalid literal for int() with base 10: '2795588, 2795588'
>>>
So ... python / urllib can't convert it with int() because it's an array with two int values= (where only one is allowed) which leads to the Content-Length error!

curl output:

Code: Select all

< HTTP/1.1 200 OK
* Server nginx/1.6.2 is not blacklisted
< Server: nginx/1.6.2
< Content-Type: application/x-nzb
< Content-Length: 2795588
< X-Powered-By: PHP/5.6.3
< Set-Cookie: PHPSESSID=44v3sk5mqps0ltremgl406ae87; path=/; secure; HttpOnly
< Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< Pragma: no-cache
< Content-Disposition: attachment; filename=Slow.Burn.2005.1080p.BluRay.DTS-HD.x264-BARC0DE.nzb
< Expires: Fri, 25 Dec 2015 02:00:06 -0800
< X-DNZB-Category: Movies > HD
< X-DNZB-Details: https://www.jruehlig.com/nzedb/details/fe1a1474a5026b74b5d63b78634f6b055add149a
< X-DNZB-MoreInfo: http://www.imdb.com/title/tt0376196
< X-DNZB-Name: Slow.Burn.2005.1080p.BluRay.DTS-HD.x264-BARC0DE
< X-DNZB-NFO: https://www.jruehlig.com/nzedb/nfo/fe1a1474a5026b74b5d63b78634f6b055add149a
< X-DNZB-RCode: 200
< X-DNZB-RText: OK, NZB content follows.
< Accept-Ranges: bytes
< Date: Thu, 25 Dec 2014 10:00:06 GMT
< X-Varnish: 1713665329
< Age: 0
< Via: 1.1 varnish
< X-Cache: MISS
< Content-Length: 2795588
< 
So: twice "Content-Length: 2795588" in the HTTP ... that could be the cause. So: server side problem. Can you solve that? Do you use a web server behind a proxy server?


Related: https://bugzilla.redhat.com/show_bug.cgi?id=692525

The part in urllib.py that goes wrong when there is more than 1 Content-Length in headers:

Code: Select all

                if "content-length" in headers:
                    size = int(headers["Content-Length"])
... changing code in a system library on your own system is not something I see as a solution. ;-)
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: Can't fetch NZB from my indexer on specific SABnzbd inst

Post by sander »

Short summary:

Good (with test.nzb just one Content-Length:):

Code: Select all

$ curl -vs -o mynzb.nzb https://www.jruehlig.com/nzedb/test.nzb 2>&1 | grep Content-Length
< Content-Length: 2795588
Bad (resulting in two times "Content-Length:" in the HTTP header):

Code: Select all

$ curl -vs -o mynzb2.nzb 'https://www.jruehlig.com/nzedb/getnzb/fe1a1474a5026b74b5d63b78634f6b055add149a&i=3&r=de390776ff47336d50e5585876aa1704'  2>&1 | grep Content-Length
< Content-Length: 2795588
< Content-Length: 2795588
So ... solve it on your server side?
josh4trunks
Newbie
Newbie
Posts: 30
Joined: August 25th, 2011, 10:32 am

Re: Can't fetch NZB from my indexer on specific SABnzbd inst

Post by josh4trunks »

Thanks for the debugging! I do have a somewhat complex setup for my hosting.
HAProxy (SSL)
Varnish (Static Caching + GZIPing)
NGINX (Static Webserver)
PHP-FPM

I already tried cutting out the SSL step and that didn't help, so I'll try further isolating the problem. What's strange is how it works on my older setup.
Thanks!
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: Can't fetch NZB from my indexer on specific SABnzbd inst

Post by sander »

Ouch. Well if you're able to construct such a setup, I'm sure you're able solve this tiny Content-Length problem ... ;-)
josh4trunks
Newbie
Newbie
Posts: 30
Joined: August 25th, 2011, 10:32 am

Re: Can't fetch NZB from my indexer on specific SABnzbd inst

Post by josh4trunks »

sander wrote:Ouch. Well if you're able to construct such a setup, I'm sure you're able solve this tiny Content-Length problem ... ;-)
Looks like Varnish 3.0.4 had a bug which was fixed in 3.0.5 about duplicate Content-Length headers. pfSense's package is only 3.0.4, so time to complain to the package maintainer =]
Thanks, my problem had been solved, move along, and have a Merry Christmas.
josh4trunks
Newbie
Newbie
Posts: 30
Joined: August 25th, 2011, 10:32 am

Re: Can't fetch NZB from my indexer on specific SABnzbd

Post by josh4trunks »

@sander
I fixed the Content-Length issue. Are you able to download the nzb's now? I still can't. I'll get some logs posted soon.
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: Can't fetch NZB from my indexer on specific SABnzbd

Post by sander »

It does not work on the stock version of SABnzbd:

Code: Select all

2015-01-04 07:46:42,344::INFO::[urlgrabber:116] Grabbing URL https://www.jruehlig.com/nzedb/getnzb/fe1a1474a5026b74b5d63b78634f6b055add149a&i=3&r=de390776ff47336d50e5585876aa1704
2015-01-04 07:46:46,366::DEBUG::[urlgrabber:372] No response from indexer, retry after 60 sec
2015-01-04 07:46:46,366::INFO::[urlgrabber:199] Retry URL https://www.jruehlig.com/nzedb/getnzb/fe1a1474a5026b74b5d63b78634f6b055add149a&i=3&r=de390776ff47336d50e5585876aa1704
In the 0.7.x git version, it doesn't work, with explanation urllib.ContentTooShortError:

Code: Select all

$ git branch
* 0.7.x
  develop
  master

2015-01-04 07:56:50,572::INFO::[urlgrabber:117] Grabbing URL https://www.jruehlig.com/nzedb/getnzb/fe1a1474a5026b74b5d63b78634f6b055add149a&i=3&r=de390776ff47336d50e5585876aa1704
2015-01-04 07:56:54,530::DEBUG::[urlgrabber:136] Exception <class 'urllib.ContentTooShortError'> trying to get the url https://www.jruehlig.com/nzedb/getnzb/fe1a1474a5026b74b5d63b78634f6b055add149a&i=3&r=de390776ff47336d50e5585876aa1704
2015-01-04 07:56:54,531::DEBUG::[urlgrabber:375] No response from indexer, retry after 60 sec
2015-01-04 07:56:54,531::INFO::[urlgrabber:202] Retry URL https://www.jruehlig.com/nzedb/getnzb/fe1a1474a5026b74b5d63b78634f6b055add149a&i=3&r=de390776ff47336d50e5585876aa1704

It does work on the 'develop' git version of SABnzbd:

Code: Select all

$ git branch
  0.7.x
* develop
  master


2015-01-04 07:41:46,990::INFO::[__init__:466] Fetching https://www.jruehlig.com/nzedb/getnzb/fe1a1474a5026b74b5d63b78634f6b055add149a&i=3&r=de390776ff47336d50e5585876aa1704
2015-01-04 07:41:46,991::INFO::[nzbqueue:236] Saving queue
2015-01-04 07:41:46,992::DEBUG::[__init__:809] Saving data for SABnzbd_nzo_u6rO4Q in /home/sander/.sabnzbd/admin/future
2015-01-04 07:41:46,993::INFO::[__init__:884] Saving data for queue10.sab in /home/sander/.sabnzbd/admin/queue10.sab
2015-01-04 07:41:46,994::INFO::[urlgrabber:96] Grabbing URL https://www.jruehlig.com/nzedb/getnzb/fe1a1474a5026b74b5d63b78634f6b055add149a&i=3&r=de390776ff47336d50e5585876aa1704
2015-01-04 07:41:52,087::INFO::[misc:812] Creating directories: /home/sander/Downloads/incomplete/Slow.VVV.2005.1080p.BluRay.DTS-HD.x264-BARC0DE
2015-01-04 07:41:52,090::DEBUG::[__init__:809] Saving data for SABnzbd_nzf_QK_Z3i in /home/sander/Downloads/incomplete/Slow.VVV.2005.1080p.BluRay.DTS-HD.x264-BARC0DE/__ADMIN__
2015-01-04 07:41:52,090::INFO::[nzbstuff:425] File Slow.VVV.2005.1080p.BluRay.DTS-HD.x264-BARC0DE - [00/63] - "BARC0DE.nzb" yEnc (1/8) added to queue
The small python test snippet does work:

Code: Select all

sander@flappie:~$ python
Python 2.7.6 (default, Mar 22 2014, 22:59:56) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib
>>> opener = urllib.FancyURLopener({})
>>> url = 'https://www.jruehlig.com/nzedb/getnzb/fe1a1474a5026b74b5d63b78634f6b055add149a&i=3&r=de390776ff47336d50e5585876aa1704'
>>> fn, header = opener.retrieve(url)
>>> print header
Server: nginx/1.6.2
Content-Type: application/x-nzb
X-Powered-By: PHP/5.6.3
Set-Cookie: PHPSESSID=ugenhd6lcj43o21b18srlbbl61; path=/; secure; HttpOnly
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Content-Disposition: attachment; filename=Slow.Burn.2005.1080p.BluRay.DTS-HD.x264-BARC0DE.nzb
Expires: Sun, 03 Jan 2016 22:53:24 -0800
X-DNZB-Category: Movies > HD
X-DNZB-Details: https://www.jruehlig.com/nzedb/details/fe1a1474a5026b74b5d63b78634f6b055add149a
X-DNZB-MoreInfo: http://www.imdb.com/title/tt0376196
X-DNZB-Name: Slow.Burn.2005.1080p.BluRay.DTS-HD.x264-BARC0DE
X-DNZB-NFO: https://www.jruehlig.com/nzedb/nfo/fe1a1474a5026b74b5d63b78634f6b055add149a
X-DNZB-RCode: 200
X-DNZB-RText: OK, NZB content follows.
Accept-Ranges: bytes
Date: Sun, 04 Jan 2015 06:53:24 GMT
X-Varnish: 780103128
Age: 0
Via: 1.1 varnish
Connection: close
X-Cache: MISS
Content-Length: 2795588

>>> print fn
/tmp/tmpFkHX0r
>>> 

So .. quite a mystery to me. Hopefully not something intermittent ... :'(
josh4trunks
Newbie
Newbie
Posts: 30
Joined: August 25th, 2011, 10:32 am

Re: Can't fetch NZB from my indexer on specific SABnzbd

Post by josh4trunks »

@sander
Thanks for the testing, that log from 0.7.X points me in a direction to test.

I tested querying my webserver directly NGINX > PHP from the install that is having an issue and that "fixes" it so somewhere is the Varnish layer.
I have Varnish GZIP plaintext so I'll double check if this file is GZIPed and if Content-Length is being represented incorrectly.

I know clients (browsers / urllib) are supposed to say what what compression they support (GZIP/DEFLATE) and servers can then choose to do transparent compression, for hopefully compressible files. I'll try to find out what length they are supposed to give (compressed || uncompressed), if my particular setup is doing this wrong or if sabnzbd 0.7.X has a bug for this.
josh4trunks
Newbie
Newbie
Posts: 30
Joined: August 25th, 2011, 10:32 am

Re: Can't fetch NZB from my indexer on specific SABnzbd

Post by josh4trunks »

@sander
I think I found the issue after reading this https://sabnzbd.org/viewtopic.php?f=3&t=12992&p=78409

Sabnzbd actually passes the header saying it do gzip encoding.
Varnish then see's this and gives..

Code: Select all

Content-Encoding: gzip
Transfer-Encoding: chunked
Since sabnzbd uses urllib, which is HTTP/1.0, this issue occurs "if the sender does not include a Content-Length header, the recipient cannot tell if the message has been truncated due to transmission problems."
from.. http://www8.org/w8-papers/5c-protocols/key/key.html

I think my Varnish is setup to assume everyone can handle HTTP/1.1 headers, so in a way it is my problem.

####

So, here's my question =]

* When is sabnzbd switching to urllib2?

Thanks!
User avatar
shypike
Administrator
Administrator
Posts: 19774
Joined: January 18th, 2008, 12:49 pm

Re: Can't fetch NZB from my indexer on specific SABnzbd

Post by shypike »

josh4trunks wrote: * When is sabnzbd switching to urllib2?
It has in 0.8.0 (not out yet)
There won't be a new 0.7.x (at least not for issues like this).
josh4trunks
Newbie
Newbie
Posts: 30
Joined: August 25th, 2011, 10:32 am

Re: Can't fetch NZB from my indexer on specific SABnzbd

Post by josh4trunks »

shypike wrote:
josh4trunks wrote: * When is sabnzbd switching to urllib2?
It has in 0.8.0 (not out yet)
There won't be a new 0.7.x (at least not for issues like this).
so I take it sabnzbd 0.8.0 will be out soon =]
Post Reply