Issues with OpenSSL and Sabnzbd

[poizun]

Hey all,

Running into issues currently where sab isn't able to fetch NZBs via the URL. This includes things that are dropped in by other apps and also if I manually upload it by pointing it to a URL.

I can open the URL from my browser and obtain the NZB though.

I believe it has something to do with OpenSSL just judging by some log entries; there are some SSL handshake errors, but it's not limited to just one site/indexer.

Any ideas on how to solve this?

[shypike]

Any ideas on how to solve this?

Not without more information.
Like, which operating system.

[typo101]

I was having a similar issue, and it certainly was OpenSSL related, specifically that SSLv3 is disabled by default in Python 2.7.9. I found two solutions: downgrade to python 2.7.8 or patch urlgrabber.

Here is my patch:

Code: Select all

--- urlgrabber.py.old	2015-03-19 22:40:59.643171308 -0400
+++ urlgrabber.py.new	2015-03-19 22:40:07.611581288 -0400
@@ -27,0 +28 @@
+import ssl
@@ -116,0 +118,3 @@
+                context = ssl.create_default_context()
+                context = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
+                context.verify_mode = ssl.CERT_NONE
@@ -118 +122 @@
-                    opener = urllib.URLopener({})
+                    opener = urllib.URLopener(context=context)
@@ -120 +124 @@
-                    opener = urllib.FancyURLopener({})
+                    opener = urllib.FancyURLopener(context=context)

EDIT: I pasted the wrong patch. On the indexer I was using TLSv1 and SSLv23 work, but not TLSv1_1 or TLSv1_2. I meant to paste the patch that used TLSv1 because it is more secure than SSLv23

[typo101]

So it wasnt until after I posted (and even edited) that I realized the contradiction in my patch and the apparent reason for my patch. The fact is the indexer I was using supports a connection that doesn't use SSLv3, so I shouldn't have to patch the SSL context at all. The real problem was that my python environment wasn't finding the CA file. I am running FreeBSD 9.3 and the root cause was that security/ca_root_nss wasn not installing the crt file in the one place Python was looking: /etc/ssl/cert.pem

Before:

Code: Select all

Python 2.7.9 (default, Feb 26 2015, 02:55:37) 
[GCC 4.2.1 20070831 patched [FreeBSD]] on freebsd9
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.get_default_verify_paths()
DefaultVerifyPaths(cafile=None, capath='/etc/ssl/certs', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/etc/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/etc/ssl/certs')

Fix:

Code: Select all

$ sudo ln -s /usr/local/share/certs/ca-root-nss.crt /etc/ssl/cert.pem

After

Code: Select all

Python 2.7.9 (default, Feb 26 2015, 02:55:37) 
[GCC 4.2.1 20070831 patched [FreeBSD]] on freebsd9
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.get_default_verify_paths()
DefaultVerifyPaths(cafile='/etc/ssl/cert.pem', capath='/etc/ssl/certs', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/etc/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/etc/ssl/certs')

[poizun]

Hello,

I really appreciate the help! Apologies for being a little log-starved. I had intended to post a follow-up/edit my post after pulling some logs, but got side-tracked by my 1 year old.

Here are some logs.

sabnzbd.error.log:

Code: Select all

Exception in thread CP WSGIServer Thread-14:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/local/share/sabnzbdplus/cherrypy/wsgiserver/__init__.py", line 1295, in run
    conn.communicate()
  File "/usr/local/share/sabnzbdplus/cherrypy/wsgiserver/__init__.py", line 1212, in communicate
    req.simple_response("408 Request Timeout")
  File "/usr/local/share/sabnzbdplus/cherrypy/wsgiserver/__init__.py", line 612, in simple_response
    self.wfile.sendall("".join(buf))
  File "/usr/local/share/sabnzbdplus/cherrypy/wsgiserver/__init__.py", line 1126, in sendall
    return self._safe_call(False, super(SSL_fileobject, self).sendall, *args, **kwargs)
  File "/usr/local/share/sabnzbdplus/cherrypy/wsgiserver/__init__.py", line 1078, in _safe_call
    return call(*args, **kwargs)
  File "/usr/local/share/sabnzbdplus/cherrypy/wsgiserver/__init__.py", line 748, in sendall
    bytes_sent = self.send(data)
  File "/usr/local/share/sabnzbdplus/cherrypy/wsgiserver/__init__.py", line 1129, in send
    return self._safe_call(False, super(SSL_fileobject, self).send, *args, **kwargs)
  File "/usr/local/share/sabnzbdplus/cherrypy/wsgiserver/__init__.py", line 1108, in _safe_call
    raise FatalSSLAlert(*e.args)
FatalSSLAlert: [('SSL routines', 'SSL23_WRITE', 'ssl handshake failure')]

sabnzbd.log:

Code: Select all

2015-03-20 22:08:57,566::INFO::[urlgrabber:116] Grabbing URL https://XXXXX
2015-03-20 22:08:57,815::INFO::[urlgrabber:199] Retry URL https://XXXXX

[poizun]

So it wasnt until after I posted (and even edited) that I realized the contradiction in my patch and the apparent reason for my patch. The fact is the indexer I was using supports a connection that doesn't use SSLv3, so I shouldn't have to patch the SSL context at all. The real problem was that my python environment wasn't finding the CA file. I am running FreeBSD 9.3 and the root cause was that security/ca_root_nss wasn not installing the crt file in the one place Python was looking: /etc/ssl/cert.pem

Before:

Code: Select all

Python 2.7.9 (default, Feb 26 2015, 02:55:37) 
[GCC 4.2.1 20070831 patched [FreeBSD]] on freebsd9
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.get_default_verify_paths()
DefaultVerifyPaths(cafile=None, capath='/etc/ssl/certs', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/etc/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/etc/ssl/certs')

Fix:

Code: Select all

$ sudo ln -s /usr/local/share/certs/ca-root-nss.crt /etc/ssl/cert.pem

After

Code: Select all

Python 2.7.9 (default, Feb 26 2015, 02:55:37) 
[GCC 4.2.1 20070831 patched [FreeBSD]] on freebsd9
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.get_default_verify_paths()
DefaultVerifyPaths(cafile='/etc/ssl/cert.pem', capath='/etc/ssl/certs', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/etc/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/etc/ssl/certs')

Thanks so much for your help! I tried the symbolic link, but that did not work. Maybe I'll try patching urlgrabber next...

[poizun]

I was having a similar issue, and it certainly was OpenSSL related, specifically that SSLv3 is disabled by default in Python 2.7.9. I found two solutions: downgrade to python 2.7.8 or patch urlgrabber.

Here is my patch:

Code: Select all

--- urlgrabber.py.old	2015-03-19 22:40:59.643171308 -0400
+++ urlgrabber.py.new	2015-03-19 22:40:07.611581288 -0400
@@ -27,0 +28 @@
+import ssl
@@ -116,0 +118,3 @@
+                context = ssl.create_default_context()
+                context = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
+                context.verify_mode = ssl.CERT_NONE
@@ -118 +122 @@
-                    opener = urllib.URLopener({})
+                    opener = urllib.URLopener(context=context)
@@ -120 +124 @@
-                    opener = urllib.FancyURLopener({})
+                    opener = urllib.FancyURLopener(context=context)

EDIT: I pasted the wrong patch. On the indexer I was using TLSv1 and SSLv23 work, but not TLSv1_1 or TLSv1_2. I meant to paste the patch that used TLSv1 because it is more secure than SSLv23

typo101, I'm not much of a drinker, but I'd buy you a beer right now.

I did a manual installation of py-urlgrabber via:

Code: Select all

portmaster /usr/ports/www/py-urlgrabber

And now things are working perfectly! Thanks for the idea! You're awesome!

[poizun]

OK some good news and some bad news.

The good news is that the issue went away with one of my indexers. The bad news is that it's still present with my other one.

Same errors. Spoke too soon. :/

Maybe something that might help is the error from a wget for the same URL:

Code: Select all

OpenSSL: error:14077458:SSL routines:SSL23_GET_SERVER_HELLO:reason(1112)
Unable to establish SSL connection.

As usual, any help is greatly appreciated!

[shypike]

Isn't this simply a matter of the site having an invalid certificate?

[poizun]

Isn't this simply a matter of the site having an invalid certificate?

So, I'm starting to piece things together. Their site disabled SSLv3 to avoid the POODLE issue, which I thought was pretty common. It seems like perhaps, it's not falling back to SSLv2 correctly.

I'm using FreeNAS and this is the standard current port of sabnzbd. I just used portmaster to install it (and reinstall a few times), but no dice.

Do you know if changing the Switch in sabnzbd's config to SSL type V2 instead of V23 would do anything? From the text in that area, it seems to only apply to the providers...

EDIT:

The Switch didn't do anything. On to exploring more options...

[shypike]

While working on 0.8.0, I added TLS1 as the default protocol for Usenet connections.
I discovered that older Ubuntu versions do not support it (at least not from Python).
When your operating system's Python/OpenSSL combo doens't support TLS1, then you have a problem.
SABnzbd doesn't specify a protocol for RSS and URL-based URL fetches,
so whatever is negotiated between the libraries and the server is OK.