Page 1 of 1

Web gui crash with https enabled

Posted: October 29th, 2014, 3:14 pm
by nock
***Resaon: SABconnect++ is the cause of this problem. Let SABconnect++ connect over HTTP, and your problem is gone. No difficult stuff with system settings needed.***


Been running Sabnzbd for years, but switched to Ubuntu for 1 year ago. Since that I've been having thees web GUI "crashes" when using https to connect.
When i run

Code: Select all

sudo service sabnzbdplus status
it says its up and running.. But the web GUI is not accessible.
I have always just done a restart of the service and every thing is fine. This error occur once in awhile, from every day to once a week. Now i'm sick of it and want to fix it.
Checked the error.log, when the error occur it is a bunch of python lines witch ends with this line:

Code: Select all

FatalSSLAlert: [('SSL routines', 'SSL23_WRITE', 'ssl handshake failure')]
Running Ubuntu 14.04
Sabnzbd Version: 0.7.19RC4, same error in 0.7.16


Web browser: Chrome, IE, Firefox. Error occur with every browser..
You can view sabnzbd.error log here: http://pastebin.com/RszkMSeX
You can view the last important lines in sabnzbd.log here: http://pastebin.com/qL9MEuAz
The crashes usually happens early in the morning or night, I only have Sickbeard or Couchpotato doing API calls to Sab, no browser plugins running in the crash period.
From the sabnzbd.log it looks like the crash happens when there is no load, aka noting is downloading.

Checking status:

Code: Select all

einar@Download:~$ sudo service sabnzbdplus status
 * SABnzbd+ binary newsgrabber: running (pid 1353)

When running openssl:

Code: Select all

einar@Download:~$ openssl s_client -connect 127.0.0.1:9090
connect: Connection refused
connect:errno=111
Any help?

-Thx

Re: Web gui crash with https enabled

Posted: October 30th, 2014, 6:52 am
by jcfp
Details pls.
nock wrote:error.log [...] a bunch of python lines
...such as?

What browser?
Load of the system when this error occurs?
Any browser plugins or other 3rd party apps doing api calls to sab?
Does the ssl layer still work or not when you try with something like

Code: Select all

openssl s_client -connect 127.0.0.1:9090

Re: Web gui crash with https enabled

Posted: October 31st, 2014, 10:26 am
by nock
***Updated first post***

Web browser: Chrome, IE, Firefox. Error occur with every browser..
You can view sabnzbd.error log here: http://pastebin.com/RszkMSeX
You can view the last important lines in sabnzbd.log here: http://pastebin.com/qL9MEuAz
The crashes usually happens early in the morning or night, I only have Sickbeard or Couchpotato doing API calls to Sab, no browser plugins running in the crash period.
From the sabnzbd.log it looks like the crash happens when there is no load, aka noting is downloading.

Checking status:

Code: Select all

einar@Download:~$ sudo service sabnzbdplus status
 * SABnzbd+ binary newsgrabber: running (pid 1353)
When running openssl:

Code: Select all

einar@Download:~$ openssl s_client -connect 127.0.0.1:9090
connect: Connection refused
connect:errno=111

Re: Web gui crash with https enabled

Posted: October 31st, 2014, 10:53 am
by sander
"Too many open files"? Hoe many open connections to 8080 and 9090 does " netstat -apon" show before the crash?

Re: Web gui crash with https enabled

Posted: October 31st, 2014, 11:15 am
by nock
Is there a log i can view for that?
Not that familiar with "netstat", when i run

Code: Select all

 netstat -apon
i cant really see whats "open connections"
Is it the columns "Recv-Q" and "Send-Q" i should look at?

Re: Web gui crash with https enabled

Posted: October 31st, 2014, 11:32 am
by nock
i'm guessing this command is as good as any:

Code: Select all

netstat -an | grep :9090 | wc -l
Right now i have 11 if that maters..

Re: Web gui crash with https enabled

Posted: October 31st, 2014, 12:42 pm
by jcfp
That open files limit is global, so that could be triggered by other unrelated apps too. But checking for possible hammering of sab certainly makes sense.

Open files of the sab process:

Code: Select all

ls -la /proc/$(pgrep sabnzbdplus)/fd/
Current limit:

Code: Select all

ulimit -n
If it's just too low, set it to something higher (some examples here)

If the issue persists, could you retry the openssl command with a different hostname or ip? Whatever other hostname or ip actually connects will do; current output suggests that sab may simply not be listening on 127.0.0.1.

Re: Web gui crash with https enabled

Posted: October 31st, 2014, 1:20 pm
by nock
Did a few changes:

Code: Select all

vi /etc/sysctl.conf
fs.file-max = 100000
and

Code: Select all

/etc/security/limits.conf
*               soft    nofile          65000
*               hard    nofile          65000

/etc/sysctl.conf
net.ipv4.ip_local_port_range = 1024 65000

ulimit -n
//old output 1024
//new output 65000
Also changed the host line in sabnzbd.ini and default to my IP-address instead of "0.0.0.0" , should not have an impact but did it anyway..

You can view the output of

Code: Select all

ls -la /proc/$(pgrep sabnzbdplus)/fd/
here: http://pastebin.com/KdfeTx41
The output does not really tell me much :/

Will update on how it goes :) Thanks for all the answers so far :D

Re: Web gui crash with https enabled

Posted: November 2nd, 2014, 10:54 am
by nock
crashed agian..
You can view the output of

Code: Select all

ls -la /proc/$(pgrep sabnzbdplus)/fd/
here: http://pastebin.com/ABBimPRk

Re: Web gui crash with https enabled

Posted: November 2nd, 2014, 2:01 pm
by sander
All those sockets are not good, AFAIK.

See http://wiki.sabnzbd.org/faq#toc33 for the relevant FAQ

You wrote "I only have Sickbeard or Couchpotato doing API calls to Sab, no browser plugins running in the crash period.". If so, Sickbeard or Couchpotato are leaving sessions open. That is an unknown behaviour to me; I only know it with SABconnect++ and HTTPS.

First try, as a woraround: let SB and CP use HTTP, not HTTPS

Re: Web gui crash with https enabled

Posted: November 2nd, 2014, 3:57 pm
by nock
Was just thinking that my self. Will post updates

Re: Web gui crash with https enabled

Posted: November 4th, 2014, 12:37 pm
by nock
Going from https to http alone did not work for me. I was think if the real problem is that "half-open" connection did not close, i "just force close " them.
Googled around and found a similar issue, but only in other applications like SQL and so on..

Its only been one day, but no crash in the usually crash period.
This setup seems to fix the error:

Decreasing FIN timeout
Added this line to /etc/sysctl.conf

Code: Select all

net.ipv4.tcp_fin_timeout=20
Default is 60.

And enabled the tcp_tw_recycle option

Code: Select all

cat /proc/sys/net/ipv4/tcp_tw_recycle
0
After this change i can see fare less "time_wait" connection hanging to Sab over time.

Code: Select all

netstat -apon | grep -e :8080 -e :9090

Source: http://www.fromdual.com/huge-amount-of- ... onnections

I have to add that Sickbeard and Couchpotato is communication over http (did not change it back to https).

Will post if fails

Re: Web gui crash with https enabled

Posted: November 4th, 2014, 12:44 pm
by sander
nock wrote:
I have to add that Sickbeard, Couchpotato and SabConnect is communication over http (did not change it back to https).
SabConnect? But in your original post you said "I only have Sickbeard or Couchpotato doing API calls to Sab, no browser plugins running in the crash period."?

Re: Web gui crash with https enabled

Posted: November 4th, 2014, 3:07 pm
by nock
That is true, i'll remove it form the post

Re: Web gui crash with https enabled

Posted: November 4th, 2014, 5:08 pm
by sander
nock wrote:That is true, i'll remove it form the post
Well, add to your OP.
Resaon: SABconnect++ is the cause of this problem. Let SABconnect++ connect over HTTP, and your problem is gone. No difficult stuff with system settings needed.