Web gui crash with https enabled

[nock]

***Resaon: SABconnect++ is the cause of this problem. Let SABconnect++ connect over HTTP, and your problem is gone. No difficult stuff with system settings needed.***


Been running Sabnzbd for years, but switched to Ubuntu for 1 year ago. Since that I've been having thees web GUI "crashes" when using https to connect.
When i run

Code: Select all

sudo service sabnzbdplus status

it says its up and running.. But the web GUI is not accessible.
I have always just done a restart of the service and every thing is fine. This error occur once in awhile, from every day to once a week. Now i'm sick of it and want to fix it.
Checked the error.log, when the error occur it is a bunch of python lines witch ends with this line:

Code: Select all

FatalSSLAlert: [('SSL routines', 'SSL23_WRITE', 'ssl handshake failure')]

Running Ubuntu 14.04
Sabnzbd Version: 0.7.19RC4, same error in 0.7.16


Web browser: Chrome, IE, Firefox. Error occur with every browser..
You can view sabnzbd.error log here: http://pastebin.com/RszkMSeX
You can view the last important lines in sabnzbd.log here: http://pastebin.com/qL9MEuAz
The crashes usually happens early in the morning or night, I only have Sickbeard or Couchpotato doing API calls to Sab, no browser plugins running in the crash period.
From the sabnzbd.log it looks like the crash happens when there is no load, aka noting is downloading.

Checking status:

Code: Select all

einar@Download:~$ sudo service sabnzbdplus status
 * SABnzbd+ binary newsgrabber: running (pid 1353)

When running openssl:

Code: Select all

einar@Download:~$ openssl s_client -connect 127.0.0.1:9090
connect: Connection refused
connect:errno=111

Any help?

-Thx

[jcfp]

Details pls.

error.log [...] a bunch of python lines

...such as?

What browser?
Load of the system when this error occurs?
Any browser plugins or other 3rd party apps doing api calls to sab?
Does the ssl layer still work or not when you try with something like

Code: Select all

openssl s_client -connect 127.0.0.1:9090

[nock]

***Updated first post***

Web browser: Chrome, IE, Firefox. Error occur with every browser..
You can view sabnzbd.error log here: http://pastebin.com/RszkMSeX
You can view the last important lines in sabnzbd.log here: http://pastebin.com/qL9MEuAz
The crashes usually happens early in the morning or night, I only have Sickbeard or Couchpotato doing API calls to Sab, no browser plugins running in the crash period.
From the sabnzbd.log it looks like the crash happens when there is no load, aka noting is downloading.

Checking status:

Code: Select all

einar@Download:~$ sudo service sabnzbdplus status
 * SABnzbd+ binary newsgrabber: running (pid 1353)

When running openssl:

Code: Select all

einar@Download:~$ openssl s_client -connect 127.0.0.1:9090
connect: Connection refused
connect:errno=111

[sander]

"Too many open files"? Hoe many open connections to 8080 and 9090 does " netstat -apon" show before the crash?

[nock]

Is there a log i can view for that?
Not that familiar with "netstat", when i run

Code: Select all

 netstat -apon

i cant really see whats "open connections"
Is it the columns "Recv-Q" and "Send-Q" i should look at?

[nock]

i'm guessing this command is as good as any:

Code: Select all

netstat -an | grep :9090 | wc -l

Right now i have 11 if that maters..

[jcfp]

That open files limit is global, so that could be triggered by other unrelated apps too. But checking for possible hammering of sab certainly makes sense.

Open files of the sab process:

Code: Select all

ls -la /proc/$(pgrep sabnzbdplus)/fd/

Current limit:

Code: Select all

ulimit -n

If it's just too low, set it to something higher (some examples here)

If the issue persists, could you retry the openssl command with a different hostname or ip? Whatever other hostname or ip actually connects will do; current output suggests that sab may simply not be listening on 127.0.0.1.

[nock]

Did a few changes:

Code: Select all

vi /etc/sysctl.conf
fs.file-max = 100000

and

Code: Select all

/etc/security/limits.conf
*               soft    nofile          65000
*               hard    nofile          65000

/etc/sysctl.conf
net.ipv4.ip_local_port_range = 1024 65000

ulimit -n
//old output 1024
//new output 65000

Also changed the host line in sabnzbd.ini and default to my IP-address instead of "0.0.0.0" , should not have an impact but did it anyway..

You can view the output of

Code: Select all

ls -la /proc/$(pgrep sabnzbdplus)/fd/

here: http://pastebin.com/KdfeTx41
The output does not really tell me much :/

Will update on how it goes Thanks for all the answers so far

[nock]

crashed agian..
You can view the output of

Code: Select all

ls -la /proc/$(pgrep sabnzbdplus)/fd/

here: http://pastebin.com/ABBimPRk

[sander]

All those sockets are not good, AFAIK.

See http://wiki.sabnzbd.org/faq#toc33 for the relevant FAQ

You wrote "I only have Sickbeard or Couchpotato doing API calls to Sab, no browser plugins running in the crash period.". If so, Sickbeard or Couchpotato are leaving sessions open. That is an unknown behaviour to me; I only know it with SABconnect++ and HTTPS.

First try, as a woraround: let SB and CP use HTTP, not HTTPS

[nock]

Was just thinking that my self. Will post updates

[nock]

Going from https to http alone did not work for me. I was think if the real problem is that "half-open" connection did not close, i "just force close " them.
Googled around and found a similar issue, but only in other applications like SQL and so on..

Its only been one day, but no crash in the usually crash period.
This setup seems to fix the error:

Decreasing FIN timeout
Added this line to /etc/sysctl.conf

Code: Select all

net.ipv4.tcp_fin_timeout=20

Default is 60.

And enabled the tcp_tw_recycle option

Code: Select all

cat /proc/sys/net/ipv4/tcp_tw_recycle
0

After this change i can see fare less "time_wait" connection hanging to Sab over time.

Code: Select all

netstat -apon | grep -e :8080 -e :9090

Source: http://www.fromdual.com/huge-amount-of- ... onnections

I have to add that Sickbeard and Couchpotato is communication over http (did not change it back to https).

Will post if fails

[sander]

I have to add that Sickbeard, Couchpotato and SabConnect is communication over http (did not change it back to https).

SabConnect? But in your original post you said "I only have Sickbeard or Couchpotato doing API calls to Sab, no browser plugins running in the crash period."?

[nock]

That is true, i'll remove it form the post

[sander]

That is true, i'll remove it form the post

Well, add to your OP.
Resaon: SABconnect++ is the cause of this problem. Let SABconnect++ connect over HTTP, and your problem is gone. No difficult stuff with system settings needed.