[fixed] Issues with special characters i.e. german umlaut

[alexr]

Hi

I'm running SABnzbd 0.6.14 on Ubuntu 11.10 and ran into issues with the
encoding of some characters. The system locale is set to de_DE.UTF-8

If I configure /etc/default/sabnzbdplus to run under my username
such that sabnzbd will startup automatically I run into the following problem.
If I download a file containing for instance a german umlaut like äüö then
all characters are right as they should be in the webinterface.
Though on the disk they are stored as unknown characters replaced by a '?'.

I know I could run a script which will fix the broken characters but this will
only cure the symptoms and not the source of the problem I think.

The funny thing is, if I start sabnzbd manually as root or as another user using

Code: Select all

sudo service sabnzbdplus start

or

Code: Select all

service sabnzbdplus start

it will run pretty
well and not mixing up with any characters.

Does someone can explain this behavior to me?

Regards
Alex

[shypike]

If I download a file containing for instance a german umlaut like äüö then
all characters are right as they should be in the webinterface.
Though on the disk they are stored as unknown characters replaced by a '?'.

What do you mean "on the disk"?
Do they show up properly in Ubuntu's file manager?
Also, there's a difference between a job name and the ultimate files.
The latter are created by unrar and unrar has a will of its own.
It blindly assumes that the OS supports UTF-8, even if not.

The fact that a normal user account and root give different results, indicates that
something is wrong in your system's setup.

What happens if you manually unpack a rar file containing German characters?

Can you do a little experiment?
Start a Terminal session.
Start python
type this mini-program

Code: Select all

import locale
locale.getdefaultlocale()

[alexr]

What do you mean "on the disk"?
Do they show up properly in Ubuntu's file manager?

I mean the actual filename. In Termin the characters like äöü then
show up as '?'.

It blindly assumes that the OS supports UTF-8, even if not.

As I mentioned before my system is set to de_DE.UTF-8.
But here the results of your little experiment

Code: Select all

Python 2.7.2+ (default, Oct  4 2011, 20:03:08) 
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.getdefaultlocale()
('de_DE', 'UTF-8')

The fact that a normal user account and root give different results, indicates that
something is wrong in your system's setup.

If I run

Code: Select all

service sabnzbdplus start

as a normal user there are no issues with characters.

if I run

Code: Select all

sudo service sabnzbdplus start

as a normal user there are also no issues.

If I change to root with "su root" and run

Code: Select all

service sabnzbdplus start

there are also no issues.

But if I let sabnzbd autostart using the provided init.d script which is configured through /etc/default/sabnzbdplus
to run as a normal user called alex, it will mess up the characters.

What happens if you manually unpack a rar file containing German characters?

All characters are right as they should be.

[jcfp]

Interesting, this suggests a difference between the locale setting for the user accounts and that used (if any) by init on system boot. Could you add the following line to /etc/default/sabnzbdplus, all on a single line of its own (not as part of any setting!):

Code: Select all

/bin/echo -e "***argv***\nCMD=$0 ARGS=$@\n***locale***\n$(locale)\n***env***\n$(env)" > "/tmp/sabinitlog_$(date +%s-%N)"

It will output a file to /tmp each time the init script is run, with rather obvious filenames. I need the output file from a reboot, and another one resulting from calling the init script from the terminal (sudo service sabnzbdplus start). After that, please email the resulting two files to me ([email protected]), and just remove the line again from /etc/default/sabnzbdplus.

Any content in /etc/default/locale ?

[alexr]

Interesting, this suggests a difference between the locale setting for the user accounts and that used (if any) by init on system boot.

Hi

thanks for your reply and you were right. I've mailed you the two resulting files,
which show that on boot (executing the init script) following locale is set

Code: Select all

***locale***
LANG=
LANGUAGE=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

when acting after boot in terminal I get the following

Code: Select all

***locale***
LANG=de_DE.UTF-8
LANGUAGE=
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=

that does explain the problem I've described in the initial post, doesn't it?

Oh and here is the content of /etc/default/locale

Code: Select all

LANG="de_DE.UTF-8"

[jcfp]

Try adding this to /etc/default/sabnzbdplus, and see if it fixes your issue when the program is started on boot:

Code: Select all

[ -z "$LANG" ] && {
	[ -r /etc/default/locale ] && . /etc/default/locale;
	[ -n "$LANG" ] && export LANG;
}

Another question too: is this an older ubuntu version, or an old install upgraded (again and again) to newer releases?

[alexr]

Hi

thanks a lot - this fixed the issue!

Another question too: is this an older ubuntu version, or an old install upgraded (again and again) to newer releases?

No it was a fresh minimal command-line install of Ubuntu 11.10 Server Edition installed with ubuntu-11.10-server-i386.iso.
German keyboard and language were selected during install.

[bastian433]

As of Ubuntu 14.04 LTS and an nl_NL.UTF-8 locale I still need to apply the fix as given by jcfp in the startup script, because it otherwise still uses the wrong locale.

[jcfp]

Only been two years and 5 months... time to commit the fix I guess.
http://anonscm.debian.org/viewvc/python ... hrev=10888