[fixed] Issues with special characters i.e. german umlaut

[alexr]

Hi

I'm running SABnzbd 0.6.14 on Ubuntu 11.10 and ran into issues with the
encoding of some characters. The system locale is set to de_DE.UTF-8

If I configure /etc/default/sabnzbdplus to run under my username
such that sabnzbd will startup automatically I run into the following problem.
If I download a file containing for instance a german umlaut like äüö then
all characters are right as they should be in the webinterface.
Though on the disk they are stored as unknown characters replaced by a '?'.

I know I could run a script which will fix the broken characters but this will
only cure the symptoms and not the source of the problem I think.

The funny thing is, if I start sabnzbd manually as root or as another user using

Code: Select all

sudo service sabnzbdplus start

or

Code: Select all

service sabnzbdplus start

it will run pretty
well and not mixing up with any characters.

Does someone can explain this behavior to me?

Regards
Alex

[shypike]

If I download a file containing for instance a german umlaut like äüö then
all characters are right as they should be in the webinterface.
Though on the disk they are stored as unknown characters replaced by a '?'.

What do you mean "on the disk"?
Do they show up properly in Ubuntu's file manager?
Also, there's a difference between a job name and the ultimate files.
The latter are created by unrar and unrar has a will of its own.
It blindly assumes that the OS supports UTF-8, even if not.

The fact that a normal user account and root give different results, indicates that
something is wrong in your system's setup.

What happens if you manually unpack a rar file containing German characters?

Can you do a little experiment?
Start a Terminal session.
Start python
type this mini-program

Code: Select all

import locale
locale.getdefaultlocale()

[alexr]

What do you mean "on the disk"?
Do they show up properly in Ubuntu's file manager?

I mean the actual filename. In Termin the characters like äöü then
show up as '?'.

It blindly assumes that the OS supports UTF-8, even if not.

As I mentioned before my system is set to de_DE.UTF-8.
But here the results of your little experiment

Code: Select all

Python 2.7.2+ (default, Oct  4 2011, 20:03:08) 
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.getdefaultlocale()
('de_DE', 'UTF-8')

The fact that a normal user account and root give different results, indicates that
something is wrong in your system's setup.

If I run

Code: Select all

service sabnzbdplus start

as a normal user there are no issues with characters.

if I run

Code: Select all

sudo service sabnzbdplus start

as a normal user there are also no issues.

If I change to root with "su root" and run

Code: Select all

service sabnzbdplus start

there are also no issues.

But if I let sabnzbd autostart using the provided init.d script which is configured through /etc/default/sabnzbdplus
to run as a normal user called alex, it will mess up the characters.

What happens if you manually unpack a rar file containing German characters?

All characters are right as they should be.

[jcfp]

Interesting, this suggests a difference between the locale setting for the user accounts and that used (if any) by init on system boot. Could you add the following line to /etc/default/sabnzbdplus, all on a single line of its own (not as part of any setting!):

Code: Select all

/bin/echo -e "***argv***\nCMD=$0 ARGS=$@\n***locale***\n$(locale)\n***env***\n$(env)" > "/tmp/sabinitlog_$(date +%s-%N)"

It will output a file to /tmp each time the init script is run, with rather obvious filenames. I need the output file from a reboot, and another one resulting from calling the init script from the terminal (sudo service sabnzbdplus start). After that, please email the resulting two files to me ([email protected]), and just remove the line again from /etc/default/sabnzbdplus.

Any content in /etc/default/locale ?

[alexr]

Interesting, this suggests a difference between the locale setting for the user accounts and that used (if any) by init on system boot.

Hi

thanks for your reply and you were right. I've mailed you the two resulting files,
which show that on boot (executing the init script) following locale is set

Code: Select all

***locale***
LANG=
LANGUAGE=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

when acting after boot in terminal I get the following

Code: Select all

***locale***
LANG=de_DE.UTF-8
LANGUAGE=
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=

that does explain the problem I've described in the initial post, doesn't it?

Oh and here is the content of /etc/default/locale

Code: Select all

LANG="de_DE.UTF-8"

[jcfp]

Try adding this to /etc/default/sabnzbdplus, and see if it fixes your issue when the program is started on boot:

Code: Select all

[ -z "$LANG" ] && {
	[ -r /etc/default/locale ] && . /etc/default/locale;
	[ -n "$LANG" ] && export LANG;
}

Another question too: is this an older ubuntu version, or an old install upgraded (again and again) to newer releases?

[alexr]

Hi

thanks a lot - this fixed the issue!

Another question too: is this an older ubuntu version, or an old install upgraded (again and again) to newer releases?

No it was a fresh minimal command-line install of Ubuntu 11.10 Server Edition installed with ubuntu-11.10-server-i386.iso.
German keyboard and language were selected during install.

[bastian433]

As of Ubuntu 14.04 LTS and an nl_NL.UTF-8 locale I still need to apply the fix as given by jcfp in the startup script, because it otherwise still uses the wrong locale.

[jcfp]

Only been two years and 5 months... time to commit the fix I guess.
http://anonscm.debian.org/viewvc/python ... hrev=10888

[FabianX2]

So it's nearly 2025, but the behavior is exactly the same as in the post. The content of /etc/default/locale also shows LANG="de_DE.UTF-8".

Sadly, the code @jcfp provided does not fix it for me. My /etc/default/sabnzbdplus file only includes specifications for user, host, and port, along with jcfp's code at the end.

The system is a fresh Ubuntu Server setup. Everything is up to date.

Any ideas?

The share I am writing to is a SMB share wich is mouted with fstab. With the following parameters: //pathtoshare /mountingpoint cifs credentials=pathtocredentials,noperm 0 0

[sander]

Grave digging? Previous post was 10 years old?

Anyway:
Can you provide specifics? And: a link to the .NZB ?

[FabianX2]

I am new here and not shure whats best practis. The logs include lots of personal stuff.

While browsing through the logs this seems relevant:
2024-12-17 08:21:51,968::INFO::[sabnzbdplus:1163] Platform = posix - Linux-6.8.0-51-generic-x86_64-with-glibc2.39
2024-12-17 08:21:51,968::INFO::[sabnzbdplus:1169] Preferred encoding = UTF-8

The content of /etc/default/locale is LANG="de_DE.UTF-8".

Beside that it is a clean Ubuntu Server 24.04.1 Sab is inbstalled with the method displayed here: wiki/installation/install-ubuntu-repo#toc3 As Full install and then started as Service. Its a VM on a TrueNas Scale System.

My /etc/default/sabnzbdplus file only includes specifications for user, host, and port, along with jcfp's code at the end.

The share I am writing to is a SMB share wich is mouted with fstab. With the following parameters: //pathtoshare /mountingpoint cifs credentials=pathtocredentials,noperm 0 0

The share lives on a TureNas Scale machine. Wich supports german umlaut. I can write files form the Ubuntu VM lives on or any other System to the SMB share with Umlaut without any problems.

What else should I provide?

[sander]

> What else should I provide?

The .NZB that causes a problem. Because: I'm assuming you have a problem with certain NZBs with an umlaut in the NZB? Or in the resulting files?

With the .NZB I can verify on my Ubuntu.

FYI:
- Indexers can only handle post names with pure ASCII. So if a poster puts non-ASCII stuff in the name of a post (like umlaut), garbage will result.
- UTF8 in filenames is fine. However, AFAIK: not windows encoding like cp1252 ... strange things may happen, certainly when downloading with Linux/MacOS.

[sander]

PS:

this is a great UTF8 post: https://raw.githubusercontent.com/sande ... %8D%81.nzb

Put that URL into SABnzbd, and tell the result.

On my Ubuntu with ext4 filesystem:

$ ll Downloads/complete/Unicode_Mix_2023_/
total 102412
drwxr--r-- 2 sander sander 4096 Dez 17 10:43 ./
drwxr--r-- 4 sander sander 4096 Dez 17 10:43 ../
-rw-r--r-- 1 sander sander 104857600 Mai 26 2023 Unicode_Mix_2023_∀∁∂∃∄∅∆∇∈∉∊↖♔♕♖♗♘♙♚♛♜♝♞♟🗳♻..bin

So ... all good.


EDIT

And one with German umlauts in the resulting files:
https://raw.githubusercontent.com/sande ... c1c39c.nzb

Result:

sander@haring1:~$ ll Downloads/complete/reftestnzb-German-Umlauts-ea6f31c1c39c/
total 102408
drwxr--r-- 2 sander sander 4096 Dez 17 10:55 ./
drwxr--r-- 6 sander sander 4096 Dez 17 10:55 ../
-rw-r--r-- 1 sander sander 104857600 Okt 6 2019 Heizölrückstoßabdämpfung.bin

... umlauts and ringel-S!

[FabianX2]

Thanks for the help, by the way. Honestly, there is no problem yet. It's just the warning message that makes me nervous:

WARNING::[misc:95] Dateinamen mit Umlauten können nicht in /PFAD gespeichert werden. Dies kann zu Problemen führen.

I did download the Unicode Mix, but I do not know how to check if it's all right. The filename itself contains lots of special characters, which are fine. Trying to open it with the Windows editor results in a crash of the editor.