[Synology NAS] Fixed non-ASCII chars and more...(2013/10/02)

Come up with a useful post-processing script? Share it here!
Post Reply
User avatar
LapinFou
Jr. Member
Jr. Member
Posts: 79
Joined: April 23rd, 2012, 7:35 am
Location: France, Caen

[Synology NAS] Fixed non-ASCII chars and more...(2013/10/02)

Post by LapinFou » April 23rd, 2012, 7:43 am

If you've got some bucks to spend, please give first to SABnzbd and SynoCommunity, then you can think to me :)
Image


Salut les francophones,
Version en Français de ce tutoriel www.nas-forum.com

[Edit 2nd of October 2013 (version 1.9 is out):]
  • Fixed an issue with some NZB and Sickbeard option
  • In order to simplify the support, the script version is now displayed
[/b]
[Edit 19th of February 2013 (version 1.8 is out):]
  • added Sickbear post-processing
  • more evoluate (2 modes) move to a destination folder
  • using CP850 code page instead cp437
[/b]
Hi folks,

As many happy Synology's customers, I'm running a third party SABnzbd client (www.synocommunity.com) on my DS411+.
However there is an painful issue: some NZB contains an archive encoded into a foreign (ISOxxx something) format. So when the archive is unpacked on the NAS (UTF8 encoding style), most of the file containing a foreign character (such as éèçù, etc...) will not be accessible through Samba shares (samba is expecting an UTF8 system files).

This script also provides 4 options:
- Ability to uncompress automatically the .7z archives (Enabled by default).
- Move the SABnzbd folder to a specified destination when the job is done (Disabled by default).
- Added the completed download in the Synology DLNA indexer (Disabled by default).
- Send the job to SickBeard post-processing (Disabled by default).

/!\ Important update announcement!!
Since 2013, the script version 1.9 has been directly included in the SynoCommunity SABnzbd package. Thank to Diaoul !! :)
From a scratch install, you may directly jump to section 4, then the section Adding folder in the DLNA server if you are concerned.
If you want to upgrade your current version (probably from original 1.3 version), you must go to section 2 and overwrites the existing CharTranslator.py script file.
The default script folder for SABnzbd SynoCommunity package is:
/usr/local/sabnzbd/var/scripts
However the script will be installed only if you don't have a SABnzbd already installed ! The information below is valid only for first time SynoCommunity installation.
If you are updating an existing version, then the full tutorial must be followed. Or you can uninstall SABnzbd and do a fresh install from scratch. ;)
For SynoCommunity package users, please stop SABnzbd service before launching the update.
/!\End of update announcement

TIP: You may use the Mertymade Config File Editor to update the CharTranslator.py script.
Launch the apps from DSM, then select Config File Editor from the list, add the following line at he end (example for SynoCommunity package):

Code: Select all

/usr/local/sabnzbd/var/scripts/CharTranslator.py,CharTrans
Click "Save" button, quit then launch again Config File Editor et voilà! The script can be edited from DSM. :)

Here is the solution to fix this issue, a post-processing script must be executed by SABnzbd:
1- On the [url]http://"your_nas_ip":8080/[/url] GUI (Config -> Folder) , you must setup the Post-Processing Scripts Folder folder. In my case /volume1/download/SABnzbdScripts
2- Create into the specified script folder a CharTranslator.py file (don't forget to setup right access to 755). /!\Warning/!\ this file must be Unix format ! So it is better to create it with vi under a SSH connection.
3- Copy/paste the following script:

When you copy/paste the code below do NOT use the SELECT ALL button, otherwise a tabulation will be added at the beginning of each lines (see this post)

Code can be found here (200,000 characters forum limit... ;)):
OneDrive link: CharTranslator.py
Source code: Version 1.9 Update (2013/10/02)


Off course if you are using an another source than SynoCommunity package, don't forget to update the first line: #!/usr/local/python/bin/python -OO
SPK SynoCommunity:

Code: Select all

#!/usr/local/python/bin/python -OO
SPK MertyMade:

Code: Select all

#!/var/packages/pythonutils/target/utils/bin/python -OO
or

Code: Select all

[email protected]/sab2/utils/bin/python -OO
You must check the right path through a Telnet session.

SPK Zebulon:

Code: Select all

#!/usr/local/python26/bin/python -OO
4- Under SABnzbd GUI (Config -> Categories) you must update the default category. A column script is available, please select the CharTranslator.py script and save the modification.
Image

NB: If you want to create the file from a Window platform, you may use the great editor Notepad++. Don't forget to save the file in Unix format (Edit -> EOL Conversion -> Unix).

PS: The script may also be executed as a command line through a Telnet session.
For example: /volume1/download/SABnzbdScripts/CharTranslator.py "folder path which must be fixed"
If you are executing the command from the folder where is stored the script, the command is
./CharTranslator.py "folder path which must be fixed"
/!\: All extra options must be disable before using it this way (Set all options to False)



Adding folder in the DLNA server:
Some people are using directly the SABnzbd download folder as a final destination (starsys member). A indexing command (synoindex) must be added in order to make visible all downloaded stuffs through the DLNA server.
If you are also locking for this feature, please edit CharTranslator.py and enable the option line 68

Code: Select all

IndexInSynoDLNA = True

Sending folder to SickBeard post-processing:
NB: The script sabToSickBeard.py must be installed first.
The CharTranslator.py will parse the config.ini SABnzbd file to get the scripts folder installation. Then it will check that the autoProcessTV module is available.
Enable this option by editing the line 72

Code: Select all

SickBeardPostProcessing = True
Attention: the option IndexInSynoDLNA and SickBeardPostProcessing are exclusive. If you want to have both function at the same time, enable only the option SickBeardPostProcessing in CharTranslator.py, then follow those steps to enable the DLNA indexer in SickBeard:
SickBeard GUI -> Config -> Notifications -> Enable 'Synology Indexer'


Moving folder to an another destination:
Enable this option by specifying a destination folder line 51, for ex:

Code: Select all

MoveToThisFolder = '/volume1/video/News'
Be aware that the specified path is the one you use in a Telnet session.

Change the move mode be editing the line 62; the default value is

Code: Select all

MoveMergeSubFolder = True
  • If MoveMergeSubFolder = True (Move/Replace Windows-like), then equivalent to unix command:
    cp -rf srcFolder destFolder/
    rm -rf srcFolder
    In case of conflict between an already existing sub-folder in the destination folder:
    the destination sub-folder will be merged with source sub-folder (kind of incremental)
  • If MoveMergeSubFolder = False (Move Mac OS X-like), then it is equivalent to the Unix cmmand:
    rm -rf destFolder
    mv -rf srcFolder destFolder
    In case of conflict between an already existing sub-folder in the destination folder:
    the destination sub-folder will be replaced with source sub-folder

Unpack .7z archive:
It is enabled by default. If you want to disable it, please edit the line 76

Code: Select all

Unpack7z = False


Off course, you can cumulate the option, for example the script can execute the following functions:
1- Unpack all .7z archives.
2- Convert all non-ascii characters to UTF-8 format.
3- Move the completed job to a specified destination.
4- Send it to SickBeard post-processing.
or
1- Unpack all .7z archives.
2- Convert all non-ascii characters to UTF-8 format.
3- Move the completed job to a specified destination.
4- Add multimedia file present in the destination folder into the Synology DLNA indexer.



If you want only the .7z feature with SickBeard:
Edit sabToSickBeard.py file and replace with the following code.
For SynoCommunity users the script is located here:
/usr/local/sickbeard/share/SickBeard/autoProcessTV/

Code: Select all

#!/usr/bin/env python

# Author: Nic Wolfe <[email protected]>
# URL: http://code.google.com/p/sickbeard/
#
# This file is part of Sick Beard.
#
# Sick Beard is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# Sick Beard is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
# 
# You should have received a copy of the GNU General Public License
# along with Sick Beard.  If not, see <http://www.gnu.org/licenses/>.


import sys
import os
import subprocess
import autoProcessTV

# scan .7z files and unpack them
def unpack7zFunc(DirName):
	print "Scanning for .7z file(s), then unpack them"
	print "Scanning files..."
	DetectedFiles = False
	for dirname, dirnames, filenames in os.walk(DirName):
		for filename in filenames:
			if (filename[-3:] == ".7z"):
				print "Unpack %s..." %(filename)
				DetectedFiles = True
				try:
					filepath = os.path.join(dirname, filename)
					syno7z_cmd = ['/usr/syno/bin/7z', 'x', '-y', filepath]
					p = subprocess.Popen(syno7z_cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
					out, err = p.communicate()
					if (str(out) != ''):
						print("7z result: " + filepath + " successfully unpacked")
						os.remove(filepath)
						print(filepath + " has been deleted")
					if (str(err) != ''):
						print("7z failed: " + str(err))
				except OSError, e:
					print("Unable to run 7z: "+str(e))
	if DetectedFiles:
		print "Scanning for .7z files Done !"
	else:
		print "No .7z file Detected !"
	return

if len(sys.argv) < 2:
	print "No folder supplied - is this being called from SABnzbd?"
	sys.exit()
else:
	os.chdir(sys.argv[1])
	currentFolder = os.getcwd()
	unpack7zFunc(currentFolder)
	autoProcessTV.processEpisode(currentFolder)
As usual check the 1st line depending of which Pyton SPK your are using.
If you get an error message such as:
-ash: CharTranslator.py: not found
then you file is probably NOT in Unix format, but in Win format. Keep in minds that end of lines are differnent between Unix and Windows.

Et voilà !!
This script is recursive. So any files/directory will be scan and renamed if necessary.
Please, please, keep me inform if some people has got some issue. You can also post a message if you are happy!! ;D
Have fun folks !!
:D

If every things went well, you should get this message:
Character encoding translation done! (More)
Image


Last comment: even if the script failed to detect correctly the right encoding format, modified files will be accessible from Samba share and File Station. In another words, the biggest risk of using this script is to be constraint to rename files through DSM or Samba. In all case, you don't need anymore to use a Telnet connection to fix its.
Last edited by LapinFou on November 23rd, 2015, 8:39 am, edited 70 times in total.
Synology Model: DS-411+
Firmware Version: DSM 5.2-5592
HDD Model: 4xSeagate ST2000DL003 2To - SHR

How to fix foreign accents issue with SABnzbd
Corriger Les Problème D'accents Sous Sabnzbd

User avatar
LapinFou
Jr. Member
Jr. Member
Posts: 79
Joined: April 23rd, 2012, 7:35 am
Location: France, Caen

Re: Fixed foreign accents on Synology NAS running SABnzbd

Post by LapinFou » April 23rd, 2012, 7:58 am

In few words for developers guys, here is the problem explanation:
1- Synology NAS are storing data in UTF8 format.
2- downloaded data & unzip archive will generate ISO-8859-15 (Windows) files/directories
3- unrar archive (when using the built-in command) will generate CP437 (DOS) files/directories

So the script is detecting if there is some CP437 characters, if not it will assume that ISO-8859-15 is used.
Next step: files/directories names will be change to UTF8 format, then a move command will be executed.
::)
Synology Model: DS-411+
Firmware Version: DSM 5.2-5592
HDD Model: 4xSeagate ST2000DL003 2To - SHR

How to fix foreign accents issue with SABnzbd
Corriger Les Problème D'accents Sous Sabnzbd

User avatar
LapinFou
Jr. Member
Jr. Member
Posts: 79
Joined: April 23rd, 2012, 7:35 am
Location: France, Caen

Re: Fixed foreign accents on Synology NAS running SABnzbd

Post by LapinFou » April 24th, 2012, 8:53 am

Major modification of the script -> version 1.2

Code: Select all

# 12-04-24 |   1.2   | Mixed encoding is now supported
#                    | UTF-8 encoding format detected
This new script will detect directory by directory, file by file which encoding style is used, then do the appropriate translation if necessary.
From now there is NO known restriction. The script can be executed on mixed encoding file system.
Last edited by LapinFou on April 29th, 2012, 3:59 pm, edited 1 time in total.
Synology Model: DS-411+
Firmware Version: DSM 5.2-5592
HDD Model: 4xSeagate ST2000DL003 2To - SHR

How to fix foreign accents issue with SABnzbd
Corriger Les Problème D'accents Sous Sabnzbd

User avatar
LapinFou
Jr. Member
Jr. Member
Posts: 79
Joined: April 23rd, 2012, 7:35 am
Location: France, Caen

Re: Fixed foreign accents on Synology NAS running SABnzbd

Post by LapinFou » April 24th, 2012, 3:59 pm

Version 1.3 fixed a stupid typo

Code: Select all

# 12-04-24 |   1.3   | Fixed typo line 57 (test must be OxA0, not 0xA1)
Synology Model: DS-411+
Firmware Version: DSM 5.2-5592
HDD Model: 4xSeagate ST2000DL003 2To - SHR

How to fix foreign accents issue with SABnzbd
Corriger Les Problème D'accents Sous Sabnzbd

User avatar
LapinFou
Jr. Member
Jr. Member
Posts: 79
Joined: April 23rd, 2012, 7:35 am
Location: France, Caen

Re: Fixed foreign accents on Synology NAS running SABnzbd

Post by LapinFou » May 13th, 2012, 3:18 pm

Depending of which Synology package you are using, you must update the first commented line.

Here is a non-exhaustive list of how to update this 1st line:

SynCommunity SPK:

Code: Select all

#!/usr/local/python/bin/python -OO
MertyMade SPK:

Code: Select all

#!/var/packages/pythonutils/target/utils/bin/python -OO
Zebulon SPK:

Code: Select all

#!/usr/local/python26/bin/python -OO
Synology Model: DS-411+
Firmware Version: DSM 5.2-5592
HDD Model: 4xSeagate ST2000DL003 2To - SHR

How to fix foreign accents issue with SABnzbd
Corriger Les Problème D'accents Sous Sabnzbd

User avatar
LapinFou
Jr. Member
Jr. Member
Posts: 79
Joined: April 23rd, 2012, 7:35 am
Location: France, Caen

Re: Fixed foreign accents on Synology NAS running SABnzbd

Post by LapinFou » May 20th, 2012, 4:07 am

No one tried my script??
:'(

Or it is working so well, that nobody need to complain ?? ;)
Synology Model: DS-411+
Firmware Version: DSM 5.2-5592
HDD Model: 4xSeagate ST2000DL003 2To - SHR

How to fix foreign accents issue with SABnzbd
Corriger Les Problème D'accents Sous Sabnzbd

ChrisB3110
Newbie
Newbie
Posts: 1
Joined: May 20th, 2012, 3:22 pm

Re: Fixed foreign accents on Synology NAS running SABnzbd

Post by ChrisB3110 » May 20th, 2012, 3:27 pm

Hello LapinFou,

I tried your script and I am very happy about it!

Works fine, nothing to complain! I have been looking for this for a long time!

Thank you very much!

User avatar
LapinFou
Jr. Member
Jr. Member
Posts: 79
Joined: April 23rd, 2012, 7:35 am
Location: France, Caen

Re: Fixed foreign accents on Synology NAS running SABnzbd

Post by LapinFou » May 21st, 2012, 4:28 am

Hi ChrisB3110,

Thanks for taking some times to answer me. I'm happy to see that the script is working well on your Syno.
By the way, I updated the first post: the script may also be used as a command line.
Synology Model: DS-411+
Firmware Version: DSM 5.2-5592
HDD Model: 4xSeagate ST2000DL003 2To - SHR

How to fix foreign accents issue with SABnzbd
Corriger Les Problème D'accents Sous Sabnzbd

starsys
Newbie
Newbie
Posts: 33
Joined: July 26th, 2011, 4:17 am

Re: Fixed foreign accents on Synology NAS running SABnzbd

Post by starsys » May 21st, 2012, 9:07 am

@LapinFou : Merci pour ton super script. Ca à l’air d’être exactement ce que je recharchais depuis des mois !! Vraiment sympa.

Petite question : j’ai déjà un petit script qui tourne sur la categorie par défaut (script qui index les fichiers téléchargés pour qu’ils soient visibles en UPnP sur la freebox player).
Comment je fais pour mettre ton script aussi en catégorie par défaut ? J’ai pas trouvé comment mettre 2 scripts par défaut.
Dernier service à te demander : pourrais tu mettre à dispo directement le fichier de ton script en .sh ? J’aimerais le tester mais je suis pas chez moi et pas d’accès SSH distant possible pour lancer un vi.
Merci encore pour ton super taf.

starsys
Newbie
Newbie
Posts: 33
Joined: July 26th, 2011, 4:17 am

Re: Fixed foreign accents on Synology NAS running SABnzbd

Post by starsys » May 21st, 2012, 9:45 am

Hello Lapin.
I've tried to create the script with notepad++, changing "saut de ligne" to Unix but it doesn't work.
I get following error :
"Exit(-1) Cannot run script /volume1/video/CharTranslator.py (Plus)"
"Cannot run script /volume1/video/CharTranslator.py"
I cannot change the permissions with SSH (currently no ssh access) but in file station the script has write and read access for group users. I think it's ok.
Can you help ?
Thanks.

starsys
Newbie
Newbie
Posts: 33
Joined: July 26th, 2011, 4:17 am

Re: Fixed foreign accents on Synology NAS running SABnzbd

Post by starsys » May 21st, 2012, 10:44 am

Hello.
In your code, the first line is :
#!/usr/local/python/bin/python -OO
You say : "Off course if you are using an another source as Zebulong package, don't forget to update the first line: /usr/local/python26/bin/python"
And in your 5th post, you say : "Depending of which Synology package you are using, you must update the first commented line.
Zebulon SPK:
#!/usr/local/python26/bin/python -OO"

I use Zebulon python package.
I've tried both configuration, the script refuse to launch.
Can you help ?
Thanks.

User avatar
LapinFou
Jr. Member
Jr. Member
Posts: 79
Joined: April 23rd, 2012, 7:35 am
Location: France, Caen

Re: Fixed foreign accents on Synology NAS running SABnzbd

Post by LapinFou » May 21st, 2012, 10:52 am

I'm at work, so I will give you a quick answer (I will have more time during the evening):
- You really need a Telnet session to determine where is the Python binary, otherwise the script cannot be executed
- When you copy/paste the code below don't use the SELECT ALL button, otherwise a tabulation will be added on each lines
Following code is OK

Code: Select all

#!/data0/applications/python/2.7.2/bin/python -OO
#-*- coding: iso-8859-15 -*-
#
# If a file has been archieved under an ISO-8859 environment and unarchived
# under an UTF8 environment, then you will get an encoding format problem.
[...]
Following code is BAD

Code: Select all

    #!/data0/applications/python/2.7.2/bin/python -OO
    #-*- coding: iso-8859-15 -*-
    #
    # If a file has been archieved under an ISO-8859 environment and unarchived
    # under an UTF8 environment, then you will get an encoding format problem.
[...]
Last edited by LapinFou on May 21st, 2012, 1:35 pm, edited 1 time in total.
Synology Model: DS-411+
Firmware Version: DSM 5.2-5592
HDD Model: 4xSeagate ST2000DL003 2To - SHR

How to fix foreign accents issue with SABnzbd
Corriger Les Problème D'accents Sous Sabnzbd

User avatar
LapinFou
Jr. Member
Jr. Member
Posts: 79
Joined: April 23rd, 2012, 7:35 am
Location: France, Caen

Re: Fixed foreign accents on Synology NAS running SABnzbd

Post by LapinFou » May 21st, 2012, 10:59 am

starsys wrote:@LapinFou : Merci pour ton super script. Ca à l’air d’être exactement ce que je recharchais depuis des mois !! Vraiment sympa.

Petite question : j’ai déjà un petit script qui tourne sur la categorie par défaut (script qui index les fichiers téléchargés pour qu’ils soient visibles en UPnP sur la freebox player).
Comment je fais pour mettre ton script aussi en catégorie par défaut ? J’ai pas trouvé comment mettre 2 scripts par défaut.
Dernier service à te demander : pourrais tu mettre à dispo directement le fichier de ton script en .sh ? J’aimerais le tester mais je suis pas chez moi et pas d’accès SSH distant possible pour lancer un vi.
Merci encore pour ton super taf.
Zut, j'avais pas vu que tu causais la langue de Molière... ;)
Pour le coup, je ne sais pas comment exécuter 2 scripts par défaut. Il faut se renseigner. Sinon il faut fusionner les 2 fichiers.
Mon script ne peut pas être converti simplement en SH, car j'utilise des fonctions bien spécifique. Désolé...
Sinon comme expliqué ci-dessus, il faut vraiment que tu te connectes en telnet. Il y a tellement de version différente de SABnzbd sous Syno que je ne connais pas tout les emplacements.
Sinon je te conseille d'aller voir ici: http://www.nas-forum.com/forum/forum/28-tutorials/
De plus si tu es inscrit, tu peux télécharger directement le fichier ZIP.
Synology Model: DS-411+
Firmware Version: DSM 5.2-5592
HDD Model: 4xSeagate ST2000DL003 2To - SHR

How to fix foreign accents issue with SABnzbd
Corriger Les Problème D'accents Sous Sabnzbd

starsys
Newbie
Newbie
Posts: 33
Joined: July 26th, 2011, 4:17 am

Re: Fixed foreign accents on Synology NAS running SABnzbd

Post by starsys » May 21st, 2012, 1:25 pm

Merci pour ton aide. J'ai réussi à faire fonctionner ton script, et il est vraiment top. Ça marche impeccable. C'était bien un problème de copier coller. Pour la fusion de scripts, sais tu si mes 2 lignes de codes peuvent être interprétées par le langage python de ton script ? Merci en tout cas.

User avatar
LapinFou
Jr. Member
Jr. Member
Posts: 79
Joined: April 23rd, 2012, 7:35 am
Location: France, Caen

Re: Fixed foreign accents on Synology NAS running SABnzbd

Post by LapinFou » May 21st, 2012, 1:41 pm

Glad to see that it is working.
Concerning the ability to launch 2 post-script files by default: I guess the solution will be to create a .sh file which will call both post-script files. First launch the renaming script, then the indexing one. I don't have time today, bu I will think about it this week-end....
Could you give the URL where I can findyour indexing script??

PS: I switched back to English, since we aren't in the French forum
Synology Model: DS-411+
Firmware Version: DSM 5.2-5592
HDD Model: 4xSeagate ST2000DL003 2To - SHR

How to fix foreign accents issue with SABnzbd
Corriger Les Problème D'accents Sous Sabnzbd

Post Reply