template inconsistency in 0.4.0

[eydaimon]

I'm the port maintainer for FreeBSD. The interfaces get installed as per FreeBSD architecture and are not stored in a subdirectory to that of the binary.

As a result, I have to patch SABnzbd.py which has been working fine up until 0.4.0 with templates. However, using this patch:

Code: Select all

--- SABnzbd.py.orig     2008-07-05 15:35:03.000000000 -0700
+++ SABnzbd.py  2008-07-05 15:35:11.000000000 -0700
@@ -301,7 +301,7 @@
     sabnzbd.MY_FULLNAME = os.path.normpath(os.path.abspath(sys.argv[0]))
     sabnzbd.MY_NAME = os.path.basename(sabnzbd.MY_FULLNAME)
     sabnzbd.DIR_PROG = os.path.dirname(sabnzbd.MY_FULLNAME)
-    sabnzbd.DIR_INTERFACES = real_path(sabnzbd.DIR_PROG, DEF_INTERFACES)
+    sabnzbd.DIR_INTERFACES = real_path(%%PREFIX%%, DEF_INTERFACES)

     # Need console logging for SABnzbd.py and SABnzbd-console.exe
     consoleLogging = (not hasattr(sys, "frozen")) or (sabnzbd.MY_NAME.lower().find('-console') > 0)

I get this error:

Page handler: 'No such file or directory'
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/site-packages/cherrypy/_cphttptools.py", line 110, in _run
    applyFilters('before_finalize')
  File "/usr/local/lib/python2.5/site-packages/cherrypy/filters/__init__.py", line 151, in applyFilters
    method()
  File "/usr/local/lib/python2.5/site-packages/sabnzbd/interface.py", line 210, in beforeFinalize
    **rsrc.callable_kwargs)
  File "/usr/local/lib/python2.5/site-packages/sabnzbd/interface.py", line 731, in index
    return template.respond()
  File "cheetah__usr_local_share_sabnzbdplus_interfaces_Default_templates_queue_tmpl_1215299252_47_99950.py", line 98, in respond
  File "/usr/local/lib/python2.5/site-packages/Cheetah-2.0.1-py2.5-freebsd-7.0-STABLE-i386.egg/Cheetah/Template.py", line 1512, in _handleCheetahInclude
    nestedTemplateClass = compiler.compile(source=source,file=file)
  File "/usr/local/lib/python2.5/site-packages/Cheetah-2.0.1-py2.5-freebsd-7.0-STABLE-i386.egg/Cheetah/Template.py", line 693, in compile
    fileHash = str(hash(file))+str(os.path.getmtime(file))
  File "/usr/local/lib/python2.5/posixpath.py", line 143, in getmtime
    return os.stat(filename).st_mtime
OSError: [Errno 2] No such file or directory: '/usr/local/bin/interfaces/Default/templates/inc_top.tmpl'

Note that all I'm doing in the patch is to change the path to sabnzbd.DIR_INTERFACES.  (%%PREFIX%% gets resolved to /usr/local/share/sabnzbdplus).

However, in the "No such file or directory" error above, it's clear that the interface path is not being honored while looking for the template. The  initial queue.tmpl is resolved fine, but not sure about after that.

What do I need to do for the interfaces directory to be honored? I've tried to find how the includes work for templates, but have been unable to find out how the paths are resolved.

Help is greatly appriciated.

[eydaimon]

Seems that the #include directive looks from where the binary is called. I looked at the cheetah #include documentation but couldn't figure out how to include templates from a directory other than the dir it's currently in, which seems absurd. Maybe I'm overlooking something.

It seems that if the absolute path is specified, it does work. I couldn't figure out how to give the path as a variable. Maybe someone here knows?

I can create patch files for each template file which includes the appropriate directory, but that's very tedious, and I'd rather not. Plus, it seems to be a design issue if the interface dir is not honored correctly anyway.

[eydaimon]

Just as a little side-note. bookmark feature has to be a new favorite. awesome job

[shypike]

It took me a while to find out.
Cheetah use the program's current directory as a starting point.

Only when you pre-compile the templates to python programs, you can use
compiler directives to tell where the include files are.
This is not an option for you as it would require substantial changes to the SABnzbd code.

When used dynamically, you're stuck with specifying paths relative to the program directory.

BTW: I see no reason to separate the skin templates from the program code.
They are 100% linked and the SABnzbd code also expects the templates to be in
the "interfaces" subdirectory of the program directory.
Can you explain, why freeBSD would require this?
The fact that it is possible to add skins, does not alter the fact that the code and the skins
are one product.

I appreciate the effort you are willing to put into porting SABnzbd.
However, I would prefer porters not to change the code unless absolutely necessary.
And in this case, I see no reason for it.
Remember that we will get the support calls, not you.

[jcfp]

Shypike, putting files in various locations according to their function is the common way of doing things on linux & unix. Having a package create /usr/bin/interfaces/etc... would be considered very bad manners if not outright unacceptable.

Anyway, I myself also ran into this small inconvenience when packaging rc5, the end result being:

Code: Select all

diff -urNad sabnzbdplus-0.4.0~rc5~/SABnzbd.py sabnzbdplus-0.4.0~rc5/SABnzbd.py
--- sabnzbdplus-0.4.0~rc5~/SABnzbd.py   2008-06-30 23:09:49.000000000 +0200
+++ sabnzbdplus-0.4.0~rc5/SABnzbd.py    2008-07-02 14:20:21.000000000 +0200
@@ -684,7 +684,7 @@

     log_dir = dir_setup(cfg, 'log_dir', sabnzbd.DIR_LCLDATA, DEF_LOG_DIR)

-    os.chdir(sabnzbd.DIR_PROG)
+    os.chdir( os.path.join(sabnzbd.DIR_PROG, '../share/sabnzbdplus') )

     web_dir  = Web_Template('web_dir',  DEF_STDINTF,  web_dir)
     web_dir2 = Web_Template('web_dir2', '', web_dir2)

Now this was just a quick fix; it would be nicer if the os.chdir used a directory based on/relative to the interfaces directory variable. That would work for a "just-unzip-it" install, as well as for all others.

[shypike]

I simply don't see a problem.

I'd say put the sources in /usr/local/src/sabnzbd and make a symlink in /usr/local/bin.
ln -s /usr/local/src/sabnzbd/SABnzbd.py /usr/local/bin/sabnzbd

The "interfaces" folder should be a sub folder of the program folder.
/usr/local/src/sabnzbd
/usr/local/src/sabnzbd/interfaces
/usr/local/src/sabnzbd/sabnzbd
/usr/local/src/sabnzbd/cherrypy

The WebMin package installs in a similar way.

Maybe /usr/share is better, but you probably know better what to choose.

BTW when I browsed through your Ubuntu package I noticed that you split the Python software
and the skins. Why? This is asking for trouble.
The Python files and the skins are ONE product, there is no good reason to split them.

[shypike]

And that's what I object too, the design does not support it.
Also, you cannot just combine arbitrary versions of the Python code and the skins.
There's no separation of the two through a well-defined and stable API.
Maybe one day, but not now.

[eydaimon]

BTW: I see no reason to separate the skin templates from the program code.
They are 100% linked and the SABnzbd code also expects the templates to be in
the "interfaces" subdirectory of the program directory.

It's exactly what jcfp says. Putting non-binary-executable code in a binary directory is simply not acceptable. Take a look at /usr/bin or /usr/local/bin on any unix system, and there will be nothing in that directory besides binaries. Supporting files go in other directories as appropriate.

Just because the location of the binary and the template files are separate does not mean the architecture promotes mixing and matching of binary/templates as you seem to have the impression of (correct me if I'm wrong please).  The two are still coupled.

You would no more put binary files in a directory called 'config_files' than you would put template files in a directory named 'bin'. I hope that's a clear picture of how the architecture works.

Now this was just a quick fix; it would be nicer if the os.chdir used a directory based on/relative to the interfaces directory variable. That would work for a "just-unzip-it" install, as well as for all others.

The quick fix worked well for me too, thanks!

[jcfp]

And that's what I object too, the design does not support it.
Also, you cannot just combine arbitrary versions of the Python code and the skins.
There's no separation of the two through a well-defined and stable API.
Maybe one day, but not now.

One thing at a time pls. The interfaces being installed in another location than directly relative to the executable doesn't mean that arbitrary versions of the Python code and the skins are being combined. In fact, the control of this via packaging is very tight, with every skins having a hard dependency on the exactly matching version of the main package.

As to the reason why they are split: ubuntu/debian strongly dislike packages that consist mostly of "shared data" (e.g. the stuff that belongs in /usr/share or similar places) when it is possible to split that off. And in this case, it sure is: we all know adding or removing skins can be done at will by adding or deleting them in the interfaces directory (regardless of where that resides). Check for example how many a game is packaged in two parts, one small main package, and a -data containing stuff like levels, terrain, etc. Even if these don't even function without the other (!).

And last but not least, yes, putting all in one place with a symlink from /usr/bin is indeed possible. But even so it's not preferred, as simple as that: whether one moves the shared data to the bin directory or the binary the other way...

[shypike]

I did not ask to put the whole sabnzbd structure in /usr/bin.
If I look at how other packages are installed (like Webmin and Python), I'd say put the folder structure
in /usr/share or another appropriate location and just provide a symlink to the one executable file SABnzbnd.py.
So the only thing in the path would be /usr/bin/sabnzbd being a symlink to /usr/share/sabnzbd/SABnzbd.py.

I understand that trying to second-guess a program designer's intentions isn't always easy.
That's why I have just created a Wiki page with our point of view on the matter.
http://sabnzbd.wikidot.com/unix-packaging
I welcome improvement suggestions.

My writing style tends to be rather direct. Please don't mistake this for lack of appreciation
for the effort you put in packaging.
It isn't always pleasant to see your (brain) child being ripped apart by other people, however
good their intentions.

[eydaimon]

I did not ask to put the whole sabnzbd structure in /usr/bin.
If I look at how other packages are installed (like Webmin and Python), I'd say put the folder structure
in /usr/share or another appropriate location and just provide a symlink to the one executable file SABnzbnd.py.
So the only thing in the path would be /usr/bin/sabnzbd being a symlink to /usr/share/sabnzbd/SABnzbd.py.

I understand that trying to second-guess a program designer's intentions isn't always easy.
That's why I have just created a Wiki page with our point of view on the matter.
http://sabnzbd.wikidot.com/unix-packaging
I welcome improvement suggestions.

My writing style tends to be rather direct. Please don't mistake this for lack of appreciation
for the effort you put in packaging.
It isn't always pleasant to see your (brain) child being ripped apart by other people, however
good their intentions.

Noone is trying to second guess your work. All I'm trying to do is to conform to my operating systems architecture. If I don't, then the package will not be accepted, and there will be no port for sabnzbd on my OS. It's not necessarily up to us port maintainers to pick and choose how things get installed, and even if it were up to me, if there's a particular architecture, then I stick to it.

As I said, binary files do not belong in folders for configuration files, and vice versa. That's simply how it is.  Your solution of using a symbolic link I will try next time I submit the port.  In my own install of FreeBSD, I have 81 symlinks in the /usr/local/bin folder, and ONLY 1 of them have a binary in /usr/local/share so it may be acceptable.

Believe me, we're not trying to be difficult on purpose. Thanks for your help on working this out and of course for continuing great work with sabnzbd

[shypike]

Separating the code from the templates creates new problems,
How do we make sure that SABnzbd can find the templates without adding
new hard-coded paths?

The suggestion ../share/sabnzbdplus/interfaces is not acceptable (non-portable and ugly too).
Isn't putting a symlink "interfaces" in the program folder better?
This way you can seperate the templates and keep the Python code as is.

Also keep in mind that we support Windows too and there this typical Linux
separation is not a custom.
Please, refrain from telling me why Linux is better than Windows 

[jcfp]

I did not ask to put the whole sabnzbd structure in /usr/bin.
If I look at how other packages are installed (like Webmin and Python), I'd say put the folder structure
in /usr/share or another appropriate location and just provide a symlink to the one executable file SABnzbnd.py.
So the only thing in the path would be /usr/bin/sabnzbd being a symlink to /usr/share/sabnzbd/SABnzbd.py.

I think enough has been said about the merits of that solution. Btw, no webmin package exists in debian or ubuntu, and the python package only symlinks to other executables in /usr/bin.

I understand that trying to second-guess a program designer's intentions isn't always easy.
That's why I have just created a Wiki page with our point of view on the matter.
http://sabnzbd.wikidot.com/unix-packaging
I welcome improvement suggestions.

As you say, it's a point of view, an opinion. If I were to suggest an improvement, I'd say: stick to strictly facts, and (re)write it in a neutral and unbiased manner. It's fine to have a preference as a designer, but clearly denote it as such, and provide full disclosure. Certainly some real caveats do exist, and pointing them out, esp. to those new to the project and unfamiliar with it's code can be very useful.

How do we make sure that SABnzbd can find the templates without adding
new hard-coded paths?

Now that's the good part, you don't have to . A hard-coded path to tell sabnzbd where to find its templates is already in the program, to be precise, in constants.py around line 63. All you have to do, is use that to point to the interfaces (that's what it's there for in the first place, isn't it?), and subsequently modify these #include statements to work from that rather than DIR_PROG. Anybody that wants to put the interfaces elsewhere simply patches this one line in constants.py, from (relative path) "interfaces" to the (once again, relative) path of his choice. Been that way in the Ubuntu packaging since 0.3.0.

[eydaimon]

[15:01] So, what's the way out?
[15:01] Seprating the templates from the code is awkward.

I honestly think the best solution is to use something like setuptools, or distutils  as SABnzbd used to have, since it's OS agnostic.  Reading that documentation I'm not convinced it's for libraries only, but for  any package distribution. The documentation covers  installing scripts/applications.

Failing that, I can try /usr/local/share/ for the full install and sym-linking, but if that doesn't work, I'll be stuck patching the source to make it work.   Obviously jcfp's latest post before this one should be considered too, maybe in combination with setup.py

Nice chat on IRC! See you again

[jcfp]

Failing that, I can try /usr/local/share/ for the full install and sym-linking, but if that doesn't work, I'll be stuck patching the source to make it work.   Obviously jcfp's latest post before this one should be considered too, maybe in combination with setup.py

On second thought, disregard my suggestion of using the stuff in constants.py - that might just lead to yet more issues in the future. I'll change things over to putting the relevant python stuff under /usr/share (and use a script in /usr/bin to start the program).