Page 1 of 1

"Rename" of _UNPACK_ folder to final folder name does actually copy instead of rename!?!?

Posted: October 5th, 2020, 8:04 am
by Tensai
Hi

I don't know if this is a new bug in v3.x or if this was also already present in v2.x (and I just never noticed it because my downloads usually go completely unattended) but today I have found out that after SABnzbd has finished unpacking the download into the "_UNPACK_xyz" folder, it does not simply rename the folder "_UNPACK_xyz" to "xyz" but it actually does copy the files from "_UNPACK_xyz" to "xyz" and delete them from "_UNPACK_xyz" thereafter. This is incredibly slow and can take even more time as the unpacking process itself. Actually renaming the folder would take only a fraction of a second.
In addition this also might unnecessarily lead to problems with disks running out of space if files are copied and hence need double the disk space. And moreover, during the copy process, SABnzbd seems to be blocked (in my case web gui was stuck for about 15 minutes while SABnzbd was "renaming" a 50GB download).

I wonder why this has not been spotted so far and hope to see this fixed soon.

Regards, Tensai

Re: "Rename" of _UNPACK_ folder to final folder name does actually copy instead of rename!?!?

Posted: October 5th, 2020, 8:13 am
by sander
It is a move / rename ... unless ... see below.
But: Are Incomplete and Complete on the same filesystem / partition?

https://github.com/sabnzbd/sabnzbd/blob ... #L766-L796

https://docs.python.org/3/library/shutil.html tells for move():
If the destination is on the current filesystem, then os.rename() is used. Otherwise, src is copied to dst using copy_function and then removed. In case of symlinks, a new symlink pointing to the target of src will be created in or as dst and src will be removed.

Re: "Rename" of _UNPACK_ folder to final folder name does actually copy instead of rename!?!?

Posted: October 5th, 2020, 8:44 am
by Tensai
Hi Sander

My setup is the following:

SABnzbd is installed in a docker container on my dedicated docker server (Ubuntu 14 LTS).
/tmp is a local volume on a SSD on my docker server and is the temporary download folder for SABnzbd
/downloads is a mounted network share on my Synology NAS and the final download destination folder
SABnzbd does unpack from /tmp/xyz in the folder /downloads/_UNPACK_xyz and then "rename" /downloads/_UNPACK_xyz to /downloads/xyz. But this "rename" is clearly a copy process because on my NAS I can definitely see that both subfolders ("_UNPACK_xyz" and "xyz") exist in parallel during the copy process and the files and folders are successively copied to "xyz". I can even watch how each file copied to "xyz" increases in file size during the copy process ad I can also see hard disk space being eaten up by the copied files.
All this should not happen with an actual rename/move command.

EDIT: as the "rename" happens on the same mount and hence on the same filesystem, this then seems then to be a bug in shutil. os.rename works perfectly on network share mounts and copy/remove is not required in this case.
Anyway, I wonder why SABnzbd doesn't simply use os.rename in the first place.

Regards, Tensai

Re: "Rename" of _UNPACK_ folder to final folder name does actually copy instead of rename!?!?

Posted: October 5th, 2020, 9:21 am
by sander
I did a 10GB download, with all on the same file system / partition, and this is what sabnzbd.log says:

Code: Select all

2020-10-05 16:17:45,704::DEBUG::[filesystem:674] Moving (overwrite: 0) /media/zeegat/incomplete/bla10GB/bla10GB.bin => /media/zeegat/_UNPACK_bla10GB/bla10GB.bin
2020-10-05 16:17:45,704::DEBUG::[filesystem:769] Renaming "/media/zeegat/incomplete/bla10GB/bla10GB.bin" to "/media/zeegat/_UNPACK_bla10GB/bla10GB.bin"
2020-10-05 16:17:45,706::DEBUG::[filesystem:491] Applying permissions 0o775 (octal) to /media/zeegat/_UNPACK_bla10GB
2020-10-05 16:17:45,706::DEBUG::[filesystem:491] Applying permissions 0o664 (octal) to /media/zeegat/_UNPACK_bla10GB/bla10GB.bin
2020-10-05 16:17:45,708::DEBUG::[filesystem:769] Renaming "/media/zeegat/_UNPACK_bla10GB" to "/media/zeegat/bla10GB"
2020-10-05 16:17:45,709::DEBUG::[filesystem:811] [sabnzbd.postproc.rename_and_collapse_folder] Removing dir /media/zeegat/_UNPACK_bla10GB
2020-10-05 16:17:45,709::INFO::[postproc:523] Running deobfuscate

So Moving and then Renaming all winthin one second (within .005 seconds even)

EDIT: And now with Incomplete on another filesystem / partition than Complete:

Code: Select all

2020-10-05 16:48:51,813::DEBUG::[filesystem:674] Moving (overwrite: 0) /media/sander/FD6C-68F5/incomplete/bla10GB/bla10GB.bin => /media/zeegat/_UNPACK_bla10GB.1/bla10GB.bin
2020-10-05 16:48:51,813::DEBUG::[filesystem:769] Renaming "/media/sander/FD6C-68F5/incomplete/bla10GB/bla10GB.bin" to "/media/zeegat/_UNPACK_bla10GB.1/bla10GB.bin"
2020-10-05 16:52:21,748::DEBUG::[filesystem:491] Applying permissions 0o755 (octal) to /media/zeegat/_UNPACK_bla10GB.1
2020-10-05 16:52:21,750::DEBUG::[filesystem:491] Applying permissions 0o644 (octal) to /media/zeegat/_UNPACK_bla10GB.1/bla10GB.bin
2020-10-05 16:52:21,751::DEBUG::[filesystem:769] Renaming "/media/zeegat/_UNPACK_bla10GB.1" to "/media/zeegat/bla10GB.1"
So that takes 3.5 minutes for a 10GB file, which is about 48 MB/s, which is about the disk speed reported by the Wrench symbol test.

Can you show yours? You must set SAB's logging to +Debug

Re: "Rename" of _UNPACK_ folder to final folder name does actually copy instead of rename!?!?

Posted: October 5th, 2020, 11:41 am
by safihre
I don't know why it doesn't work. But we use shutil.move because it will use os.rename if it can.
Seems something specific to your system is causing it to not do that.

Due to a previous bug report I already tried to use os.rename first and only use shutil.move when it errors. But it turned out that shutil.move was always right: when it used copy, using os.rename would not work even if called manually due to some underlying OS problem.

Try running sabnzbd on the host directly.

Re: "Rename" of _UNPACK_ folder to final folder name does actually copy instead of rename!?!?

Posted: October 6th, 2020, 1:50 am
by Tensai
Hi Sander and Safihre

I have found the culprit. It was Sickchill already processing the _UNPACK_ folder and obviously preventing shutil from renaming it with os.rename so shutil fall back to copy/remove. I went through the SABnzbd and Sickchill logs and found out that this obviously happens a lot. But if it does not, shutil correctly uses os.rename. So it is certainly not a bug of SABnzbd or shutil. Sorry for this accusation ;)

But this leads me to another question: Is it possible to define a dedicated unpack folder where the downloads are unpacked into and then renamed/moved to the final destination folder only after unpack/post processing has finished? This would prevent third party scripts/programs like Sickchill to already process the _UNPACK_ folders.

E.g.:
/tmp --> temporary download folder
/media/unpack --> temporary unpack folder
/media/downloads --> final destination folder monitored by third party programs/scripts for further processing

I anyway think it is a very bad idea to directly unpack into the final destination folder and I never was fond of this behavior of SABnzbd.
It therefore would be great to have such a feature to define a temporary unpack folder the same way the temporary download folder can be defined. And like this you wouldn't need to prepend the unpack folder of the download with _UNPACK_ at all.

Regards, Tensai

Re: "Rename" of _UNPACK_ folder to final folder name does actually copy instead of rename!?!?

Posted: October 6th, 2020, 4:03 am
by safihre
Usually tools like Sickbeard and Sonar ignore the _UNPACK_ folders.. Would be good to report at their forums this problem so they can fix it.

Re: "Rename" of _UNPACK_ folder to final folder name does actually copy instead of rename!?!?

Posted: October 6th, 2020, 4:29 am
by Tensai
safihre wrote: October 6th, 2020, 4:03 am Usually tools like Sickbeard and Sonar ignore the _UNPACK_ folders.. Would be good to report at their forums this problem so they can fix it.
Yeah, there is obviously a bug in sickchill which results in sickchill still processing subfolders in the _UNPACK folders if the subfolder themselves do not include the _UNPACK pre- or suffix in the name.

However, this is not the point.
With one simple change in your program you would make all other programs work without having them to relay on some pseudo standards like _UNPACK folders or "nomedia markers". If the unpacking would be done in a temporary folder outside of the final destination folder and the final destination folder would only ever contain the final files, all that programs monitoring a folder for further processing would simply work. It really is as simple as that!

Being a programmer myself I couldn't think of any simpler solution to make your program as compatible as possible. And you always should strive for the simplest solution. Think it over ;)

Regards, Tensai

Re: "Rename" of _UNPACK_ folder to final folder name does actually copy instead of rename!?!?

Posted: October 6th, 2020, 6:27 am
by safihre
This is how all the usenet programs work, so I don't see the need to deviate from that :)

Re: "Rename" of _UNPACK_ folder to final folder name does actually copy instead of rename!?!?

Posted: October 6th, 2020, 7:29 am
by Tensai
safihre wrote: October 6th, 2020, 6:27 am This is how all the usenet programs work, so I don't see the need to deviate from that :)
Yeah I know, unfortunately. NZBGet exhibits the exact same stupid behavior.
But for NZBGet I could at least write a pre-/post-processing script to add a dedicated unpack folder and the script is completely compatible with category folders, etc. and can be directly configured in the GUI.
Sabnzbd doesn't seem to provide the same integration of scripts into the GUI and also does not provide all required config information to the scripts, so I don't think I could port the NZBGet script to sabnzbd with the same functionality.

Anyway, the one who "invented" this bullshit with the "_UNPACK" folder naming should be tarred and feathered because it is such an extremely bad design. Temporary data should always be clearly separated from final data and files currently being extracted are temporary data. And using a temporary unpack folder outside the final destination folder wouldn't break any compatibility with existing programs, but only would make it even more compatible.

I am not going to argue about this any further. But please just keep this in mind if you really want to improve sabnzbd ;)

Thanks for your work anyway and kind regards, Tensai