Page 1 of 2

unpack ; unrar and then move the file

Posted: February 15th, 2013, 5:50 am
by shpapy
Hello,
i have a dedicated machine which runs sabnzb; the machine has a 500Gb but my final complete folder sits on a NAS server.
currently when the sab works he downloads locally checks the release and then unpacks over the network. i would like to unpack locally and then transfer the file to the network.

can it be done? and how please

thanks a lot for the help.

Re: unpack ; unrar and then move the file

Posted: February 15th, 2013, 9:26 am
by sander
Easy: point the Complete directory to a local directory, and in Category assign your realy destination directory to the Default category ...

Re: unpack ; unrar and then move the file

Posted: November 18th, 2016, 10:44 am
by pxr5
Hi. Can someone explain this a little more clearly please. I don't mind unpacking across the network, but since 1.1.1 after the unpack and going onto the move phase, I appear to be getting a long period of read/writes across the network. Previously it would unpack and do the move instantly. Now, after the file has been unpacked and moved across to my NAS a lot of activity seems to carry on from the NAS and back again.

So what it's doing now is:

The download at 10 MB/s
Unpack to NAS at 90 MB/s
Move is a up/down at 14 MB/s

This makes the whole process nearly as long again as the original download. This never happened on any build before 1.1.1 and I have unchecked "Check result of unpacking", but that makes no difference.

So I was thinking to get round this, Unpack to the local drive, then move to the NAS. The explanation above doesn't seem clear to me.

Thank you.

Re: unpack ; unrar and then move the file

Posted: November 19th, 2016, 7:34 am
by pxr5
Right. I've been looking at this some more, watching the folders as the process completes. As I said above, the files unpack fine across the network and the file completes and is in a folder on the NAS finished. But then during the move phase further read/writes take place across the newtwork. The complete folder on the NAS has the finished file inside, also the TEMP folder on the NAS has the finished file inside as this move operation takes place (which as I undestand should take place very quickly on the final NAS folder). The incomplete/admin folder has a few smaller files inside. Eventually the final file ends up in the correct destination as per the sorting and all other files disappear.

So, in a network graphical way it does this (d=download/repair, u=unpack, m=move), now it's really slowing down the whole process:

dddddddddddddddddd-uuu-mmmmmmmmmm

Before 1.1.1 the move phase was almost instaneous:

ddddddddddddddddd-uuu-m

Leaving the download to complete to local drive (complete folder) works correctly - if I can't fix this I'm going to have to manually copy files from the local folder to the NAS.

Re: unpack ; unrar and then move the file

Posted: November 19th, 2016, 8:55 am
by safihre
Odd, nothing changed regarding the moving between 1.1.0 and 1.1.1 as far as I know.
So if you now downgrade to 1.1.0, you can confirm its faster again?

Re: unpack ; unrar and then move the file

Posted: November 20th, 2016, 6:00 am
by pxr5
OK. Went back to 1.1.0 and everything was OK. Tried 1.1.1 and the issue came back. 1.1.0 again and now OK. I noticed in the final destination folder on 1.1.0 there is one folder created (a temporary UNPACK folder), on 1.1.1 there are two folders (a temporry UNPACK folder and a 'real' folder) - all of which disppeared at the end leaving the file in the right place.

In this network graph the one in the middle is 1.1.1, either side are 1.1.0 (red=download, green=unpack, yellow=move (a mix of up/download across the network on 1.1.1).

Image

I'll stick with 1.1.0 for now.

Re: unpack ; unrar and then move the file

Posted: November 20th, 2016, 6:35 am
by safihre
So weird.. I checked all the code but I have no clue what's possibly changed since 1.1.0.

To make make sure I try to reproduce correctly:
Your temporary folder is on the pc, the final is on the NAS? Is there also a category specific folder active?
So green in your graph is unpack to the NAS and then during the yellow the status in Sab is 'Moving' still?

Re: unpack ; unrar and then move the file

Posted: November 20th, 2016, 8:04 am
by pxr5
safihre wrote:So weird.. I checked all the code but I have no clue what's possibly changed since 1.1.0.

To make make sure I try to reproduce correctly:
Your temporary folder is on the pc, the final is on the NAS? Yes Is there also a category specific folder active? Yes
So green in your graph is unpack to the NAS and then during the yellow the status in Sab is 'Moving' still? Yes
Here is an extract from the log. There is a big delay after the unpack on 1.1.1, but nothing is showing as occuring in this period:

1.1.0
10GB file
2016-11-12 09:06:38,151::INFO::[postproc:371] unpack_magic finished on xxxxxxxxxxxxxxxx
2016-11-12 09:06:38,250::INFO::[postproc:564] Cleaning up xxxxxxxxxxxxxx (keep_basic=False)


1.1.1
5GB file
2016-11-18 14:45:37,552::INFO::[postproc:371] unpack_magic finished on yyyyyyyyyyyyyyyyy
2016-11-18 14:51:23,260::INFO::[postproc:564] Cleaning up yyyyyyyyyyyyyyy (keep_basic=False)

340MB file
2016-11-19 12:18:48,164::INFO::[postproc:371] unpack_magic finished on zzzzzzzzzzzzzz
2016-11-19 12:19:10,282::INFO::[postproc:564] Cleaning up zzzzzzzzzzzzz(keep_basic=False)

Re: unpack ; unrar and then move the file

Posted: November 25th, 2016, 4:22 am
by pxr5
safihre wrote:So weird.. I checked all the code but I have no clue what's possibly changed since 1.1.0.

To make make sure I try to reproduce correctly:
Your temporary folder is on the pc, the final is on the NAS? Is there also a category specific folder active?
So green in your graph is unpack to the NAS and then during the yellow the status in Sab is 'Moving' still?
safihre - thanks, did you manage to reproduce this as I've remained on 1.1.0.

Re: unpack ; unrar and then move the file

Posted: November 25th, 2016, 4:28 am
by safihre
I have send you 3 PM's last week because I need more information! But you didn't read them yet ;)

Re: unpack ; unrar and then move the file

Posted: November 25th, 2016, 5:34 am
by mcboss86
Can you maybe post some further information?

SabNzbd is doing the same for me (1.1.1). My default download folder is on C:, my movies category points towards a network drive on a NAS. I saw it happen today and was looking a bit more closely.
Basically SabNZB is unpacking from C: to Network drive (Command line RAR has a network usage in the task manager of around 500 Mbit). It creates the folder _UNPACK_Moviename in the specified complete directory for a movie. However now instead of just renaming the folder from _UNPACK_Moviename to Moviename it creates a new folder in the same directory called Moviename and copies the file from the unpack folder. Therefore the Moving action takes a lot of time as the files are read from the nas, transfered to my pc and from the pc sent again back to the nas.

This happens since version 1.1.1.

Help would be much appreciated.

Best regards from Switzerland.

Re: unpack ; unrar and then move the file

Posted: November 25th, 2016, 5:35 am
by TravisH
I am having the same issue.
Temporary download folder is local.
Completed download folder is a mapped network drive. (Z:\New folder)

Unpacking appears to be going over the network which is working great. Network machine seeing about 500mbps receive.
Then after unpacking, it's starts moving which is dreadfully slow.
Watching traffic on the local machine, I see traffic going out (80-100 Mbps)
On the network machine I see send and receive traffic (send about 500mbps, receive about 80mbps).

Is it sending the unpacked files back to the local machine and then back again to the networked machine?

*Edit* just read the post above mine which was posted 1 minute before me (weird). His experience is pretty much the same as mine.

Re: unpack ; unrar and then move the file

Posted: November 25th, 2016, 8:28 am
by safihre
Can you turn on Debug logging in the Status window?
Then I created 2 versions of SABnzbd.exe where I removed changes we introduced in 1.1.1 that might cause this.
Could you test both versions for me and see if you still see the problem?
https://www.dropbox.com/s/2in8cirfs8hbd ... n.zip?dl=1
https://www.dropbox.com/s/ilcvdnv24srb5 ... n.zip?dl=1
(links expire in 7 days)

Re: unpack ; unrar and then move the file

Posted: November 25th, 2016, 9:00 am
by mcboss86
Hey safihre

Thanks for your reply. Now I did some testing myself and maybe I found the culprit. There are 2 changes I made. First I disabled "Config -> switches -> naming -> enable folder renaming"
Second and probably the more relevant change, as also Couchpotato was suffering from this problem (Basically CP is instructed to monitor Y:\Completed\Movies and move and rename to Y:\Movies). Couchpotato had the same behaviour as sabnzbd (instead of just moving and renaming the folder on the network drive copying all contents and then delete the folder under completed\movies).

Instead of autologin into Windows 10 and having my network drives mounted as a user over the gui I created a batch file (net use command) and put it into the startup folder.
Now after a restart everything works as it is intended to (Just downloaded another movie for testing purposes and neither sab nor couchpotato showed the above mentioned strange behaviour, instead the moving parts were done within a second and no noticeable network traffic). If TravisH is also running Win10 I guess it is a mess up with folder permissions or something similar, that prevented the rename/move actions and I suggest you try to put a net use batch file into your autostartfolder.

Re: unpack ; unrar and then move the file

Posted: November 25th, 2016, 10:31 am
by safihre
Interesting!
Because the whole moving in sabnzbd is a simple command to the Operating system. Only when the OS says that a simple move command is somehow not possible, the copy procedure is used.