unpack ; unrar and then move the file

Get help with all aspects of SABnzbd
Forum rules
Help us help you:
  • Are you using the latest stable version of SABnzbd? Downloads page.
  • Tell us what system you run SABnzbd on.
  • Adhere to the forum rules.
  • Do you experience problems during downloading?
    Check your connection in Status and Interface settings window.
    Use Test Server in Config > Servers.
    We will probably ask you to do a test using only basic settings.
  • Do you experience problems during repair or unpacking?
    Enable +Debug logging in the Status and Interface settings window and share the relevant parts of the log here using [ code ] sections.
shpapy
Newbie
Newbie
Posts: 4
Joined: February 15th, 2013, 5:40 am

unpack ; unrar and then move the file

Post by shpapy »

Hello,
i have a dedicated machine which runs sabnzb; the machine has a 500Gb but my final complete folder sits on a NAS server.
currently when the sab works he downloads locally checks the release and then unpacks over the network. i would like to unpack locally and then transfer the file to the network.

can it be done? and how please

thanks a lot for the help.
User avatar
sander
Release Testers
Release Testers
Posts: 8808
Joined: January 22nd, 2008, 2:22 pm

Re: unpack ; unrar and then move the file

Post by sander »

Easy: point the Complete directory to a local directory, and in Category assign your realy destination directory to the Default category ...
pxr5
Newbie
Newbie
Posts: 26
Joined: July 22nd, 2008, 5:10 am

Re: unpack ; unrar and then move the file

Post by pxr5 »

Hi. Can someone explain this a little more clearly please. I don't mind unpacking across the network, but since 1.1.1 after the unpack and going onto the move phase, I appear to be getting a long period of read/writes across the network. Previously it would unpack and do the move instantly. Now, after the file has been unpacked and moved across to my NAS a lot of activity seems to carry on from the NAS and back again.

So what it's doing now is:

The download at 10 MB/s
Unpack to NAS at 90 MB/s
Move is a up/down at 14 MB/s

This makes the whole process nearly as long again as the original download. This never happened on any build before 1.1.1 and I have unchecked "Check result of unpacking", but that makes no difference.

So I was thinking to get round this, Unpack to the local drive, then move to the NAS. The explanation above doesn't seem clear to me.

Thank you.
pxr5
Newbie
Newbie
Posts: 26
Joined: July 22nd, 2008, 5:10 am

Re: unpack ; unrar and then move the file

Post by pxr5 »

Right. I've been looking at this some more, watching the folders as the process completes. As I said above, the files unpack fine across the network and the file completes and is in a folder on the NAS finished. But then during the move phase further read/writes take place across the newtwork. The complete folder on the NAS has the finished file inside, also the TEMP folder on the NAS has the finished file inside as this move operation takes place (which as I undestand should take place very quickly on the final NAS folder). The incomplete/admin folder has a few smaller files inside. Eventually the final file ends up in the correct destination as per the sorting and all other files disappear.

So, in a network graphical way it does this (d=download/repair, u=unpack, m=move), now it's really slowing down the whole process:

dddddddddddddddddd-uuu-mmmmmmmmmm

Before 1.1.1 the move phase was almost instaneous:

ddddddddddddddddd-uuu-m

Leaving the download to complete to local drive (complete folder) works correctly - if I can't fix this I'm going to have to manually copy files from the local folder to the NAS.
User avatar
safihre
Administrator
Administrator
Posts: 5335
Joined: April 30th, 2015, 7:35 am
Contact:

Re: unpack ; unrar and then move the file

Post by safihre »

Odd, nothing changed regarding the moving between 1.1.0 and 1.1.1 as far as I know.
So if you now downgrade to 1.1.0, you can confirm its faster again?
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
pxr5
Newbie
Newbie
Posts: 26
Joined: July 22nd, 2008, 5:10 am

Re: unpack ; unrar and then move the file

Post by pxr5 »

OK. Went back to 1.1.0 and everything was OK. Tried 1.1.1 and the issue came back. 1.1.0 again and now OK. I noticed in the final destination folder on 1.1.0 there is one folder created (a temporary UNPACK folder), on 1.1.1 there are two folders (a temporry UNPACK folder and a 'real' folder) - all of which disppeared at the end leaving the file in the right place.

In this network graph the one in the middle is 1.1.1, either side are 1.1.0 (red=download, green=unpack, yellow=move (a mix of up/download across the network on 1.1.1).

Image

I'll stick with 1.1.0 for now.
User avatar
safihre
Administrator
Administrator
Posts: 5335
Joined: April 30th, 2015, 7:35 am
Contact:

Re: unpack ; unrar and then move the file

Post by safihre »

So weird.. I checked all the code but I have no clue what's possibly changed since 1.1.0.

To make make sure I try to reproduce correctly:
Your temporary folder is on the pc, the final is on the NAS? Is there also a category specific folder active?
So green in your graph is unpack to the NAS and then during the yellow the status in Sab is 'Moving' still?
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
pxr5
Newbie
Newbie
Posts: 26
Joined: July 22nd, 2008, 5:10 am

Re: unpack ; unrar and then move the file

Post by pxr5 »

safihre wrote:So weird.. I checked all the code but I have no clue what's possibly changed since 1.1.0.

To make make sure I try to reproduce correctly:
Your temporary folder is on the pc, the final is on the NAS? Yes Is there also a category specific folder active? Yes
So green in your graph is unpack to the NAS and then during the yellow the status in Sab is 'Moving' still? Yes
Here is an extract from the log. There is a big delay after the unpack on 1.1.1, but nothing is showing as occuring in this period:

1.1.0
10GB file
2016-11-12 09:06:38,151::INFO::[postproc:371] unpack_magic finished on xxxxxxxxxxxxxxxx
2016-11-12 09:06:38,250::INFO::[postproc:564] Cleaning up xxxxxxxxxxxxxx (keep_basic=False)


1.1.1
5GB file
2016-11-18 14:45:37,552::INFO::[postproc:371] unpack_magic finished on yyyyyyyyyyyyyyyyy
2016-11-18 14:51:23,260::INFO::[postproc:564] Cleaning up yyyyyyyyyyyyyyy (keep_basic=False)

340MB file
2016-11-19 12:18:48,164::INFO::[postproc:371] unpack_magic finished on zzzzzzzzzzzzzz
2016-11-19 12:19:10,282::INFO::[postproc:564] Cleaning up zzzzzzzzzzzzz(keep_basic=False)
pxr5
Newbie
Newbie
Posts: 26
Joined: July 22nd, 2008, 5:10 am

Re: unpack ; unrar and then move the file

Post by pxr5 »

safihre wrote:So weird.. I checked all the code but I have no clue what's possibly changed since 1.1.0.

To make make sure I try to reproduce correctly:
Your temporary folder is on the pc, the final is on the NAS? Is there also a category specific folder active?
So green in your graph is unpack to the NAS and then during the yellow the status in Sab is 'Moving' still?
safihre - thanks, did you manage to reproduce this as I've remained on 1.1.0.
User avatar
safihre
Administrator
Administrator
Posts: 5335
Joined: April 30th, 2015, 7:35 am
Contact:

Re: unpack ; unrar and then move the file

Post by safihre »

I have send you 3 PM's last week because I need more information! But you didn't read them yet ;)
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
mcboss86
Newbie
Newbie
Posts: 3
Joined: November 25th, 2016, 5:24 am

Re: unpack ; unrar and then move the file

Post by mcboss86 »

Can you maybe post some further information?

SabNzbd is doing the same for me (1.1.1). My default download folder is on C:, my movies category points towards a network drive on a NAS. I saw it happen today and was looking a bit more closely.
Basically SabNZB is unpacking from C: to Network drive (Command line RAR has a network usage in the task manager of around 500 Mbit). It creates the folder _UNPACK_Moviename in the specified complete directory for a movie. However now instead of just renaming the folder from _UNPACK_Moviename to Moviename it creates a new folder in the same directory called Moviename and copies the file from the unpack folder. Therefore the Moving action takes a lot of time as the files are read from the nas, transfered to my pc and from the pc sent again back to the nas.

This happens since version 1.1.1.

Help would be much appreciated.

Best regards from Switzerland.
TravisH
Newbie
Newbie
Posts: 1
Joined: November 25th, 2016, 5:08 am

Re: unpack ; unrar and then move the file

Post by TravisH »

I am having the same issue.
Temporary download folder is local.
Completed download folder is a mapped network drive. (Z:\New folder)

Unpacking appears to be going over the network which is working great. Network machine seeing about 500mbps receive.
Then after unpacking, it's starts moving which is dreadfully slow.
Watching traffic on the local machine, I see traffic going out (80-100 Mbps)
On the network machine I see send and receive traffic (send about 500mbps, receive about 80mbps).

Is it sending the unpacked files back to the local machine and then back again to the networked machine?

*Edit* just read the post above mine which was posted 1 minute before me (weird). His experience is pretty much the same as mine.
User avatar
safihre
Administrator
Administrator
Posts: 5335
Joined: April 30th, 2015, 7:35 am
Contact:

Re: unpack ; unrar and then move the file

Post by safihre »

Can you turn on Debug logging in the Status window?
Then I created 2 versions of SABnzbd.exe where I removed changes we introduced in 1.1.1 that might cause this.
Could you test both versions for me and see if you still see the problem?
https://www.dropbox.com/s/2in8cirfs8hbd ... n.zip?dl=1
https://www.dropbox.com/s/ilcvdnv24srb5 ... n.zip?dl=1
(links expire in 7 days)
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
mcboss86
Newbie
Newbie
Posts: 3
Joined: November 25th, 2016, 5:24 am

Re: unpack ; unrar and then move the file

Post by mcboss86 »

Hey safihre

Thanks for your reply. Now I did some testing myself and maybe I found the culprit. There are 2 changes I made. First I disabled "Config -> switches -> naming -> enable folder renaming"
Second and probably the more relevant change, as also Couchpotato was suffering from this problem (Basically CP is instructed to monitor Y:\Completed\Movies and move and rename to Y:\Movies). Couchpotato had the same behaviour as sabnzbd (instead of just moving and renaming the folder on the network drive copying all contents and then delete the folder under completed\movies).

Instead of autologin into Windows 10 and having my network drives mounted as a user over the gui I created a batch file (net use command) and put it into the startup folder.
Now after a restart everything works as it is intended to (Just downloaded another movie for testing purposes and neither sab nor couchpotato showed the above mentioned strange behaviour, instead the moving parts were done within a second and no noticeable network traffic). If TravisH is also running Win10 I guess it is a mess up with folder permissions or something similar, that prevented the rename/move actions and I suggest you try to put a net use batch file into your autostartfolder.
User avatar
safihre
Administrator
Administrator
Posts: 5335
Joined: April 30th, 2015, 7:35 am
Contact:

Re: unpack ; unrar and then move the file

Post by safihre »

Interesting!
Because the whole moving in sabnzbd is a simple command to the Operating system. Only when the OS says that a simple move command is somehow not possible, the copy procedure is used.
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
Post Reply