UNRAR/Categorizing behaviour

[ab]

At the moment sabnzbd (4beta2) seems to unrar downloaded (and verified) posts into a directory tagged with the "_UNPACKING_"prefix inside the main category folder

As an example:
At the moment it unrars the verified post into "_UNPACKING_Lost (2004) - 2x14 - One Of Them" inside /data/tv and after its done unraring it it moves it into the final destination "/data/tv/Lost (2004)/Season 2/Lost (2004) - 2x14 - One Of Them".

It may not play any role if the /data folder (the completed downloads folder) is a local disk. It just changes the path (or renames if you please) to move it to the new directory. But in the case where the completed downloads folder may be a networked share (samba share or nfs mount or whatever) or a complex of different partitions using for example unix links, moving it around introduces great overhead (the file has to move around the network (or various hdd partitions) multiple times instead of once ).

What I suggest is that the file should be unrared directly into the final folder (even the final definite pathname) instead of tagging it with the _UNPACKING_ prefix inside a different directory and then moving it. Another solution would be to unrar it inside of the temporary downloading folder and then move the (already unrared) content into the final destination.

[neilt0]

introduces great overhead (the file has to move around the network (or various hdd partitions) multiple times instead of once ).

No it doesn't. Where are people getting these crazy ideas from?!

SAB just renames the dest folder after unpacking. Look at the folder live as it is unpacking.

[ab]

introduces great overhead (the file has to move around the network (or various hdd partitions) multiple times instead of once ).

No it doesn't. Where are people getting these crazy ideas from?!

SAB just renames the dest folder after unpacking. Look at the folder live as it is unpacking.

It moves it around which results in renaming ONLY if both the source and target parts are on the same LOCAL partition.

If those are different partitions (eg. tv can be different partition than movies is ... both *nix`s and NTFS allows this) then it gets moved there (eg copied and source deleted afterwards).

If the partition is not local (eg samba or nfs share). Then it gets copied from the remote share to the client and then again from the client to a different location in the remote share.

Got it ?

PS: To sum it up ...
-it works without overhead only if both source and target are the same LOCAL partition.
-you have to copy it around twice if source partition is different than the target partition.
-you have to copy it around THREE times if it is a network mount.

[shypike]

We are giving the unpacking code a critical review (old 0.2.5 code).
This issue will be looked at too.

[neilt0]

Still not seeing your point. If you are unpacking to (in my case)

\\LSLIVE\TV\UNPACKING_This is a nice show

and then SAB renames

\\LSLIVE\TV\UNPACKING_This is a nice show

to

\\LSLIVE\TV\This is a nice show

How can this be on a different partition? It's in the same folder.

[switch]

ab, what is your TV Sorting set to?

[ab]

I gave an example in the first post that should have made it pretty clear. I have it set to sort by shows and seasons.

By the way i believe this needs a more through explanation.

So let me tell you step by step what exactly happens.

My /down (which is the temporary download folder) is on my local machine while my /data (completed downloads folder) is on a remote machine (NFS mount).

When a posts starts downloading SAB downloads it into the local /down folder .. decodes the articles there and par2-verifies the whole post. Afterwards it runs winrar on the files while extracting the files directly into the /data/categody/_UNPACKING_postname/ folder. Thats one network transfer already happening. Then when it is done extracting, it decides that /data/categody/_UNPACKING_postname/ is not the proper location of the post so it moves it from the old location into the proper (and final location). Now as you hopefully know ... files are not physically moved when its within the same LOCAL partition, they are just renamed with a different pathname. The story changes when the partition is on a network share (eg not locally on your computer). When trying to move a file into another location ... what it basically does is it copies the file from the remote machine to the local just in order to upload it to a different location on the remote machine.

This all could be avoided by unraring it directly into the proper FINAL folder instead of the temporary folder inside the category root (means not within the show/season structure).

[neilt0]

OK, I didn't read your first post properly then.

You're saying /data/tv is a different partition to data/tv/Lost?

I can see how that's possible, even if it's a bit unusual.
I see where your problem is now.

[ab]

Hmm i just did some testing because of inpheaux`s input about the problem.

I found out that its not the problem with NFS not being able to do a remote mv by just renaming it, because a simple mv command on the local machine does indeed just rename it. It seems like its just the python method of moving files around that copies it to the local machine and then uploads it to the remote again.

And plz dont post posts like it cant be happening etc .. it is happening and ive got traffic (and disk i/o) logs to prove it is behaving exactly the way i described.

[ab]

OK, I didn't read your first post properly then.

You're saying /data/tv is a different partition to data/tv/Lost?

I can see how that's possible, even if it's a bit unusual.
I see where your problem is now.

Well lets say that you have 400GB in TV and 400GB in movies and you have just 2 500GB disks and a 50GB system disk  ... so you mount one disk to /data/tv and another to /data/movies while /data is on the system disk ... gotcha ?

BTW As you might have noticed the thread just split into 2 topics. First is the unproper behaviour of the python moving mechanisms while operating on NFS mounts. Second one is the suitability of moving the files around the Completed downloads folder at all.

That way /data is a different partition in regard to /data/tv or /data/movies

BTW As you might have noticed the thread just split into 2 topics. First is the unproper behaviour of the python moving mechanisms while operating on NFS mounts. Second one is the suitability of moving the files around the Completed downloads folder at all.

[neilt0]

And plz dont post posts like it cant be happening etc .. it is happening and ive got traffic (and disk i/o) logs to prove it is behaving exactly the way i described.

The problem is when you wrote:

"But in the case where the completed downloads folder may be a networked share (samba share or nfs mount or whatever) or"

The part before the "or" doesn't make sense if it's a simple network share.

I understand this issue if you mount a drive inside a directory, but the part in quotes above was misleading.  If you didn't read on. As I didn't.

[ab]

And plz dont post posts like it cant be happening etc .. it is happening and ive got traffic (and disk i/o) logs to prove it is behaving exactly the way i described.

The problem is when you wrote:

"But in the case where the completed downloads folder may be a networked share (samba share or nfs mount or whatever) or"

The part before the "or" doesn't make sense if it's a simple network share.

I understand this issue if you mount a drive inside a directory, but the part in quotes above was misleading.  If you didn't read on. As I didn't.

It seems like the python moving mechanism does not work properly when operating on a remote partition (one on a different machine). Basic unix mv or windows move does work properly tho so it has to be a bug somewhere in the python method.