Direct unpack files disappearing when another unpack in history queue ends.

Report & discuss bugs found in SABnzbd
Forum rules
Help us help you:
  • Are you using the latest stable version of SABnzbd? Downloads page.
  • Tell us what system you run SABnzbd on.
  • Adhere to the forum rules.
  • Do you experience problems during downloading?
    Check your connection in Status and Interface settings window.
    Use Test Server in Config > Servers.
    We will probably ask you to do a test using only basic settings.
  • Do you experience problems during repair or unpacking?
    Enable +Debug logging in the Status and Interface settings window and share the relevant parts of the log here using [ code ] sections.
Post Reply
jdevrie
Newbie
Newbie
Posts: 6
Joined: February 13th, 2017, 6:19 am

Direct unpack files disappearing when another unpack in history queue ends.

Post by jdevrie »

When I have multiple large downloads and there is one in the queue downloading and direct unpacking and there are two in the history/unpack queue of which one is unpacking, then the partly direct unpack that has already taken place is deleted when the unpacking entry in the history queue ends.
It starts unpacking all over again when it gets its turn in the history queue.
This is a high waist of resources.

Update: Just checked some details and it seems that the direct unpack files are not deleted but never created. It reports that it had done a direct unpack of files which it actually hasn't. I saw that when it got its turn to be unpacked the files had to be repaired so maybe it could not do a direct unpack because it required repair. It looks like it forgets to reset the counters for direct unpack to zero when it can not do that because of repairs that are required.
In the download queue it does not show the counters for direct unpack but when moved to the history queue it suddenly shows those numbers erroneously..
This is confusing because no real direct unpack files exist at that moment for that entry.

I searched in the logfile for one of the entries that said that there where direct unpack files while there actullay weren't and I found the following entries that indicated that direct unpack failed. Here are the log entries:

2021-01-16 11:53:06,812::INFO::[directunpacker:202] Error in DirectUnpack of zeTHErPTiONTRimumERMen: Unexpected end of archive
2021-01-16 11:53:12,816::INFO::[directunpacker:426] Aborting DirectUnpack for zeTHErPTiONTRimumERMen

So it seems like I am right about the fact that the entries in the history queue show info about the direct unpack files that is no longer valid because the direct unpack was aborted because of repairs that are required.
User avatar
safihre
Administrator
Administrator
Posts: 5338
Joined: April 30th, 2015, 7:35 am
Contact:

Re: Direct unpack files disappearing when another unpack in history queue ends.

Post by safihre »

It can pause the Direct Unpack when it detects problems, maybe that's what you are seeing.
From just the description it's bit hard to understand what is going on.
Can you enable +Debug logging in the Status and Interface and then try again? Share the logs via pastebin or send them to me at [email protected]
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
jdevrie
Newbie
Newbie
Posts: 6
Joined: February 13th, 2017, 6:19 am

Re: Direct unpack files disappearing when another unpack in history queue ends.

Post by jdevrie »

I did some more testing with +Debug logging on:
I found out from the log file that when the unrar thinks that the password is wrong all files and subdirectories are recursively deleted. However the unrar sometimes mistakenly thinks the password is wrong when a sub part of the rar archive is severely damaged. So in that case all the direct unpack files are deleted while in fact that was not necessary. The archive could have been repaired and direct unpack restarted. But of course SABNZBD can’t know that since it depends on unrar to find out what is the status of the archive’s subpart.
Here are the log files records that lead me to this conclusion:

Code: Select all

2021-01-18 12:10:32,887::DEBUG::[filesystem:769] Renaming "\\?\y:\##Nieuw\zeTHErPTiONTRimumERMen (2)" to "\\?\y:\##Nieuw\_UNPACK_zeTHErPTiONTRimumERMen (2)"
2021-01-18 12:10:32,907::INFO::[misc:986] [sabnzbd\directunpacker.py.create_unrar_instance] Running external command: "C:\Program Files\SABnzbd\win\unrar\x64\UnRAR.exe" "x" "-vp" "-idp" "-o+" "-ai" "-p-" "D:\Downloads\zeTHErPTiONTRimumERMen (2)\zeTHErPTiONTRimumERMen.part01.rar" "\\?\y:\##Nieuw\_UNPACK_zeTHErPTiONTRimumERMen (2)\"
2021-01-18 12:10:32,907::DEBUG::[misc:987] Popen arguments: {'stdin': -1, 'stdout': -1, 'stderr': -2, 'startupinfo': <subprocess.STARTUPINFO object at 0x000001A2FB8D8E80>, 'creationflags': 32, 'bufsize': 0}
2021-01-18 12:10:32,913::INFO::[directunpacker:420] DirectUnpacked volume 1 for zeTHErPTiONTRimumERMen
2021-01-18 12:10:32,914::DEBUG::[assembler:89] Decoding part of \\?\D:\Downloads\zeTHErPTiONTRimumERMen (2)\zeTHErPTiONTRimumERMen.part02.rar
2021-01-18 12:10:33,003::INFO::[directunpacker:202] [b][size=150]Error in DirectUnpack of zeTHErPTiONTRimumERMen: Incorrect password[/size][/b]
2021-01-18 12:10:33,003::INFO::[directunpacker:426] Aborting DirectUnpack for zeTHErPTiONTRimumERMen
2021-01-18 12:10:33,035::DEBUG::[assembler:89] Decoding part of \\?\D:\Downloads\zeTHErPTiONTRimumERMen (2)\zeTHErPTiONTRimumERMen.part02.rar
2021-01-18 12:10:33,425::DEBUG::[filesystem:837] [b][size=150]Removing dir recursively \\?\y:\##Nieuw\_UNPACK_zeTHErPTiONTRimumERMen (2)[/size][/b]
2021-01-18 12:10:33,440::DEBUG::[directunpacker:315] DirectUnpack Unrar output 
UNRAR 5.91 x64 freeware      Copyright (c) 1993-2020 Alexander Roshal
[code]

[b][size=150]Incorrect password[/size][/b]
jdevrie
Newbie
Newbie
Posts: 6
Joined: February 13th, 2017, 6:19 am

Re: Direct unpack files disappearing when another unpack in history queue ends.

Post by jdevrie »

I did notice another behavior that could be improved.
When an archive has a password and the downloader or unpacker detetcs that before the user has provided the password in the user interface, the download is paused automatically by the program and no direct unpack was started because it cannot do that without the password.
However when the password is then provided by the user and the download restarted, the program does not start a direct unpack for that archive anymore.
It would be nice if it would start a direct unpack after the restart of the fownload by the userinterface if a proper password has been provided by the user.
User avatar
safihre
Administrator
Administrator
Posts: 5338
Joined: April 30th, 2015, 7:35 am
Contact:

Re: Direct unpack files disappearing when another unpack in history queue ends.

Post by safihre »

Indeed, this is how it works.
You have to understand that the Direct Unpack system is very tricky to manage correctly with all the timings happening in SABnzbd and as such it's made as a "best-effort". We try our best, but if it's too special or something is wrong, it's back to old-skool unpacking.
Because all code that we add for "regular unpacking" to make sure it always works, we have to then add also for Direct Unpack, because it's just slightly different. Leading to a lot of extra code to maintain.
You could help SABnzbd by providing the password with the job, for example inside the name of the job! https://sabnzbd.org/wiki/advanced/passw ... ected-rars
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
jdevrie
Newbie
Newbie
Posts: 6
Joined: February 13th, 2017, 6:19 am

Re: Direct unpack files disappearing when another unpack in history queue ends.

Post by jdevrie »

I agree that writing multitasking or multi threading programs can be quite a hassle. I have written a few myself in the past (I am retired now ;D )
Only in this particular case it seems like the logical thing to do to retry direct unpack on a restart that was paused on a missing password if a password has been provided in the meantime.
The bypass I use now is to first pause the queue before I throw in the first download. Then I provide the password and then unpause the queue.
Subsequent downloads have to wait for the first one to finish so there I have enough time to put in the password before it actually starts downloading and unpacking.
Post Reply