Page 3 of 3
Re: Unable to Rename Sample+ file
Posted: February 17th, 2021, 11:51 am
by jcfp
@sander iirc par2 can handle subdirs these days, at least its
changelog seems to suggest so. The substitution of the directory separator with a plus sign is probably sab's doing: there's some function in filesystem.py doing that, might be related to sanitizing input or safeguarding against directory traversals.
Re: Unable to Rename Sample+ file
Posted: February 17th, 2021, 4:57 pm
by sander
When I set SABnzbd, to
turn off Deobfuscate (because running the par2-deobfuscated causes a Traceback)
turn on Download all par2 (because they are needed because the rar files are damaged)
set download to +Delete
... the download success without errors, and the result is a directory with rars, par2s and the sample.mkv.
Then I did the below
mv xxx.s10e13.hog.heaven.720p.web.h264-caffeine.rar.1 xxx.s10e13.hog.heaven.720p.web.h264-caffeine.rar
unrar x xxx.s10e13.hog.heaven.720p.web.h264-caffeine.rar
par2 r 478f6eb128f245010f9cbd2b88f2a2c4.par2
unrar x xxx.s10e13.hog.heaven.720p.web.h264-caffeine.rar
... and everything is there.
Re: Unable to Rename Sample+ file
Posted: February 17th, 2021, 5:03 pm
by sander
jcfp wrote: ↑February 17th, 2021, 11:51 am
@sander iirc par2 can handle subdirs these days, at least its
changelog seems to suggest so. The substitution of the directory separator with a plus sign is probably sab's doing: there's some function in filesystem.py doing that, might be related to sanitizing input or safeguarding against directory traversals.
These lines ... ?
CH_ILLEGAL = "/"
CH_LEGAL = "+"
CH_ILLEGAL_WIN = '\\/<>?*|"\t:'
CH_LEGAL_WIN = "++{}!@#'+-"
EDIT
I changed to
CH_ILLEGAL = "#"
CH_LEGAL = "+"
but a double Traceback and still "sample+Sample..." ...
Re: Unable to Rename Sample+ file
Posted: February 18th, 2021, 8:45 am
by timmyc
sander wrote: ↑February 17th, 2021, 4:57 pm
When I set SABnzbd, to
turn off Deobfuscate (because running the par2-deobfuscated causes a Traceback)
turn on Download all par2 (because they are needed because the rar files are damaged)
set download to +Delete
... the download success without errors, and the result is a directory with rars, par2s and the sample.mkv.
Then I did the below
mv, unrar, par2, unrar etc
... and everything is there.
Yep, the fact that Sabnzbd can download the rar file fine, and a manual/simple unrar command works, that lends further credence to my theory that it's something to do with the way Sabnzbd is parsing arguments/filenames to unrar
Re: Unable to Rename Sample+ file
Posted: February 19th, 2021, 6:18 am
by sander
I've created a reference NZB:
https://raw.githubusercontent.com/sande ... 7a2982.nzb
Special thing: it has a subdirectory Sample (because that seems to be an issue?)
par2 created with par2command (indeed @jcfp that works)
All stuff is available, so no repair needed
My results:
OK on Windows
Failure on Linux: "Repair failed, not enough repair blocks (1199 short)"
On Linux, I see stuff like;
2021-02-19 12:03:01,471::DEBUG::[filesystem:819] Renaming "/home/sander/.sabnzbd/Downloads/incomplete/OK-with-subdir-5a62fa7a2982/bla1" to "/home/sander/.sabnzbd/Downloads/incomplete/OK-with-subdir-5a62fa7a2982/Sample/sub1"
Why would SAB do that?
Contents of the post:
bla1
bla2
bla3
bla4
Sample
Sample/sub1
Sample/sub2
with-sub-dir.par2
with-sub-dir.vol000+02.par2
with-sub-dir.vol002+04.par2
with-sub-dir.vol006+08.par2
with-sub-dir.vol014+16.par2
with-sub-dir.vol030+32.par2
with-sub-dir.vol062+64.par2
with-sub-dir.vol126+74.par2
Re: Unable to Rename Sample+ file
Posted: February 19th, 2021, 6:39 am
by safihre
Par2cmdline seems to fail specifically if we provide the full paths to all the files, but succeed if we don't.
But we need to provide it the full paths because otherwise it doesn't scan fully obfuscated files.
What we could do is improve the quick check to also handle sub dirs, but that wouldn't work if the file in the sub directory is dameged itself..
Re: Unable to Rename Sample+ file
Posted: February 19th, 2021, 8:58 am
by sander
Another reference NZB: all files have unique bytes:
https://raw.githubusercontent.com/sande ... 9604e5.nzb
Results in a Failed, with SAB saying:
/home/sander/.sabnzbd/Downloads/incomplete/unique_files_with_subdir_ee0fba9604e5.1/mysubdir+insub1 cannot be renamed to /home/sander/.sabnzbd/Downloads/incomplete/unique_files_with_subdir_ee0fba9604e5.1/mysubdir/insub1
That is the "+ instead of /" kicking in ?
myfile1
myfile2
myfile3
mysubdir
mysubdir/insub1
mysubdir/insub2
important-subdir.par2
important-subdir.vol000+02.par2
important-subdir.vol002+04.par2
important-subdir.vol006+08.par2
important-subdir.vol014+16.par2
important-subdir.vol030+32.par2
important-subdir.vol062+64.par2
important-subdir.vol126+74.par2
EDIT:
This is the download result
Code: Select all
__ADMIN__/
important-subdir.par2
important-subdir.vol000+02.par2
important-subdir.vol002+04.par2
important-subdir.vol006+08.par2
important-subdir.vol014+16.par2
myfile1
myfile2
myfile3
mysubdir+insub1
mysubdir+insub2
... mysubdir+insub1 ... that happens both with "Make Windows compatible" on and off.
Code: Select all
sander@brixit:~$ cat .sabnzbd/logs/sabnzbd.log | grep insub1
2021-02-19 14:54:30,937::INFO::[nzbparser:133] File insub1 added to queue
2021-02-19 14:54:30,944::DEBUG::[nzbstuff:383] Finishing import on insub1
2021-02-19 14:54:36,515::INFO::[nzbstuff:1683] Detected filename based on par2: insub1 -> mysubdir/insub1
2021-02-19 14:54:36,578::DEBUG::[assembler:102] Decoding part of /home/sander/.sabnzbd/Downloads/incomplete/unique_files_with_subdir_ee0fba9604e5/mysubdir+insub1
2021-02-19 14:54:36,618::DEBUG::[filesystem:541] Applying permissions 0o600 (octal) to /home/sander/.sabnzbd/Downloads/incomplete/unique_files_with_subdir_ee0fba9604e5/mysubdir+insub1
2021-02-19 14:54:36,619::INFO::[assembler:127] Decoding finished /home/sander/.sabnzbd/Downloads/incomplete/unique_files_with_subdir_ee0fba9604e5/mysubdir+insub1
2021-02-19 14:54:37,306::DEBUG::[newsunpack:2079] Quick-check will rename mysubdir+insub1 to mysubdir/insub1
2021-02-19 14:54:37,306::DEBUG::[filesystem:819] Renaming "/home/sander/.sabnzbd/Downloads/incomplete/unique_files_with_subdir_ee0fba9604e5/mysubdir+insub1" to "/home/sander/.sabnzbd/Downloads/incomplete/unique_files_with_subdir_ee0fba9604e5/mysubdir/insub1"
2021-02-19 14:54:37,306::INFO::[newsunpack:2096] Cannot Quick-check missing file mysubdir/insub1!
So SAB creates mysubdir+insub1 ... and then says it renames to mysubdir/insub1 ... but it can't do that:
Code: Select all
You have 30 recovery blocks available.
Repair is possible.
You have an excess of 30 recovery blocks.
None of the recovery blocks will be used for the repair.
/home/sander/.sabnzbd/Downloads/incomplete/unique_files_with_subdir_ee0fba9604e5.2/mysubdir+insub1 cannot be renamed to /home/sander/.sabnzbd/Downloads/incomplete/unique_files_with_subdir_ee0fba9604e5.2/mysubdir/insub1
Re: Unable to Rename Sample+ file
Posted: February 19th, 2021, 1:25 pm
by Puzzled
It will stop complaining about the rename if you make the directory first, at least in Windows. I'm not sure if that fixes everything, though.
Very quick and dirty code to be inserted before the rename in newsunpack:
Code: Select all
new_path = re.search("(.*)/", file)
if new_path:
new_path = os.path.join(nzo.download_path, new_path.group(1))
logging.debug("mkdir %s", new_path)
try:
os.mkdir(new_path)
except:
logging.debug("Failed to make path")
else:
logging.debug("No path found")
renamer(os.path.join...
Re: Unable to Rename Sample+ file
Posted: February 20th, 2021, 2:08 am
by sander
Cool, that works: both for my reference NZB and for the IRL NZB provided earliear on.
@puzzled I'll take the discussion to github.
Re: Unable to Rename Sample+ file
Posted: February 21st, 2021, 4:02 am
by timmyc
Awesome, so is this something that is/will be tracked in a Github issue moving forward? I had a look but couldn't see one already created.
Is this something likely to be worked through the development cycle and resolved in a subsequent release?
Re: Unable to Rename Sample+ file
Posted: February 21st, 2021, 4:17 am
by sander
timmyc wrote: ↑February 21st, 2021, 4:02 am
Awesome, so is this something that is/will be tracked in a Github issue moving forward? I had a look but couldn't see one already created.
Is this something likely to be worked through the development cycle and resolved in a subsequent release?
See
https://github.com/sabnzbd/sabnzbd/pull/1802 ... that git branch is working. If you know how to use git (and run SAB from source), you could try it yourself.
In subsequent release of SAB ... if/when main maintainer Safihre approves it.
Re: Unable to Rename Sample+ file
Posted: February 21st, 2021, 4:27 am
by timmyc
Lovely. I am very familiar with git, and will run a build from source tomorrow.
Thanks heaps for the work on this guys.
Re: Unable to Rename Sample+ file
Posted: February 24th, 2021, 9:22 am
by sander
timmyc wrote: ↑February 21st, 2021, 4:27 am
Lovely. I am very familiar with git, and will run a build from source tomorrow.
And? Feedback?
Re: Unable to Rename Sample+ file
Posted: April 5th, 2021, 6:39 am
by Luchtmacht
The latest update of my SABnzbd docker container seems to have fixed the bug.
Thank you very much for investigating so quickly.