Unable to Rename Sample+ file

Get help with all aspects of SABnzbd
Forum rules
Help us help you:
  • Are you using the latest stable version of SABnzbd? Downloads page.
  • Tell us what system you run SABnzbd on.
  • Adhere to the forum rules.
  • Do you experience problems during downloading?
    Check your connection in Status and Interface settings window.
    Use Test Server in Config > Servers.
    We will probably ask you to do a test using only basic settings.
  • Do you experience problems during repair or unpacking?
    Enable +Debug logging in the Status and Interface settings window and share the relevant parts of the log here using [ code ] sections.
User avatar
jcfp
Release Testers
Release Testers
Posts: 986
Joined: February 7th, 2008, 12:45 pm

Re: Unable to Rename Sample+ file

Post by jcfp »

@sander iirc par2 can handle subdirs these days, at least its changelog seems to suggest so. The substitution of the directory separator with a plus sign is probably sab's doing: there's some function in filesystem.py doing that, might be related to sanitizing input or safeguarding against directory traversals.
User avatar
sander
Release Testers
Release Testers
Posts: 8812
Joined: January 22nd, 2008, 2:22 pm

Re: Unable to Rename Sample+ file

Post by sander »

When I set SABnzbd, to
turn off Deobfuscate (because running the par2-deobfuscated causes a Traceback)
turn on Download all par2 (because they are needed because the rar files are damaged)
set download to +Delete

... the download success without errors, and the result is a directory with rars, par2s and the sample.mkv.

Then I did the below

mv xxx.s10e13.hog.heaven.720p.web.h264-caffeine.rar.1 xxx.s10e13.hog.heaven.720p.web.h264-caffeine.rar
unrar x xxx.s10e13.hog.heaven.720p.web.h264-caffeine.rar
par2 r 478f6eb128f245010f9cbd2b88f2a2c4.par2
unrar x xxx.s10e13.hog.heaven.720p.web.h264-caffeine.rar

... and everything is there.
User avatar
sander
Release Testers
Release Testers
Posts: 8812
Joined: January 22nd, 2008, 2:22 pm

Re: Unable to Rename Sample+ file

Post by sander »

jcfp wrote: February 17th, 2021, 11:51 am @sander iirc par2 can handle subdirs these days, at least its changelog seems to suggest so. The substitution of the directory separator with a plus sign is probably sab's doing: there's some function in filesystem.py doing that, might be related to sanitizing input or safeguarding against directory traversals.
These lines ... ?

CH_ILLEGAL = "/"
CH_LEGAL = "+"
CH_ILLEGAL_WIN = '\\/<>?*|"\t:'
CH_LEGAL_WIN = "++{}!@#'+-"

EDIT

I changed to
CH_ILLEGAL = "#"
CH_LEGAL = "+"

but a double Traceback and still "sample+Sample..." ... ???
timmyc
Newbie
Newbie
Posts: 5
Joined: February 17th, 2021, 3:09 am

Re: Unable to Rename Sample+ file

Post by timmyc »

sander wrote: February 17th, 2021, 4:57 pm When I set SABnzbd, to
turn off Deobfuscate (because running the par2-deobfuscated causes a Traceback)
turn on Download all par2 (because they are needed because the rar files are damaged)
set download to +Delete

... the download success without errors, and the result is a directory with rars, par2s and the sample.mkv.

Then I did the below

mv, unrar, par2, unrar etc

... and everything is there.
Yep, the fact that Sabnzbd can download the rar file fine, and a manual/simple unrar command works, that lends further credence to my theory that it's something to do with the way Sabnzbd is parsing arguments/filenames to unrar
User avatar
sander
Release Testers
Release Testers
Posts: 8812
Joined: January 22nd, 2008, 2:22 pm

Re: Unable to Rename Sample+ file

Post by sander »

I've created a reference NZB: https://raw.githubusercontent.com/sande ... 7a2982.nzb

Special thing: it has a subdirectory Sample (because that seems to be an issue?)
par2 created with par2command (indeed @jcfp that works)
All stuff is available, so no repair needed

My results:
OK on Windows
Failure on Linux: "Repair failed, not enough repair blocks (1199 short)"

On Linux, I see stuff like;

2021-02-19 12:03:01,471::DEBUG::[filesystem:819] Renaming "/home/sander/.sabnzbd/Downloads/incomplete/OK-with-subdir-5a62fa7a2982/bla1" to "/home/sander/.sabnzbd/Downloads/incomplete/OK-with-subdir-5a62fa7a2982/Sample/sub1"

Why would SAB do that?


Contents of the post:

bla1
bla2
bla3
bla4
Sample
Sample/sub1
Sample/sub2
with-sub-dir.par2
with-sub-dir.vol000+02.par2
with-sub-dir.vol002+04.par2
with-sub-dir.vol006+08.par2
with-sub-dir.vol014+16.par2
with-sub-dir.vol030+32.par2
with-sub-dir.vol062+64.par2
with-sub-dir.vol126+74.par2
User avatar
safihre
Administrator
Administrator
Posts: 5338
Joined: April 30th, 2015, 7:35 am
Contact:

Re: Unable to Rename Sample+ file

Post by safihre »

Par2cmdline seems to fail specifically if we provide the full paths to all the files, but succeed if we don't.
But we need to provide it the full paths because otherwise it doesn't scan fully obfuscated files.

What we could do is improve the quick check to also handle sub dirs, but that wouldn't work if the file in the sub directory is dameged itself..
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
User avatar
sander
Release Testers
Release Testers
Posts: 8812
Joined: January 22nd, 2008, 2:22 pm

Re: Unable to Rename Sample+ file

Post by sander »

Another reference NZB: all files have unique bytes: https://raw.githubusercontent.com/sande ... 9604e5.nzb

Results in a Failed, with SAB saying:

/home/sander/.sabnzbd/Downloads/incomplete/unique_files_with_subdir_ee0fba9604e5.1/mysubdir+insub1 cannot be renamed to /home/sander/.sabnzbd/Downloads/incomplete/unique_files_with_subdir_ee0fba9604e5.1/mysubdir/insub1

That is the "+ instead of /" kicking in ?



myfile1
myfile2
myfile3
mysubdir
mysubdir/insub1
mysubdir/insub2
important-subdir.par2
important-subdir.vol000+02.par2
important-subdir.vol002+04.par2
important-subdir.vol006+08.par2
important-subdir.vol014+16.par2
important-subdir.vol030+32.par2
important-subdir.vol062+64.par2
important-subdir.vol126+74.par2

EDIT:

This is the download result

Code: Select all

 __ADMIN__/
 important-subdir.par2
 important-subdir.vol000+02.par2
 important-subdir.vol002+04.par2
 important-subdir.vol006+08.par2
 important-subdir.vol014+16.par2
 myfile1
 myfile2
 myfile3
 mysubdir+insub1
 mysubdir+insub2

... mysubdir+insub1 ... that happens both with "Make Windows compatible" on and off.

Code: Select all

sander@brixit:~$ cat .sabnzbd/logs/sabnzbd.log | grep insub1
2021-02-19 14:54:30,937::INFO::[nzbparser:133] File insub1 added to queue
2021-02-19 14:54:30,944::DEBUG::[nzbstuff:383] Finishing import on insub1
2021-02-19 14:54:36,515::INFO::[nzbstuff:1683] Detected filename based on par2: insub1 -> mysubdir/insub1
2021-02-19 14:54:36,578::DEBUG::[assembler:102] Decoding part of /home/sander/.sabnzbd/Downloads/incomplete/unique_files_with_subdir_ee0fba9604e5/mysubdir+insub1
2021-02-19 14:54:36,618::DEBUG::[filesystem:541] Applying permissions 0o600 (octal) to /home/sander/.sabnzbd/Downloads/incomplete/unique_files_with_subdir_ee0fba9604e5/mysubdir+insub1
2021-02-19 14:54:36,619::INFO::[assembler:127] Decoding finished /home/sander/.sabnzbd/Downloads/incomplete/unique_files_with_subdir_ee0fba9604e5/mysubdir+insub1
2021-02-19 14:54:37,306::DEBUG::[newsunpack:2079] Quick-check will rename mysubdir+insub1 to mysubdir/insub1
2021-02-19 14:54:37,306::DEBUG::[filesystem:819] Renaming "/home/sander/.sabnzbd/Downloads/incomplete/unique_files_with_subdir_ee0fba9604e5/mysubdir+insub1" to "/home/sander/.sabnzbd/Downloads/incomplete/unique_files_with_subdir_ee0fba9604e5/mysubdir/insub1"
2021-02-19 14:54:37,306::INFO::[newsunpack:2096] Cannot Quick-check missing file mysubdir/insub1!
So SAB creates mysubdir+insub1 ... and then says it renames to mysubdir/insub1 ... but it can't do that:

Code: Select all

You have 30 recovery blocks available.
Repair is possible.
You have an excess of 30 recovery blocks.
None of the recovery blocks will be used for the repair.
/home/sander/.sabnzbd/Downloads/incomplete/unique_files_with_subdir_ee0fba9604e5.2/mysubdir+insub1 cannot be renamed to /home/sander/.sabnzbd/Downloads/incomplete/unique_files_with_subdir_ee0fba9604e5.2/mysubdir/insub1
Puzzled
Full Member
Full Member
Posts: 160
Joined: September 2nd, 2017, 3:02 am

Re: Unable to Rename Sample+ file

Post by Puzzled »

It will stop complaining about the rename if you make the directory first, at least in Windows. I'm not sure if that fixes everything, though.

Very quick and dirty code to be inserted before the rename in newsunpack:

Code: Select all

                    new_path = re.search("(.*)/", file)
                    if new_path:
                        new_path = os.path.join(nzo.download_path, new_path.group(1))
                        logging.debug("mkdir %s", new_path)
                        try:
                            os.mkdir(new_path)
                        except:
                            logging.debug("Failed to make path")
                    else:
                        logging.debug("No path found")
renamer(os.path.join...
User avatar
sander
Release Testers
Release Testers
Posts: 8812
Joined: January 22nd, 2008, 2:22 pm

Re: Unable to Rename Sample+ file

Post by sander »

Cool, that works: both for my reference NZB and for the IRL NZB provided earliear on.

@puzzled I'll take the discussion to github.
timmyc
Newbie
Newbie
Posts: 5
Joined: February 17th, 2021, 3:09 am

Re: Unable to Rename Sample+ file

Post by timmyc »

Awesome, so is this something that is/will be tracked in a Github issue moving forward? I had a look but couldn't see one already created.

Is this something likely to be worked through the development cycle and resolved in a subsequent release?
User avatar
sander
Release Testers
Release Testers
Posts: 8812
Joined: January 22nd, 2008, 2:22 pm

Re: Unable to Rename Sample+ file

Post by sander »

timmyc wrote: February 21st, 2021, 4:02 am Awesome, so is this something that is/will be tracked in a Github issue moving forward? I had a look but couldn't see one already created.

Is this something likely to be worked through the development cycle and resolved in a subsequent release?
See https://github.com/sabnzbd/sabnzbd/pull/1802 ... that git branch is working. If you know how to use git (and run SAB from source), you could try it yourself.

In subsequent release of SAB ... if/when main maintainer Safihre approves it.
timmyc
Newbie
Newbie
Posts: 5
Joined: February 17th, 2021, 3:09 am

Re: Unable to Rename Sample+ file

Post by timmyc »

Lovely. I am very familiar with git, and will run a build from source tomorrow.

Thanks heaps for the work on this guys.
User avatar
sander
Release Testers
Release Testers
Posts: 8812
Joined: January 22nd, 2008, 2:22 pm

Re: Unable to Rename Sample+ file

Post by sander »

timmyc wrote: February 21st, 2021, 4:27 am Lovely. I am very familiar with git, and will run a build from source tomorrow.
And? Feedback?
Luchtmacht
Newbie
Newbie
Posts: 2
Joined: February 17th, 2021, 3:15 am

Re: Unable to Rename Sample+ file

Post by Luchtmacht »

The latest update of my SABnzbd docker container seems to have fixed the bug.
Thank you very much for investigating so quickly.
Post Reply