Potential race condition in Windows create_all_dirs

Report & discuss bugs found in SABnzbd
Forum rules
Help us help you:
  • Are you using the latest stable version of SABnzbd? Downloads page.
  • Tell us what system you run SABnzbd on.
  • Adhere to the forum rules.
  • Do you experience problems during downloading?
    Check your connection in Status and Interface settings window.
    Use Test Server in Config > Servers.
    We will probably ask you to do a test using only basic settings.
  • Do you experience problems during repair or unpacking?
    Enable +Debug logging in the Status and Interface settings window and share the relevant parts of the log here using [ code ] sections.
Post Reply
ordeneus
Newbie
Newbie
Posts: 6
Joined: November 23rd, 2020, 3:49 pm

Potential race condition in Windows create_all_dirs

Post by ordeneus »

Latest Windows version.
Problem exists only when waking from sleep.

I believe it's this part of the code from create_all_dirs:

Code: Select all

        if sabnzbd.WIN32:
            if not os.path.exists(path):
                os.makedirs(path)
I think this is happening before the drive can be mounted (from a Synology DS220+). I don't believe it's a permissions issue, simply a race? It doesn't happen on restart, only on waking from sleep.

Could this resolve it?

Code: Select all

        if sabnzbd.WIN32:
            if check_mount(path) and not os.path.exists(path):
                os.makedirs(path)
It would loop for wait_ext_drive seconds which would resolve the race?
User avatar
safihre
Administrator
Administrator
Posts: 5338
Joined: April 30th, 2015, 7:35 am
Contact:

Re: Potential race condition in Windows create_all_dirs

Post by safihre »

Could you provide a bit more context? What error are you getting and where?

You call it a race condition, but the definition of a race condition is that 2 processed try to modify the same thing at the same time. That can't be the case, because the function is locked and can only be active from 1 process.
I think you just mean a timeout problem?
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
ordeneus
Newbie
Newbie
Posts: 6
Joined: November 23rd, 2020, 3:49 pm

Re: Potential race condition in Windows create_all_dirs

Post by ordeneus »

Sorry, it wouldn't let me poste the error because I'm a newb.

When Windows wakes up from sleep with Sabnzbd running as an app I am seeing the following error appear in the console:
2022-10-29 09:05:02,328::INFO::[filesystem:703] Creating directories: n:\Downloads
2022-10-29 09:05:02,344::INFO::[notifier:123] Sending notification: Error - Failed making (n: \ Downloads) (type=error, job_cat=None)
2022-10-29 09:05:02,342::ERROR::[filesystem:725] Failed making (n: \ Downloads)
Traceback (most recent call last):
File sabnzbd \ filesystem . py, line 708, in create_all_dirs
File os . py , line 215, in makedirs
File os . py , line 225, in makedirs
FileNotFoundError: [WinError 53] The network path was not found:
I presumed this was because Windows had not restored the connection to the drive before sab was trying to create a directory on the drive? This only happens on waking from sleep. If I restart sab (with the machine running obviously) the error doesn't occur.
User avatar
safihre
Administrator
Administrator
Posts: 5338
Joined: April 30th, 2015, 7:35 am
Contact:

Re: Potential race condition in Windows create_all_dirs

Post by safihre »

Yes you are right. Indeed try the wait_ext_drive, that's made for this.
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
ordeneus
Newbie
Newbie
Posts: 6
Joined: November 23rd, 2020, 3:49 pm

Re: Potential race condition in Windows create_all_dirs

Post by ordeneus »

I have that value set to 10 (default is 5 I believe). It happened again this morning.

The only reference to wait_ext_drive is in check_mount, which is referenced by get_unique_dir which is referenced by nzbstuff . py and postproc . py.

I may be reading this wrong of course, but does check_mount (which would look for wait_ext_drive) get called by create_all_dirs?
User avatar
safihre
Administrator
Administrator
Posts: 5338
Joined: April 30th, 2015, 7:35 am
Contact:

Re: Potential race condition in Windows create_all_dirs

Post by safihre »

Yes, that's only for during startup of SABnzbd.
Just to be sure, you are not putting your Incomplete directory on the NAS right? Only the Complete directory?
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
ordeneus
Newbie
Newbie
Posts: 6
Joined: November 23rd, 2020, 3:49 pm

Re: Potential race condition in Windows create_all_dirs

Post by ordeneus »

Sorry yes, should've been clearer.

Temp download is a local SSD.
Completed directory is the NAS.
User avatar
safihre
Administrator
Administrator
Posts: 5338
Joined: April 30th, 2015, 7:35 am
Contact:

Re: Potential race condition in Windows create_all_dirs

Post by safihre »

Good that the download is local!
Hmmm, I'm not sure how to solve this then.
It seems indeed the connection just isn't up yet when SABnzbd wants to write to it. But we can't check before every write operation if the disk is there, that would cause too much overhead.
Maybe there's a way to force Windows to reconnect quicker?
Have you tried instead of using N:\ to use the network path? \\NAS\Downloads or something like that?
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: Potential race condition in Windows create_all_dirs

Post by sander »

Or: keep Complete local too, and use a post-processing script to move to your NAS?
ordeneus
Newbie
Newbie
Posts: 6
Joined: November 23rd, 2020, 3:49 pm

Re: Potential race condition in Windows create_all_dirs

Post by ordeneus »

The current line from create_all_dirs is:

Code: Select all

if not os.path.exists(path):
                os.makedirs(path)
If this were changed to:

Code: Select all

if not os.path.exists(path) and check_mount(path):
                os.makedirs(path)
The "and" is a short circuit operator, so "check_mount(path)" would only be evaluated if the first part of the if statement were true? Meaning it would not be evaluated if the path does exist? You would only be adding a check for the disk if the directory doesn't exist (which is already being evaluated).
User avatar
safihre
Administrator
Administrator
Posts: 5338
Joined: April 30th, 2015, 7:35 am
Contact:

Re: Potential race condition in Windows create_all_dirs

Post by safihre »

Yes, you highlight now the one spot where you experience a problem ;)
But there are multiple spots throughout downloading and post-processing where we write to the Complete Directory, so all those would need to be adapted to use check_mount.

It seems that more people would have this problem, so maybe there's an easier solution? We have not had this reported before on the forums.
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
ordeneus
Newbie
Newbie
Posts: 6
Joined: November 23rd, 2020, 3:49 pm

Re: Potential race condition in Windows create_all_dirs

Post by ordeneus »

Ha. Always good to find a novel issue ;D
Post Reply