3.2.0 [7be9281] - Free Space is wrong [MacOS]

Report & discuss bugs found in SABnzbd
Forum rules
Help us help you:
  • Are you using the latest stable version of SABnzbd? Downloads page.
  • Tell us what system you run SABnzbd on.
  • Adhere to the forum rules.
  • Do you experience problems during downloading?
    Check your connection in Status and Interface settings window.
    Use Test Server in Config > Servers.
    We will probably ask you to do a test using only basic settings.
  • Do you experience problems during repair or unpacking?
    Enable +Debug logging in the Status and Interface settings window and share the relevant parts of the log here using [ code ] sections.
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS] drive > 4TB

Post by sander »

blackntan wrote: March 25th, 2021, 6:35 am changing the calculation to use f_bavail (instead of f_bfree) results in identical values to df
Wow! Thank you. I'll change it in my PoC code asap.

EDIT: changed in PoC code (https://github.com/sanderjo/diskfree-st ... tvfs_df.py) , and on Mac now exact match:

Code: Select all

server:diskfree-statvfs-df sander$ python3 diskfree_statvfs_df.py /
dir is /
df is always right, so: Disk size, and free (in MB): (953049, 521921)
python's os.statvfs() says (953050, 521921)
clib statfs32 says (953050, 521921)
Now the real determination
(953050, 521921)
Nice!
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by sander »

blackntan wrote: March 24th, 2021, 7:42 am
so, maybe we should just use stat64 and only on macos ...
Yes, my PoC code is now MacOS only: "if MacOS and (df disk size >4TB) then use statfs32".
Maybe I have to change to statfs64, as that feels more 2021.
blackntan wrote: March 24th, 2021, 7:42 am leave the statvfs alone for linux ... it has something to do with the ctypes being used for the class's in python ... if i change the f_blocks to a c_long it actually reports the correct size in linux, but incorrect on macos ...
Did you get the code working with statfs32/statfs64 on Linux? If so, can you post it? Maybe we need it later on if the same problem pops up on Linux systems after all ... my first suspect would 32-bit Linux (raspi) with a Big Disk (>4TB) ...
blackntan
Newbie
Newbie
Posts: 16
Joined: March 21st, 2021, 7:16 am

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by blackntan »

yes, I didn't define the entire struct ... but this works for linux:

Code: Select all

def linux_disk_free_clib_statfs(directory):
	class statfs(Structure):
	    _fields_ = [
                ("f_type",       c_int64),
                ("f_bsize",      c_int64),
                ("f_blocks",     c_ulong),
                ("f_bfree",      c_ulong),
                ("f_bavail",     c_ulong),
                ("f_files",      c_ulonglong),
                ("f_ffree",      c_ulonglong),
                ("f_frsize",     c_ulong),
		       ]

	kern = CDLL(util.find_library('c'), use_errno=True)
	root_volume = create_string_buffer(str.encode(directory))
	fs_info = statfs()
	result = kern.statfs(root_volume, byref(fs_info)) # you have to call this to get fs_info filled out
	disk_size_MB = fs_info.f_blocks * fs_info.f_bsize / 1024**2
	free_size_MB = fs_info.f_bavail  * fs_info.f_bsize / 1024**2
	return disk_size_MB, free_size_MB

Code: Select all

dir is /media/psf/Big_One/
df is always right, so: Disk size, and free (in MB): (38149816, 35697196)
python's os.statvfs() says (38149816, 35697195)
clib statfs32 says (8158310.44921875, 18201800055525.207)
clib linux statfs says (38149815.9609375, 35697195.234375)
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by sander »

Thanks. Correct output on Linux, ... but as a bonus I also get a coredump :-(

Code: Select all

sander@brixit:~$ df -m /
Filesystem     1M-blocks  Used Available Use% Mounted on
/dev/sdb2         111659 43070     62875  41% /
sander@brixit:~$

Code: Select all

sander@brixit:~$ python3 diskfree_linux.py /
/
(111658.16015625, 62874.61328125)
Segmentation fault (core dumped)
sander@brixit:~$

... coredump because of too little parameters, which are used by the byref() ... ?
blackntan
Newbie
Newbie
Posts: 16
Joined: March 21st, 2021, 7:16 am

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by blackntan »

hmmm ... i'm not getting a coredump, i am JUST running the ```linux_disk_free_clib_statfs``` though .. are you sure it's coming from linux_disk_free_clib_statfs?
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by sander »

@blackntan

I've created the SABnzbd code that fixes the >4TB problem on Mac

On https://github.com/sabnzbd/sabnzbd/acti ... /690543972 , go to the bottom of the page ... do you see "macOS binary (not notarized)
14.1 MB".
Can you download it? It seems that is only possible if you have a github account

If so, can you run it, and report back if the Free Space is now correct?
blackntan
Newbie
Newbie
Posts: 16
Joined: March 21st, 2021, 7:16 am

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by blackntan »

works like a champ!
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by sander »

blackntan wrote: March 26th, 2021, 2:34 pm works like a champ!
Great!

... because of your knowledge, can you keep an eye on the pull request I made based on this thread: https://github.com/sabnzbd/sabnzbd/pull/1838 ? If you have feedback, please give it there.
blackntan
Newbie
Newbie
Posts: 16
Joined: March 21st, 2021, 7:16 am

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by blackntan »

Absolutely. Thanks!
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by sander »

@blackntan

We would like to have a bug report at python, or at least discuss this problem on the python mailing list.

I think this is needed
- newest, uptodate MacOS ("Big Sur"?), with newest python = python 3.9.2
- disk >4TB
- output of "df -m /"
- output of a few lines of python3:


>>> import os
>>> s = os.statvfs("/")
>>> s.f_bavail * s.f_frsize / 1024**2


In both cases, replace "/" with the path to your Big Drive.


As I haven't got such a Mac, could you do it?

It would be handiest for me, if we can communicate in https://github.com/sabnzbd/sabnzbd/issues/1837
blackntan
Newbie
Newbie
Posts: 16
Joined: March 21st, 2021, 7:16 am

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by blackntan »

so, something like this?

Code: Select all

#!/usr/bin/env python3
import os, sys

def disk_free_os_df(directory):
        # On any any POSIX with any disksize, "df" is always right, but heavy as it needs a process
        directory = '"' + directory + '"'
        cmd = "df -m " + directory  # show in MB
        for thisline in os.popen(cmd).readlines():
                if not thisline.startswith("Filesystem"):
                        _, df_blocks_MB, _, df_available_MB = thisline.split()[:4]
                        df_blocks_MB = int(df_blocks_MB) # blocks of 1MB, so blocks is MBs
                        df_available_MB = int(df_available_MB)
                        break # line found, so we're done
        return df_blocks_MB, df_available_MB

if (len(sys.argv) < 2):
        print("pass a directory")
        exit()
print("checking diskspace for ",sys.argv[1])
df = disk_free_os_df(sys.argv[1])
print("df free",df[1])
s = os.statvfs(sys.argv[1])
avail = s.f_bavail * s.f_frsize / 1024**2
print("statvfs free",avail)
output

Code: Select all

checking diskspace for  /Volumes/Big_One/
df free 35649357
statvfs free 2094925.87109375
User avatar
safihre
Administrator
Administrator
Posts: 5338
Joined: April 30th, 2015, 7:35 am
Contact:

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by safihre »

The python bug, for reference: https://bugs.python.org/issue43638

@blackntan thanks for your input, we will add the workaround to the next release :)
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
Post Reply