Page 3 of 3

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS] drive > 4TB

Posted: March 25th, 2021, 7:26 am
by sander
blackntan wrote: March 25th, 2021, 6:35 am changing the calculation to use f_bavail (instead of f_bfree) results in identical values to df
Wow! Thank you. I'll change it in my PoC code asap.

EDIT: changed in PoC code (https://github.com/sanderjo/diskfree-st ... tvfs_df.py) , and on Mac now exact match:

Code: Select all

server:diskfree-statvfs-df sander$ python3 diskfree_statvfs_df.py /
dir is /
df is always right, so: Disk size, and free (in MB): (953049, 521921)
python's os.statvfs() says (953050, 521921)
clib statfs32 says (953050, 521921)
Now the real determination
(953050, 521921)
Nice!

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Posted: March 25th, 2021, 8:45 am
by sander
blackntan wrote: March 24th, 2021, 7:42 am
so, maybe we should just use stat64 and only on macos ...
Yes, my PoC code is now MacOS only: "if MacOS and (df disk size >4TB) then use statfs32".
Maybe I have to change to statfs64, as that feels more 2021.
blackntan wrote: March 24th, 2021, 7:42 am leave the statvfs alone for linux ... it has something to do with the ctypes being used for the class's in python ... if i change the f_blocks to a c_long it actually reports the correct size in linux, but incorrect on macos ...
Did you get the code working with statfs32/statfs64 on Linux? If so, can you post it? Maybe we need it later on if the same problem pops up on Linux systems after all ... my first suspect would 32-bit Linux (raspi) with a Big Disk (>4TB) ...

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Posted: March 25th, 2021, 9:55 am
by blackntan
yes, I didn't define the entire struct ... but this works for linux:

Code: Select all

def linux_disk_free_clib_statfs(directory):
	class statfs(Structure):
	    _fields_ = [
                ("f_type",       c_int64),
                ("f_bsize",      c_int64),
                ("f_blocks",     c_ulong),
                ("f_bfree",      c_ulong),
                ("f_bavail",     c_ulong),
                ("f_files",      c_ulonglong),
                ("f_ffree",      c_ulonglong),
                ("f_frsize",     c_ulong),
		       ]

	kern = CDLL(util.find_library('c'), use_errno=True)
	root_volume = create_string_buffer(str.encode(directory))
	fs_info = statfs()
	result = kern.statfs(root_volume, byref(fs_info)) # you have to call this to get fs_info filled out
	disk_size_MB = fs_info.f_blocks * fs_info.f_bsize / 1024**2
	free_size_MB = fs_info.f_bavail  * fs_info.f_bsize / 1024**2
	return disk_size_MB, free_size_MB

Code: Select all

dir is /media/psf/Big_One/
df is always right, so: Disk size, and free (in MB): (38149816, 35697196)
python's os.statvfs() says (38149816, 35697195)
clib statfs32 says (8158310.44921875, 18201800055525.207)
clib linux statfs says (38149815.9609375, 35697195.234375)

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Posted: March 25th, 2021, 11:51 am
by sander
Thanks. Correct output on Linux, ... but as a bonus I also get a coredump :-(

Code: Select all

sander@brixit:~$ df -m /
Filesystem     1M-blocks  Used Available Use% Mounted on
/dev/sdb2         111659 43070     62875  41% /
sander@brixit:~$

Code: Select all

sander@brixit:~$ python3 diskfree_linux.py /
/
(111658.16015625, 62874.61328125)
Segmentation fault (core dumped)
sander@brixit:~$

... coredump because of too little parameters, which are used by the byref() ... ?

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Posted: March 25th, 2021, 1:50 pm
by blackntan
hmmm ... i'm not getting a coredump, i am JUST running the ```linux_disk_free_clib_statfs``` though .. are you sure it's coming from linux_disk_free_clib_statfs?

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Posted: March 26th, 2021, 11:51 am
by sander
@blackntan

I've created the SABnzbd code that fixes the >4TB problem on Mac

On https://github.com/sabnzbd/sabnzbd/acti ... /690543972 , go to the bottom of the page ... do you see "macOS binary (not notarized)
14.1 MB".
Can you download it? It seems that is only possible if you have a github account

If so, can you run it, and report back if the Free Space is now correct?

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Posted: March 26th, 2021, 2:34 pm
by blackntan
works like a champ!

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Posted: March 26th, 2021, 4:01 pm
by sander
blackntan wrote: March 26th, 2021, 2:34 pm works like a champ!
Great!

... because of your knowledge, can you keep an eye on the pull request I made based on this thread: https://github.com/sabnzbd/sabnzbd/pull/1838 ? If you have feedback, please give it there.

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Posted: March 26th, 2021, 4:12 pm
by blackntan
Absolutely. Thanks!

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Posted: March 27th, 2021, 4:21 am
by sander
@blackntan

We would like to have a bug report at python, or at least discuss this problem on the python mailing list.

I think this is needed
- newest, uptodate MacOS ("Big Sur"?), with newest python = python 3.9.2
- disk >4TB
- output of "df -m /"
- output of a few lines of python3:


>>> import os
>>> s = os.statvfs("/")
>>> s.f_bavail * s.f_frsize / 1024**2


In both cases, replace "/" with the path to your Big Drive.


As I haven't got such a Mac, could you do it?

It would be handiest for me, if we can communicate in https://github.com/sabnzbd/sabnzbd/issues/1837

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Posted: March 27th, 2021, 4:52 am
by blackntan
so, something like this?

Code: Select all

#!/usr/bin/env python3
import os, sys

def disk_free_os_df(directory):
        # On any any POSIX with any disksize, "df" is always right, but heavy as it needs a process
        directory = '"' + directory + '"'
        cmd = "df -m " + directory  # show in MB
        for thisline in os.popen(cmd).readlines():
                if not thisline.startswith("Filesystem"):
                        _, df_blocks_MB, _, df_available_MB = thisline.split()[:4]
                        df_blocks_MB = int(df_blocks_MB) # blocks of 1MB, so blocks is MBs
                        df_available_MB = int(df_available_MB)
                        break # line found, so we're done
        return df_blocks_MB, df_available_MB

if (len(sys.argv) < 2):
        print("pass a directory")
        exit()
print("checking diskspace for ",sys.argv[1])
df = disk_free_os_df(sys.argv[1])
print("df free",df[1])
s = os.statvfs(sys.argv[1])
avail = s.f_bavail * s.f_frsize / 1024**2
print("statvfs free",avail)
output

Code: Select all

checking diskspace for  /Volumes/Big_One/
df free 35649357
statvfs free 2094925.87109375

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Posted: March 27th, 2021, 8:16 am
by safihre
The python bug, for reference: https://bugs.python.org/issue43638

@blackntan thanks for your input, we will add the workaround to the next release :)