3.2.0 [7be9281] - Free Space is wrong [MacOS]

Report & discuss bugs found in SABnzbd
Forum rules
Help us help you:
  • Are you using the latest stable version of SABnzbd? Downloads page.
  • Tell us what system you run SABnzbd on.
  • Adhere to the forum rules.
  • Do you experience problems during downloading?
    Check your connection in Status and Interface settings window.
    Use Test Server in Config > Servers.
    We will probably ask you to do a test using only basic settings.
  • Do you experience problems during repair or unpacking?
    Enable +Debug logging in the Status and Interface settings window and share the relevant parts of the log here using [ code ] sections.
blackntan
Newbie
Newbie
Posts: 16
Joined: March 21st, 2021, 7:16 am

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by blackntan »

This works for both linux and macos:

Code: Select all

print("args", str(sys.argv))
if (len(sys.argv) < 2):
    print("pass path")
    exit()
kern = CDLL(util.find_library('c'), use_errno=True)
fs_info = statfs32()
# put the path to any file on the mounted file system here
root_volume = create_string_buffer(str.encode(sys.argv[1]))
result = kern.statfs(root_volume, byref(fs_info))
print("f_blocks", fs_info.f_blocks)
print("f_bsize", fs_info.f_bsize)
print("f_bavail", fs_info.f_bavail)
print("Total Space KB", fs_info.f_blocks * fs_info.f_bsize / 1024)
print("Total Free Space KB", fs_info.f_bfree * fs_info.f_bsize / 1024)
User avatar
safihre
Administrator
Administrator
Posts: 5338
Joined: April 30th, 2015, 7:35 am
Contact:

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by safihre »

Sounds good to me!
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by sander »

blackntan wrote: March 23rd, 2021, 9:39 am This works for both linux and macos:

kern = CDLL(util.find_library('c'), use_errno=True)
Beautiful!
safihre wrote: March 23rd, 2021, 12:39 pm Sounds good to me!
Great!

@blackntan do you want to develop & test the code on your MacOS? And maybe send the PR to github? I think the right place is https://github.com/sabnzbd/sabnzbd/blob ... #L944-L977

I don't know what is best:

1) at startup check if there is a diff between statvfs() and the libc-call ... and if so, set a boolean to use the libc-call on subsequent filesystem.diskspace_base() calls

or

2) always used the libc-call. Cleaner code, but I'm not sure if the libc-call is heavier than the os.statvfs() ... Update: @safihre just informed me that for Windows also a win32api call is used https://github.com/sabnzbd/sabnzbd/blob ... em.py#L957 ... so this option seems acceptable.
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by sander »

@blackntan: how is your big drive mounted: what filesystem? AFS , APFS, SMB, ... ?
blackntan
Newbie
Newbie
Posts: 16
Joined: March 21st, 2021, 7:16 am

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by blackntan »

Usb 3.1 APFS
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by sander »

Some more notes:

Overflow is at 4TB, which is 2^42. But, more importantly, Blocksize is 1024 thus10 bits, so the rest is 32 bits ... so just a 32 bit overflow problem?
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by sander »

@blackntan

Can you share your full program here, or on pastebin? I tried some own code, with correct and wrong results: depending on OS, disk size and statfs32/statfs64() ...

Furthermore ... my earlier question to you: do you want to develop & test the code on your MacOS? If not, based on your code I can do it.
blackntan
Newbie
Newbie
Posts: 16
Joined: March 21st, 2021, 7:16 am

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by blackntan »

@sander - sorry, I would share via pastebin, but as a 'newbie' I can't post links ;-) I'll attempt to place it here. Also, feel free to implement, I'm completely unfamiliar with sabnzbd's source code and contribution procedures ...

Code: Select all

#!/usr/bin/env python3
from ctypes import CDLL, Structure, c_uint32, c_int64, c_uint64, c_char, create_string_buffer, byref, c_ubyte, c_int16, c_int64, c_int32, util
import sys

# when _DARWIN_FEATURE_64_BIT_INODE is not defined
class statfs32(Structure):
    _fields_ = [
                ("f_otype",       c_int16),
                ("f_oflags",      c_int16),
                ("f_bsize",       c_int64),
                ("f_iosize",      c_int64),
                ("f_blocks",      c_int64),
                ("f_bfree",       c_int64),
                ("f_bavail",      c_int64),
                ("f_files",       c_int64),
                ("f_ffree",       c_int64),
                ("f_fsid",        c_uint64),
                ("f_owner",       c_uint32),
                ("f_reserved1",   c_int16),
                ("f_type",        c_int16),
                ("f_flags",       c_int64),
                ("f_reserved2",   c_int64*2),
                ("f_fstypename",  c_char*15),
                ("f_mntonname",   c_char*90),
                ("f_mntfromname", c_char*90),
                ("f_reserved3",   c_char),
                ("f_reserved4",   c_int64*4),
               ]

# when _DARWIN_FEATURE_64_BIT_INODE is defined
class statfs64(Structure):
    _fields_ = [
                ("f_bsize",       c_uint32),
                ("f_iosize",      c_int32),
                ("f_blocks",      c_uint64),
                ("f_bfree",       c_uint64),
                ("f_bavail",      c_uint64),
                ("f_files",       c_uint64),
                ("f_ffree",       c_uint64),
                ("f_fsid",        c_uint64),
                ("f_owner",       c_uint32),
                ("f_type",        c_uint32),
                ("f_flags",       c_uint32),
                ("f_fssubtype",   c_uint32),
                ("f_fstypename",  c_char*16),
                ("f_mntonname",   c_char*1024),
                ("f_mntfromname", c_char*1024),
                ("f_reserved",    c_uint32*8),
               ]

print("args", str(sys.argv))
if (len(sys.argv) < 2):
    print("pass path")
    exit()
kern = CDLL(util.find_library('c'), use_errno=True)
fs_info = statfs32()
# put the path to any file on the mounted file system here
root_volume = create_string_buffer(str.encode(sys.argv[1]))
result = kern.statfs(root_volume, byref(fs_info))
print("f_blocks", fs_info.f_blocks)
print("f_bsize", fs_info.f_bsize)
print("f_bavail", fs_info.f_bavail)
print("Total Space KB", fs_info.f_blocks * fs_info.f_bsize / 1024)
print("Total Free Space KB", fs_info.f_bfree * fs_info.f_bsize / 1024)
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by sander »

Well, same incorrect result:

Just a small disk: 128 GB size, about 62 GB free. From Linux:

Reported correctly by df:

Code: Select all

sander@brixit:~$ df -m /
Filesystem     1M-blocks  Used Available Use% Mounted on
/dev/sdb2         111659 43182     62764  41% /
but the statfs figures are wron

Code: Select all

sander@brixit:~$ python3 blackntan_statfs.py /
args ['blackntan_statfs.py', '/']
f_blocks 17530136
f_bsize 4096
f_bavail 7299072
Total Space KB 70120544.0
Total Free Space KB 64269408.0
sander@brixit:~$

Total space incorrect.
Free space is correct


On a smaller drive in MacOS (476 GB):

Code: Select all

MackMyra:Downloads frank$ ./diskfree_statvfs_df.py
dir is .
statvfs results: Partition (MB): 476802 Available (MB): 69955
df -m results:   Partition (MB): 476802 Available (MB): 69954
no diff

system calls to statfs()

statfs32
f_blocks 122061321
f_bsize 4096
f_bavail 17908379
Total Space MB 476802.03515625
Total Free Space MB 91635.78125

statfs64
f_blocks 4096
f_bsize 0
f_bavail 122061321
Total Space MB 0.0
Total Free Space MB 0.0

So only correct info: statfs32() -> total space.






With a big disk (20TB, mounted via SMB), from MacOS, the statfs32() results are correct:

Code: Select all

MackMyra:Downloads frank$ ./diskfree_statvfs_df.py /Volumes/Frank/
dir is /Volumes/Frank/
statvfs results: Partition (MB): 991840 Available (MB): 2659543
df -m results:   Partition (MB): 21963360 Available (MB): 19436758
Diff! in MB. Disk: 20971520 Available: 16777215
Diff! in TB. Disk: 20.0 Available: 15.999999046325684

system calls to statfs()

statfs32
f_blocks 22490480640
f_bsize 1024
f_bavail 19903240832
Total Space MB 21963360.0
Total Free Space MB 19436758.625

statfs64
f_blocks 1024
f_bsize 0
f_bavail 22490480640
Total Space MB 0.0
Total Free Space MB 0.0
So ... when should and can we trust statfs()? See next post.
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by sander »

So ... when should and can we trust statfs32()?

At SAB startup: If on MacOS, run "df -m", and check if disk is bigger than 4TB. If so, set a boolean, so that on next runs, statfs32() is used to get total and free disk space.

Otherwise, keep current method, so os.statvfs()

Not beautiful, but it is a workaround for the problem.
blackntan
Newbie
Newbie
Posts: 16
Joined: March 21st, 2021, 7:16 am

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by blackntan »

I spoke too soon that it was working on linux as well I guess ... I just compared the output (with the same filesystem) on linux and on macos ...

macos:

Code: Select all

args ['./statfs.py', '/Volumes/Big_One/']
statfs result 0
f_blocks 9766352886
f_bsize 4096
f_bavail 9293011673
Total Space KB 39065411544.0
Total Free Space KB 38161007288.0
statfs64 result 0
f_blocks 9766352886
f_bsize 4096
f_bavail 9293011673
Total Space KB 39065411544.0
Total Free Space KB 38161007288.0
linux:

Code: Select all

args ['./statfs.py', '/media/psf/Big_One/']
statfs result 0
f_blocks 9293011672
f_bsize 4096
f_bavail 4096
Total Space KB 37172046688.0
Total Free Space KB 37172046688.0
statfs64 result 0
f_blocks 4096
f_bsize 2088527475
f_bavail 9293011672
Total Space KB 8354109900.0
Total Free Space KB 1.991923470015287e+16
code run:

Code: Select all

#!/usr/bin/env python3
from ctypes import CDLL, Structure, c_uint32, c_int64, c_uint64, c_char, create_string_buffer, byref, c_ubyte, c_int16, c_int64, c_int32, util
import sys

# when _DARWIN_FEATURE_64_BIT_INODE is not defined
class statfs32(Structure):
    _fields_ = [
                ("f_otype",       c_int16),
                ("f_oflags",      c_int16),
                ("f_bsize",       c_int64),
                ("f_iosize",      c_int64),
                ("f_blocks",      c_int64),
                ("f_bfree",       c_int64),
                ("f_bavail",      c_int64),
                ("f_files",       c_int64),
                ("f_ffree",       c_int64),
                ("f_fsid",        c_uint64),
                ("f_owner",       c_uint32),
                ("f_reserved1",   c_int16),
                ("f_type",        c_int16),
                ("f_flags",       c_int64),
                ("f_reserved2",   c_int64*2),
                ("f_fstypename",  c_char*15),
                ("f_mntonname",   c_char*90),
                ("f_mntfromname", c_char*90),
                ("f_reserved3",   c_char),
                ("f_reserved4",   c_int64*4),
               ]

# when _DARWIN_FEATURE_64_BIT_INODE is defined
class statfs64(Structure):
    _fields_ = [
                ("f_bsize",       c_uint32),
                ("f_iosize",      c_int32),
                ("f_blocks",      c_uint64),
                ("f_bfree",       c_uint64),
                ("f_bavail",      c_uint64),
                ("f_files",       c_uint64),
                ("f_ffree",       c_uint64),
                ("f_fsid",        c_uint64),
                ("f_owner",       c_uint32),
                ("f_type",        c_uint32),
                ("f_flags",       c_uint32),
                ("f_fssubtype",   c_uint32),
                ("f_fstypename",  c_char*16),
                ("f_mntonname",   c_char*1024),
                ("f_mntfromname", c_char*1024),
                ("f_reserved",    c_uint32*8),
               ]

print("args", str(sys.argv))
if (len(sys.argv) < 2):
    print("pass path")
    exit()
kern = CDLL(util.find_library('c'), use_errno=True)
fs_info = statfs32()
fs_info64 = statfs64()
# put the path to any file on the mounted file system here
root_volume = create_string_buffer(str.encode(sys.argv[1]))
result = kern.statfs(root_volume, byref(fs_info))
print('statfs result', result)
print("f_blocks", fs_info.f_blocks)
print("f_bsize", fs_info.f_bsize)
print("f_bavail", fs_info.f_bavail)
print("Total Space KB", fs_info.f_blocks * fs_info.f_bsize / 1024)
print("Total Free Space KB", fs_info.f_bfree * fs_info.f_bsize / 1024)
result64 = kern.statfs64(root_volume, byref(fs_info64))
print('statfs64 result', result64)
print("f_blocks", fs_info64.f_blocks)
print("f_bsize", fs_info64.f_bsize)
print("f_bavail", fs_info64.f_bavail)
print("Total Space KB", fs_info64.f_blocks * fs_info64.f_bsize / 1024)
print("Total Free Space KB", fs_info64.f_bfree * fs_info64.f_bsize / 1024)
so, maybe we should just use stat64 and only on macos ... leave the statvfs alone for linux ... it has something to do with the ctypes being used for the class's in python ... if i change the f_blocks to a c_long it actually reports the correct size in linux, but incorrect on macos ...
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by sander »

@blackntan

Can you get the code from https://github.com/sanderjo/diskfree-st ... tvfs_df.py and run it against your small and big drive? And post the output here?
blackntan
Newbie
Newbie
Posts: 16
Joined: March 21st, 2021, 7:16 am

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by blackntan »

smaller drive:

Code: Select all

dir is /Volumes/Macintosh HD
df is always right, so: Disk size, and free (in MB): (1908108, 221060)
python's os.statvfs() says (1908108, 221060)
clib statfs32 says (1908108.27734375, 1893752.52734375)

Measure time it takes for X loops: 1000
disk_free_os_df() method: --- 4.901981830596924 seconds ---
disk_free_python_statvfs() method: --- 0.0045850276947021484 seconds ---
disk_free_clib_statfs32() method: --- 0.30826807022094727 seconds ---
TEST_disk_free_clib_statfs32() method: --- 0.0038809776306152344 seconds ---
Done with measurement

Now the real determination
(1908108, 221060)
larger drive:

Code: Select all

dir is /Volumes/Big_One/
df is always right, so: Disk size, and free (in MB): (38149815, 35752669)
python's os.statvfs() says (4595384, 2198238)
clib statfs32 says (38149815.9609375, 37266608.6796875)

Measure time it takes for X loops: 1000
disk_free_os_df() method: --- 4.901307106018066 seconds ---
disk_free_python_statvfs() method: --- 0.004253864288330078 seconds ---
disk_free_clib_statfs32() method: --- 0.2923569679260254 seconds ---
TEST_disk_free_clib_statfs32() method: --- 0.0033349990844726562 seconds ---
Done with measurement

Now the real determination
(38149815.9609375, 37266608.6796875)
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by sander »

Thanks. Overall it works: the script correctly determines which method to use.

But ... the big drive:

Code: Select all

dir is /Volumes/Big_One/
df is always right, so: Disk size, and free (in MB): (38149815, 35752669)
python's os.statvfs() says (4595384, 2198238)
clib statfs32 says (38149815.9609375, 37266608.6796875)

So: df and statfs32() do agree on Disk Size 38149815 MB, but not on Available space: 35752669 MB versus 37266608 MB. That's a difference of 4%. In operational use not a problem, but not nice. Can you try to find what is going on?

... maybe a difference between Mega and Mebi bytes? 1024*1024 = 1048576 which is a 4% difference with 1000*1000. But if so, why is Disk Size completely correct? Ad the statfs32() is too high, but I already use " / 1024 **2" so 1024 as base ... so that already should yield the lowest value ... Weird.

(See https://en.wikipedia.org/wiki/Binary_prefix)
blackntan
Newbie
Newbie
Posts: 16
Joined: March 21st, 2021, 7:16 am

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by blackntan »

changing the calculation to use f_bavail (instead of f_bfree) results in identical values to df ... so df must be using f_bavail ... which is probably what we want anyway, as sabnzbd would not be running as root ...

Code: Select all

	free_size_MB = fs_info.f_bavail  * fs_info.f_bsize / 1024**2

Code: Select all

dir is /Volumes/Big_One/
df is always right, so: Disk size, and free (in MB): (38149815, 35744672)
python's os.statvfs() says (4595384, 2190240)
clib statfs32 says (38149815.9609375, 35744672.04296875)

Measure time it takes for X loops: 1000
disk_free_os_df() method: --- 4.9592719078063965 seconds ---
disk_free_python_statvfs() method: --- 0.00409388542175293 seconds ---
disk_free_clib_statfs32() method: --- 0.2955131530761719 seconds ---
TEST_disk_free_clib_statfs32() method: --- 0.0034389495849609375 seconds ---
Done with measurement

Now the real determination
(38149815.9609375, 35744668.25390625)
Post Reply