3.2.0 [7be9281] - Free Space is wrong [MacOS]

Report & discuss bugs found in SABnzbd
Forum rules
Help us help you:
  • Are you using the latest stable version of SABnzbd? Downloads page.
  • Tell us what system you run SABnzbd on.
  • Adhere to the forum rules.
  • Do you experience problems during downloading?
    Check your connection in Status and Interface settings window.
    Use Test Server in Config > Servers.
    We will probably ask you to do a test using only basic settings.
  • Do you experience problems during repair or unpacking?
    Enable +Debug logging in the Status and Interface settings window and share the relevant parts of the log here using [ code ] sections.
blackntan
Newbie
Newbie
Posts: 16
Joined: March 21st, 2021, 7:16 am

3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by blackntan »

3.2.0 running on Big Sur (Intel)
The completed download folder is located on a RAID with 36TB capacity.
The "Free Space" being reported is "347.7 GB Free Space".

I am assuming that there is a 'decimal' issue somewhere in interpreting the free space available within the completed folder on that volume.

EDIT:

I see this issue was reported in 2016 and it's still an issue (post 22737) ;-) It looks as though python's os.statvfs is not reporting the correct block count (f_blocks) for some volumes on MacOS? Should this be raised as an issue with Python?

total blocks SHOULD be: 9663676416
total blocks reported by statvfs f_blocks is 488475719
User avatar
safihre
Administrator
Administrator
Posts: 5338
Joined: April 30th, 2015, 7:35 am
Contact:

Re: 3.2.0 [7be9281] - Free Space is wrong

Post by safihre »

Yes, indeed it should be reported to Python.. Not much we can do..
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by sander »

Not a problem in Python, but in MacOS itself.

Read the long thread from here viewtopic.php?p=112541#p112541 . Depends on file system used: AFPS, SMB, etc.

Still not solved by Apple after 3+ years.

SAB could run "df" to get the real space, but that would need to start a separate (shell) process, which is a costly process (certainly when you run it each second as SAB does), and so not feasible.
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by sander »

Wild idea:

At SAB startup, SAB could check both df and statvfs. The difference is the offset. SAB should always add that offset to the statvfs output. So far, so good.

But what if there is a rollover (guess: at each 4.2 TB?)? Easy: SAB should recalculate the offset. But how to detect that rollover? If statvfs goes from 100MB tot 40000MB ... is that a rollover, because the disk space decreased, ... or was disk space freed / increased ... ? I think SAB cannot know that (without running df again ... which would need to run it again each few seconds :-( )

So I doubt there is a deterministic way to work around this MacOS bug.

For reference: On Linux, on a 5TB external disk:

Code: Select all

$ df /media/zeegat/
Filesystem      1K-blocks       Used  Available Use% Mounted on
/dev/sda2      4883638268 3642537472 1241100796  75% /media/zeegat

$ python3 -c "import os; print(os.statvfs('/media/zeegat/')) "
os.statvfs_result(f_bsize=4096, f_frsize=4096, f_blocks=1220909567, f_bfree=310275199, f_bavail=310275199, f_files=1241166332, f_ffree=1241156854, f_favail=1241156854, f_flag=4096, f_namemax=255)
SAB could still compare that at startup, and if there is a mismatch, warn about the it and give the offset ... And maybe we can detect the pattern: offset at 4.2 TB, or problem with blocksize 4k versus 1k, or decimal problem ...
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by sander »

@blackntan

Can you run a python3 script on your Mac, with your big partition as argument? See https://raw.githubusercontent.com/sande ... tvfs_df.py

Usage

python3 diskfree_statvfs_df.py

python3 diskfree_statvfs_df.py /some/dir/on/some/drive

Code: Select all

sander@witte2004:~$ python3 diskfree_statvfs_df.py 
dir is .
statvfs results: Partition (MB): 119171 Available (MB): 17035
df -m results:   Partition (MB): 119171 Available (MB): 17036

sander@witte2004:~$ python3 diskfree_statvfs_df.py /media/sander/6665-6464/
dir is /media/sander/6665-6464/
statvfs results: Partition (MB): 121910 Available (MB): 21268
df -m results:   Partition (MB): 121910 Available (MB): 21268
blackntan
Newbie
Newbie
Posts: 16
Joined: March 21st, 2021, 7:16 am

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by blackntan »

@sander - sure thing, here is the output ...

Code: Select all

dir is /Volumes/Big_One/
statvfs results: Partition (MB): 4595384 Available (MB): 3453177
df -m results:   Partition (MB): 38149815 Available (MB): 37007608
---

@sander & @safihre

also something to note from Apple's docs on statvfs:

"portable applications cannot depend on these functions returning any valid information at all"

I can't post the link to the docs because I'm a "new user' ...
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by sander »

Thanks. So the difference aka offset is ... 32 TB. And 32 TB exact up to 4 decimals: 32.00000 TB (with 1024 as base). Interesting.

I think others users have seen a offset of just 4TB ... so my hypothesis is: rollover at each 4TB, so offset is a multiple of 4TB (and 32 TB is ... 8 times 4TB)

@blackntan: could you please run my python script now and then, when your disk space has changed. Goal: verify it's indeed an offset of always a multiple of 4TB.
I will change my script to that it will print the offset, in GB and TB
blackntan
Newbie
Newbie
Posts: 16
Joined: March 21st, 2021, 7:16 am

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by blackntan »

@sander - sure i'll run it periodically and post.

cheers,
j

EDIT:
for a little more info, I ran a quick C program to compare statfs and statvfs on macos ... it looks like statfs returns correct info and statvfs is fubar. So maybe use a statfs python binding?

statfs ------------------
mounted on /Volumes/Big_One:
f_bsize: 4096
f_blocks: 9766352886
f_bfree: 9546613808
f_bavail: 9460083037
f_files: 390654115440
f_ffree: 390654113992

Total Space: 39065411544 KB
Available Space: 37840332148 KB
Free Space: 38186455232 KB
Capacity filled: 2%

statvfs ------------------
mounted on /Volumes/Big_One:
f_bsize: 1048576
f_blocks: 1176418294
f_frsize: 4096
f_bfree: 956679216
f_bavail: 870148445
availableFreeSpace: 3564128030720

Total Space: 4705673176 KB
Available Space: 3480593780 KB
Free Space: 3826716864 KB
Capacity filled: 20%
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by sander »

Wow ... interesting!

Random google hit:
Err, "historical reasons".

Originally 4.4BSD defined a statfs() call. Linux later implemented a slightly different call with the same name. Posix standardized it between all freenix and Unix versions by defining statvfs().

statfs() is OS-specific

statvfs() is posix-conforming

As they all return slightly different structures, later ones to come along can't replace the first.

In general you should use statvfs(), the Posix one. Be careful about "use Posix" advice, though, as in some cases (pty, for example) the BSD (or whatever) one is more portable in practice.

https://docs.python.org/3/library/os.html does know statvfs() but not statfs(). Pity.

One month ago we did something special in SAB: release memory by calling a OS C function (https://github.com/sabnzbd/sabnzbd/blob ... 1050-L1051)

Code: Select all

LIBC = ctypes.CDLL("libc.so.6")

        if sabnzbd.LIBC:
            sabnzbd.LIBC.malloc_trim(0)


Where malloc_trim() a glibc (gnu c library) call is, defined as https://man7.org/linux/man-pages/man3/m ... rim.3.html

Maybe this way we can call the statfs of MacOS to get the real numbers ... ?
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by sander »

Snippets from a MacOS machine:

Code: Select all

server:~ sander$ python3
Python 3.7.7 (default, Mar 10 2020, 15:43:27)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes

>>> bla = ctypes.CDLL("/usr/lib/libc.dylib")
>>> dir(bla)
['_FuncPtr', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattr__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_func_flags_', '_func_restype_', '_handle', '_name']
>>> bla.malloc(0)
-1050607328
>>>
>>> bla.statfs(0)
-1
>>> bla.statvfs(0)
-1
>>> bla.dkdkdkdkdkdk(0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ctypes/__init__.py", line 377, in __getattr__
    func = self.__getitem__(name)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ctypes/__init__.py", line 382, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(0x11b989780, dkdkdkdkdkdk): symbol not found
>>>
So statfs() is callable ... how to get info from it ... ?

time() does return info:

Code: Select all

>>> bla.time(0)
1616504842
>>>
>>> bla.statvfs()
-1
>>> bla.statfs()
-1
... maybe because statfs() returns a lot of info, the structure is different?


... https://gist.github.com/pudquick/2d8ac7 ... 97c97101ac might be useful ...
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by sander »

In that github code, I changed a few lines

Code: Select all

kern = CDLL('/usr/lib/system/libsystem_kernel.dylib')
fs_info = statfs64()
# put the path to any file on the mounted file system here
root_volume = create_string_buffer('/')
result = kern.statfs(root_volume, byref(fs_info))

print(fs_info.f_bsize)
print(fs_info.f_bfree)
print(fs_info.f_bavail)
with result

Code: Select all

server:statfs sander$ python 64bit_statfs_ctypes.py
0
1048576
243980744
Ah, great: output!

The df output as reference:

Code: Select all

server:statfs sander$ df /
Filesystem   512-blocks      Used  Available Capacity iused      ifree %iused  Mounted on
/dev/disk0s2 1951845952 880594384 1070739568    46% 1755517 4293211762    0%   /

Hmmm ... I don't see the link
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by sander »

@blackntan ... are you a programmer? Could you get the python script with statfs calls running on your system?
blackntan
Newbie
Newbie
Posts: 16
Joined: March 21st, 2021, 7:16 am

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by blackntan »

yes, that works ...

Code: Select all

kern = CDLL('/usr/lib/system/libsystem_kernel.dylib')
fs_info = statfs32()
# put the path to any file on the mounted file system here
root_volume = create_string_buffer(b'/Volumes/Big_One/')
result = kern.statfs(root_volume, byref(fs_info))
print("f_blocks", fs_info.f_blocks)
print("f_bsize", fs_info.f_bsize)
print("f_bavail", fs_info.f_bavail)
print("Total Space KB", fs_info.f_blocks * fs_info.f_bsize / 1024)
print("Total Free Space KB", fs_info.f_bfree * fs_info.f_bsize / 1024)

results in

Code: Select all

f_blocks 9766352886
f_bsize 4096
f_bavail 9454458576
Total Space KB 39065411544.0
Total Free Space KB 38167314076.0
User avatar
sander
Release Testers
Release Testers
Posts: 8811
Joined: January 22nd, 2008, 2:22 pm

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by sander »

Code: Select all

Total Free Space KB 38167314076.0
So, from python we can now see the real space free by using a system call ... Cool.

Why do you use the statfs32(), and not the statfs64()?

@safihre ... what do you think? OK to build this into SABnzbd based on the C system call?
Or the use offset method based on shell call "df -m" (which we could run at startup, or once a hour)?
blackntan
Newbie
Newbie
Posts: 16
Joined: March 21st, 2021, 7:16 am

Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]

Post by blackntan »

statfs64 works fine as well ...

Code: Select all

kern = CDLL('/usr/lib/system/libsystem_kernel.dylib')
fs_info = statfs64()
# put the path to any file on the mounted file system here
root_volume = create_string_buffer(b'/Volumes/Big_One/')
result = kern.statfs64(root_volume, byref(fs_info))
print("f_blocks", fs_info.f_blocks)
print("f_bsize", fs_info.f_bsize)
print("f_bavail", fs_info.f_bavail)
print("Total Space KB", fs_info.f_blocks * fs_info.f_bsize / 1024)
print("Total Free Space KB", fs_info.f_bfree * fs_info.f_bsize / 1024)
returns

Code: Select all

f_blocks 9766352886
f_bsize 4096
f_bavail 9447889254
Total Space KB 39065411544.0
Total Free Space KB 38167644900.0
Post Reply