3.2.0 running on Big Sur (Intel)
The completed download folder is located on a RAID with 36TB capacity.
The "Free Space" being reported is "347.7 GB Free Space".
I am assuming that there is a 'decimal' issue somewhere in interpreting the free space available within the completed folder on that volume.
EDIT:
I see this issue was reported in 2016 and it's still an issue (post 22737) ;-) It looks as though python's os.statvfs is not reporting the correct block count (f_blocks) for some volumes on MacOS? Should this be raised as an issue with Python?
total blocks SHOULD be: 9663676416
total blocks reported by statvfs f_blocks is 488475719
3.2.0 [7be9281] - Free Space is wrong [MacOS]
Forum rules
Help us help you:
Help us help you:
- Are you using the latest stable version of SABnzbd? Downloads page.
- Tell us what system you run SABnzbd on.
- Adhere to the forum rules.
- Do you experience problems during downloading?
Check your connection in Status and Interface settings window.
Use Test Server in Config > Servers.
We will probably ask you to do a test using only basic settings. - Do you experience problems during repair or unpacking?
Enable +Debug logging in the Status and Interface settings window and share the relevant parts of the log here using [ code ] sections.
Re: 3.2.0 [7be9281] - Free Space is wrong
Yes, indeed it should be reported to Python.. Not much we can do..
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]
Not a problem in Python, but in MacOS itself.
Read the long thread from here viewtopic.php?p=112541#p112541 . Depends on file system used: AFPS, SMB, etc.
Still not solved by Apple after 3+ years.
SAB could run "df" to get the real space, but that would need to start a separate (shell) process, which is a costly process (certainly when you run it each second as SAB does), and so not feasible.
Read the long thread from here viewtopic.php?p=112541#p112541 . Depends on file system used: AFPS, SMB, etc.
Still not solved by Apple after 3+ years.
SAB could run "df" to get the real space, but that would need to start a separate (shell) process, which is a costly process (certainly when you run it each second as SAB does), and so not feasible.
Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]
Wild idea:
At SAB startup, SAB could check both df and statvfs. The difference is the offset. SAB should always add that offset to the statvfs output. So far, so good.
But what if there is a rollover (guess: at each 4.2 TB?)? Easy: SAB should recalculate the offset. But how to detect that rollover? If statvfs goes from 100MB tot 40000MB ... is that a rollover, because the disk space decreased, ... or was disk space freed / increased ... ? I think SAB cannot know that (without running df again ... which would need to run it again each few seconds :-( )
So I doubt there is a deterministic way to work around this MacOS bug.
For reference: On Linux, on a 5TB external disk:
SAB could still compare that at startup, and if there is a mismatch, warn about the it and give the offset ... And maybe we can detect the pattern: offset at 4.2 TB, or problem with blocksize 4k versus 1k, or decimal problem ...
At SAB startup, SAB could check both df and statvfs. The difference is the offset. SAB should always add that offset to the statvfs output. So far, so good.
But what if there is a rollover (guess: at each 4.2 TB?)? Easy: SAB should recalculate the offset. But how to detect that rollover? If statvfs goes from 100MB tot 40000MB ... is that a rollover, because the disk space decreased, ... or was disk space freed / increased ... ? I think SAB cannot know that (without running df again ... which would need to run it again each few seconds :-( )
So I doubt there is a deterministic way to work around this MacOS bug.
For reference: On Linux, on a 5TB external disk:
Code: Select all
$ df /media/zeegat/
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda2 4883638268 3642537472 1241100796 75% /media/zeegat
$ python3 -c "import os; print(os.statvfs('/media/zeegat/')) "
os.statvfs_result(f_bsize=4096, f_frsize=4096, f_blocks=1220909567, f_bfree=310275199, f_bavail=310275199, f_files=1241166332, f_ffree=1241156854, f_favail=1241156854, f_flag=4096, f_namemax=255)
Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]
@blackntan
Can you run a python3 script on your Mac, with your big partition as argument? See https://raw.githubusercontent.com/sande ... tvfs_df.py
Usage
python3 diskfree_statvfs_df.py
python3 diskfree_statvfs_df.py /some/dir/on/some/drive
Can you run a python3 script on your Mac, with your big partition as argument? See https://raw.githubusercontent.com/sande ... tvfs_df.py
Usage
python3 diskfree_statvfs_df.py
python3 diskfree_statvfs_df.py /some/dir/on/some/drive
Code: Select all
sander@witte2004:~$ python3 diskfree_statvfs_df.py
dir is .
statvfs results: Partition (MB): 119171 Available (MB): 17035
df -m results: Partition (MB): 119171 Available (MB): 17036
sander@witte2004:~$ python3 diskfree_statvfs_df.py /media/sander/6665-6464/
dir is /media/sander/6665-6464/
statvfs results: Partition (MB): 121910 Available (MB): 21268
df -m results: Partition (MB): 121910 Available (MB): 21268
Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]
@sander - sure thing, here is the output ...
---
@sander & @safihre
also something to note from Apple's docs on statvfs:
"portable applications cannot depend on these functions returning any valid information at all"
I can't post the link to the docs because I'm a "new user' ...
Code: Select all
dir is /Volumes/Big_One/
statvfs results: Partition (MB): 4595384 Available (MB): 3453177
df -m results: Partition (MB): 38149815 Available (MB): 37007608
@sander & @safihre
also something to note from Apple's docs on statvfs:
"portable applications cannot depend on these functions returning any valid information at all"
I can't post the link to the docs because I'm a "new user' ...
Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]
Thanks. So the difference aka offset is ... 32 TB. And 32 TB exact up to 4 decimals: 32.00000 TB (with 1024 as base). Interesting.
I think others users have seen a offset of just 4TB ... so my hypothesis is: rollover at each 4TB, so offset is a multiple of 4TB (and 32 TB is ... 8 times 4TB)
@blackntan: could you please run my python script now and then, when your disk space has changed. Goal: verify it's indeed an offset of always a multiple of 4TB.
I will change my script to that it will print the offset, in GB and TB
I think others users have seen a offset of just 4TB ... so my hypothesis is: rollover at each 4TB, so offset is a multiple of 4TB (and 32 TB is ... 8 times 4TB)
@blackntan: could you please run my python script now and then, when your disk space has changed. Goal: verify it's indeed an offset of always a multiple of 4TB.
I will change my script to that it will print the offset, in GB and TB
Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]
@sander - sure i'll run it periodically and post.
cheers,
j
EDIT:
for a little more info, I ran a quick C program to compare statfs and statvfs on macos ... it looks like statfs returns correct info and statvfs is fubar. So maybe use a statfs python binding?
statfs ------------------
mounted on /Volumes/Big_One:
f_bsize: 4096
f_blocks: 9766352886
f_bfree: 9546613808
f_bavail: 9460083037
f_files: 390654115440
f_ffree: 390654113992
Total Space: 39065411544 KB
Available Space: 37840332148 KB
Free Space: 38186455232 KB
Capacity filled: 2%
statvfs ------------------
mounted on /Volumes/Big_One:
f_bsize: 1048576
f_blocks: 1176418294
f_frsize: 4096
f_bfree: 956679216
f_bavail: 870148445
availableFreeSpace: 3564128030720
Total Space: 4705673176 KB
Available Space: 3480593780 KB
Free Space: 3826716864 KB
Capacity filled: 20%
cheers,
j
EDIT:
for a little more info, I ran a quick C program to compare statfs and statvfs on macos ... it looks like statfs returns correct info and statvfs is fubar. So maybe use a statfs python binding?
statfs ------------------
mounted on /Volumes/Big_One:
f_bsize: 4096
f_blocks: 9766352886
f_bfree: 9546613808
f_bavail: 9460083037
f_files: 390654115440
f_ffree: 390654113992
Total Space: 39065411544 KB
Available Space: 37840332148 KB
Free Space: 38186455232 KB
Capacity filled: 2%
statvfs ------------------
mounted on /Volumes/Big_One:
f_bsize: 1048576
f_blocks: 1176418294
f_frsize: 4096
f_bfree: 956679216
f_bavail: 870148445
availableFreeSpace: 3564128030720
Total Space: 4705673176 KB
Available Space: 3480593780 KB
Free Space: 3826716864 KB
Capacity filled: 20%
Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]
Wow ... interesting!
Random google hit:
https://docs.python.org/3/library/os.html does know statvfs() but not statfs(). Pity.
One month ago we did something special in SAB: release memory by calling a OS C function (https://github.com/sabnzbd/sabnzbd/blob ... 1050-L1051)
Where malloc_trim() a glibc (gnu c library) call is, defined as https://man7.org/linux/man-pages/man3/m ... rim.3.html
Maybe this way we can call the statfs of MacOS to get the real numbers ... ?
Random google hit:
Err, "historical reasons".
Originally 4.4BSD defined a statfs() call. Linux later implemented a slightly different call with the same name. Posix standardized it between all freenix and Unix versions by defining statvfs().
statfs() is OS-specific
statvfs() is posix-conforming
As they all return slightly different structures, later ones to come along can't replace the first.
In general you should use statvfs(), the Posix one. Be careful about "use Posix" advice, though, as in some cases (pty, for example) the BSD (or whatever) one is more portable in practice.
https://docs.python.org/3/library/os.html does know statvfs() but not statfs(). Pity.
One month ago we did something special in SAB: release memory by calling a OS C function (https://github.com/sabnzbd/sabnzbd/blob ... 1050-L1051)
Code: Select all
LIBC = ctypes.CDLL("libc.so.6")
if sabnzbd.LIBC:
sabnzbd.LIBC.malloc_trim(0)
Where malloc_trim() a glibc (gnu c library) call is, defined as https://man7.org/linux/man-pages/man3/m ... rim.3.html
Maybe this way we can call the statfs of MacOS to get the real numbers ... ?
Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]
Snippets from a MacOS machine:
So statfs() is callable ... how to get info from it ... ?
time() does return info:
... maybe because statfs() returns a lot of info, the structure is different?
... https://gist.github.com/pudquick/2d8ac7 ... 97c97101ac might be useful ...
Code: Select all
server:~ sander$ python3
Python 3.7.7 (default, Mar 10 2020, 15:43:27)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> bla = ctypes.CDLL("/usr/lib/libc.dylib")
>>> dir(bla)
['_FuncPtr', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattr__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_func_flags_', '_func_restype_', '_handle', '_name']
>>> bla.malloc(0)
-1050607328
>>>
>>> bla.statfs(0)
-1
>>> bla.statvfs(0)
-1
>>> bla.dkdkdkdkdkdk(0)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ctypes/__init__.py", line 377, in __getattr__
func = self.__getitem__(name)
File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ctypes/__init__.py", line 382, in __getitem__
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(0x11b989780, dkdkdkdkdkdk): symbol not found
>>>
time() does return info:
Code: Select all
>>> bla.time(0)
1616504842
>>>
>>> bla.statvfs()
-1
>>> bla.statfs()
-1
... https://gist.github.com/pudquick/2d8ac7 ... 97c97101ac might be useful ...
Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]
In that github code, I changed a few lines
with result
Ah, great: output!
The df output as reference:
Hmmm ... I don't see the link
Code: Select all
kern = CDLL('/usr/lib/system/libsystem_kernel.dylib')
fs_info = statfs64()
# put the path to any file on the mounted file system here
root_volume = create_string_buffer('/')
result = kern.statfs(root_volume, byref(fs_info))
print(fs_info.f_bsize)
print(fs_info.f_bfree)
print(fs_info.f_bavail)
Code: Select all
server:statfs sander$ python 64bit_statfs_ctypes.py
0
1048576
243980744
The df output as reference:
Code: Select all
server:statfs sander$ df /
Filesystem 512-blocks Used Available Capacity iused ifree %iused Mounted on
/dev/disk0s2 1951845952 880594384 1070739568 46% 1755517 4293211762 0% /
Hmmm ... I don't see the link
Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]
@blackntan ... are you a programmer? Could you get the python script with statfs calls running on your system?
Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]
yes, that works ...
results in
Code: Select all
kern = CDLL('/usr/lib/system/libsystem_kernel.dylib')
fs_info = statfs32()
# put the path to any file on the mounted file system here
root_volume = create_string_buffer(b'/Volumes/Big_One/')
result = kern.statfs(root_volume, byref(fs_info))
print("f_blocks", fs_info.f_blocks)
print("f_bsize", fs_info.f_bsize)
print("f_bavail", fs_info.f_bavail)
print("Total Space KB", fs_info.f_blocks * fs_info.f_bsize / 1024)
print("Total Free Space KB", fs_info.f_bfree * fs_info.f_bsize / 1024)
results in
Code: Select all
f_blocks 9766352886
f_bsize 4096
f_bavail 9454458576
Total Space KB 39065411544.0
Total Free Space KB 38167314076.0
Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]
Code: Select all
Total Free Space KB 38167314076.0
Why do you use the statfs32(), and not the statfs64()?
@safihre ... what do you think? OK to build this into SABnzbd based on the C system call?
Or the use offset method based on shell call "df -m" (which we could run at startup, or once a hour)?
Re: 3.2.0 [7be9281] - Free Space is wrong [MacOS]
statfs64 works fine as well ...
returns
Code: Select all
kern = CDLL('/usr/lib/system/libsystem_kernel.dylib')
fs_info = statfs64()
# put the path to any file on the mounted file system here
root_volume = create_string_buffer(b'/Volumes/Big_One/')
result = kern.statfs64(root_volume, byref(fs_info))
print("f_blocks", fs_info.f_blocks)
print("f_bsize", fs_info.f_bsize)
print("f_bavail", fs_info.f_bavail)
print("Total Space KB", fs_info.f_blocks * fs_info.f_bsize / 1024)
print("Total Free Space KB", fs_info.f_bfree * fs_info.f_bsize / 1024)
Code: Select all
f_blocks 9766352886
f_bsize 4096
f_bavail 9447889254
Total Space KB 39065411544.0
Total Free Space KB 38167644900.0