Different "Temporary Download Folder" based on category?

[OneCD]

Basically a "hacky" way of solving the issue.

But why have temp locations per-category? Users are typically only downloading a single item at-a-time anyway. An exceptional example being: you have multiple usenet providers, and the first priority one doesn't have the needed articles. This job then tries the second provider, and the next job in the queue tries the first provider.

Important to note:

• It's already possible to have the temp location on the same filesystem as the completion location.

• If multiple SAB instances are trying to download from the same provider using the same account credentials, the provider will realise something is-up, and warn you about account sharing.

• If you're using multiple SABs with multiple providers, naturally you'll get faster overall speeds, but not because the temp file locations are separate.

[sab_fan444]

Basically a "hacky" way of solving the issue.

But why have temp locations per-category?

I can’t speak for anyone else but for me it’s because of how Unraid handles different disk shares and user shares in regards to atomic moves/instant move.

So for example, I can’t have a separate user share named movies/tv shows/downloads without breaking atomic moves/instant move. Instead I need to have it all in one user share.

Might be an Unraid + Docker limitation.

[OneCD]

Why not have it all go to the same completion folder, then move it in post-processing?

[sab_fan444]

Why not have it all go to the same completion folder, then move it in post-processing?

That's actually what I am doing at the moment but in order to not break atomic moves/instant moves everything has to be in the same user share like this:

Code: Select all

media/
├── downloads/
├── movies/
├── tv_shows/
└── music/

The media user share has to be spread across all my disks. But I only want the movies folder on a specific disk and tv shows on another specific disk. In order to achieve that without manually moving files, I need to make them their own user shares like this:

Code: Select all

downloads/
movies/
tv_shows/
music/

But that breaks atomic moves/instant moves as Unraid (or Docker) will NOT allow atomic moves/instant moves between user shares, so for example moving a file from downloads to movies will be a regular copy even though it may be on the same disk (which is slow). Hopefully that makes sense now, took me a while to understand it myself. Again, to be clear, this is completely a quality of life problem.

[twisted]

I would also be interested in this feature. I am running Unraid and would like to use a RAM disk for smaller (< 15GB) files and disk for larger. Since the majority of my downloads are under this limit, it will lessen the reads and writes on my SSD cache disk. It also speeds up processing greatly.

[sab_fan444]

You don't actually need the feature requested above to use RAM, because Unraid loads the entire OS into RAM on every boot, the /tmp folder is written in RAM. Therefore you can create a folder in /tmp for SABnzbd's incomplete folder. This is how I do it on my Unraid server:

User Script plugin
First you need to use the User Scripts plugin. The script below creates a SABnzbd incomplete folder in the /tmp folder and then assigns the user nobody as owner to that folder (better than using root). This script has to run every time you boot Unraid, hence the need for the plugin.

Code: Select all

#!/bin/bash

echo "Creating folder in /tmp/ for SABnzbd..."
mkdir -p /tmp/sabnzbd/downloads/incomplete

echo "Verify creation of /tmp/ for SABnzbd..."
ls -ld /tmp/sabnzbd/downloads/incomplete

echo "Setting ownership /tmp/ to `nobody` for SABnzbd..."
chown 99:100 /tmp/sabnzbd/downloads/incomplete

echo "Verifying ownership of /tmp/ for SABnzbd..."
ls -ld /tmp/sabnzbd/downloads/incomplete

Set the script to run At Startup of Array.

compose.yaml
This is my compose.yaml for SABnzbd, change for your needs:

Code: Select all

services:
  sabnzbd:
    container_name: sabnzbd
    image: lscr.io/linuxserver/sabnzbd:4.5.5-ls238
    env_file:
      - .env
    environment:
      - PUID=99
      - PGID=100
      - UMASK=022
      - TZ=America/New_York
    volumes:
      - /mnt/user/appdata/sabnzbd:/config
      - /mnt/user/media/:/media
      - /tmp/sabnzbd/downloads/incomplete:/media/Downloads/Usenet/Incomplete
    expose:
      - 8080
    networks:
      - proxy
    restart: always
    labels:
      # Unraid
      - "net.unraid.docker.webui=https://sabnzbd.${FQDN}"
      - "net.unraid.docker.icon=https://raw.githubusercontent.com/selfhst/icons/refs/heads/main/png/sabnzbd.png"
      - "net.unraid.docker.shell=bash"
      # Traefik
      - "traefik.enable=true"
      - "traefik.docker.network=proxy"
      - "traefik.http.services.sabnzbd-service.loadbalancer.server.port=8080"
      - "traefik.http.routers.sabnzbd-secure.rule=Host(`sabnzbd.internal.${FQDN}`)"
      - "traefik.http.routers.sabnzbd-secure.entrypoints=websecure-internal"
      - "traefik.http.routers.sabnzbd-secure.service=sabnzbd-service"
      - "traefik.http.routers.sabnzbd-secure.tls=true"
      - "traefik.http.routers.sabnzbd-secure.middlewares=websecure-internal-middlewares@file,authentik-basic-auth@file"

networks:
  proxy:
    external: true

SABnzbd settings
Lastly, make sure SABnzbd settings match the path above, located here: SABnzbd Config > Folders > Temporary Download Folder

[madburg]

Noted! If it's requested a lot, we can reconsider.

Please do consider this request. It would be greatly welcomed. I believe you may not receive many public request do to the lack of technical knowledge and thus user acceptance that you have to live with it functioning as is.

The basis for the additional functionality is due too how data is treat when crossing file system boundaries; irrelevant of what OS and file system(s) are used.
The boundary here is that each hard drive has its own independent tracking table; be it a MFT, or inode.

A request to "move" a file across boundaries (from one disk to another) requires the destination file system to create a brand new entry in its own metadata table, read every bit from the source and writes it to the new physical sectors on the destination. Followed by registering this new physical locations it was stored on in the destination's metadata table. Finally, it goes back to the source metadata table and deletes the original entry, marking that old space as "free". Where clearly a request to "move" a file in the same boundary is simply updating the existing metadata table and is practically instant.

We'll throw in hardlink since we are discussing file system metadata tables. When a hardlink is applied a new name entry is created that points to an existing metadata record. Clearly why a hardlink has the very same boundary limitations. Only symlink can span multiple drive/network locations, but out of scope here.

So lets move to some examples with this understanding.

If we take any OS and and make three additional physical hard drives available to it, format the HDs with any file system the OS supports.
Can we configure Sabnzbd with three categories each category with its own path? YES we can; as you do not have to use relative folders based on "Default Base Folder" nor the "Completed" folder.

e.g.
Disk1:OS\sab\
\incomplete
\completed
\...
Disk2: media\tv
Disk3: media\movies
Disk4: media\music

Can a "completed download" of any category be moved instantaneously to one of the three HDs? NO it cannot, we are crossing boundaries.
If we move the sabnzbd "download" directories to any one of the data drives above a "completed download" move request to either of the two drives the category does not reside on is still a NO, we are crossing boundaries..

If we had an option to specify individual download directories (Temporary & Completed Download) per Category, could moves not cross boundaries? YES.
Bonus, would hardlink work for each respective categories data , YES.

e.g. (any of these combinations would work for not crossing boundaries)
Disk2:
media\tv
media\incomplete (for tv)
media\completed (for tv)

Disk3:
media\incomplete (for movies)
media\completed (for movies)
media\completed\movies

Disk4:
media\sab\incomplete (for music)
media\sab\completed (for music)
media\sab\music

Only way to accomplish this today, is with multiple sabnzbd installation, each for servicing one category. But this creates an issue with too many calls for searches.


Now let's use another example, using an OS like Unraid. This OS utilizes a custom process called shfs (consider it a custom volume manager) this is built on the native Linux FUSE framework. You will notice some refer to it as FUSE while others shfs. It is this process which runs in the userspace that determines based on the configuration set what mount point to utilize for the system call that was made. ( e.g., write(), open(), mkdir(),rename() ).

e.g.#1 make a call to copy to a disk, that will be a native linux call of "cp /mnt/diskX/" whatever your copying.

e.g.#2 make a call to copy to a share, the kernel sees that request is a path that belongs to a FUSE (shfs) mount, and redirect it to shfs to process. shfs will make the determination based on the configuration of the share what to actually do. If share was configure to use cache and cache has not run out of space, then it would call "cp /mnt/cache/share". If the share is not configure to use a cache drive/pool or out of space then it would call "cp /mnt/share/"

This is where the confusion comes into play, as well as some legitimate requests that should work, but don't because shfs is masking what it transpiring under the hood.

By introducing the ability for Sabnzbd to enable individual incomplete and complete base folders to reside with the categories their data is destined for alleviates various pain points whereby file system boundaries are crossed. A dedicated share can contain books, another for IT apps, another for a specific media type. Each processing that category optimally, which only works today when everything is all under one boundary.

Supporting these feature would allow addition shares for which we could apply different access permissions and should allow "Minimum Free Space for Completed Download Folder" to be calculated per category since each could have their own (secondary sub feature).

Thanks for the consideration.

[BobiderBob]

Hi,

I also stumbled across this issue. I currently have two different datasets for Movies and Shows, and downloads need to be moved again after completion because SABnzbd doesn’t support per-category temporary folders. I would really appreciate if this feature could be introduced.

For now, I am running two instances of SABnzbd, which works around the problem but limits the usefulness of download limits and adds extra overhead.

The underlying issue seems to be:

- Moves across different datasets result in full copy + delete operations, which can be slow for large files.
- Without per-category temporary and completed folder options, the only way to avoid this is multiple instances, which increases complexity.

Having the ability to specify individual temporary and completed download folders per category could help:

- Downloads would land directly in the target dataset without extra copying.
- It could allow per-category minimum free space calculations.
- Dedicated shares could be used for different media types, with separate access permissions if needed.

Currently, the only workaround is multiple SAB instances. Supporting per-category temp and completed folders would greatly improve usability and performance, especially for users with large media libraries spread across multiple datasets or drives.

Thanks for considering this feature!
Best regards

[Puzzled]

Some of you do not seem to be aware that most usenet content is RAR compressed. SABnzbd will download the RAR files to the incomplete dir and unpack them to the complete dir. Because of this, if you put the incomplete and complete dir on the same disk then that will usually be much slower than having them on separate disks.