Direct Unpacking RAR with passwords and slow download

Get help with all aspects of SABnzbd
Forum rules
Help us help you:
  • Are you using the latest stable version of SABnzbd? Downloads page.
  • Tell us what system you run SABnzbd on.
  • Adhere to the forum rules.
  • Do you experience problems during downloading?
    Check your connection in Status and Interface settings window.
    Use Test Server in Config > Servers.
    We will probably ask you to do a test using only basic settings.
  • Do you experience problems during repair or unpacking?
    Enable +Debug logging in the Status and Interface settings window and share the relevant parts of the log here using [ code ] sections.
Post Reply
FlangeMonkey
Newbie
Newbie
Posts: 11
Joined: July 6th, 2020, 2:27 pm

Direct Unpacking RAR with passwords and slow download

Post by FlangeMonkey »

Hi,

I am downloading some password-protected rar files and I am getting a slow download when it starts to unpack directly.

I've noticed unrar hits 100% CPU on a single thread and I thought it was multi-threaded now. However, that's not the only odd issue:
  • if I disable "Direct Unpacking", it hits the same 100% load when extracting (and download is complete) but does not affect the speed of other downloads.
  • If I enable "Direct Unpacking" and pause a passworded download to download something without a password, download is slow until RAR is no longer at 100% load.
By my understanding, the loads are the same within the two examples above, but I get slow downloads.

The server is a ESXi VM and 6 cores from an E5-2680 v3, sab is in a docker container. Any thoughts?
User avatar
safihre
Administrator
Administrator
Posts: 5338
Joined: April 30th, 2015, 7:35 am
Contact:

Re: Direct Unpacking RAR with passwords and slow download

Post by safihre »

The unrar process is out of our control, but it also depends a lot on the poster. Less experienced posters will forget to disable compression when packing the download, resulting in high cpu load when unpacking. It also has problems sometimes with VM mounted drives. Have you tried running it on the system directly? If that's an option?

You van also try to disable the Encrypted posts detection, it also uses lots of cpu sometimes to inspect the Rar.
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
FlangeMonkey
Newbie
Newbie
Posts: 11
Joined: July 6th, 2020, 2:27 pm

Re: Direct Unpacking RAR with passwords and slow download

Post by FlangeMonkey »

I think you may be right on the compression side of things, especially with the high CPU. Am I wrong in thinking the unrar is multi-threaded? I found some posts that refer to adding parameters to enable it on unrar, but I haven't been able to find rar parameters, and when checking syntax for unrar, it doesn't contain options for threading.

I don't get the feeling it's the container or VM, but saying that the docker container is out of my control and I will validate things locally on the VM.

The slow speeds appear to relate to "Direct Unpacking". It's odd, a single thread hitting 100% load, leaves plenty of room for other processors. I even changed the "direct_unpack_threads" to 1 for testing and get different results for speed while unpacking with "Direct Unpacking" enabled and disabled. If it was consistent I would be looking elsewhere.

Is "Direct Unpacking" doing something different under the hood?
User avatar
safihre
Administrator
Administrator
Posts: 5338
Joined: April 30th, 2015, 7:35 am
Contact:

Re: Direct Unpacking RAR with passwords and slow download

Post by safihre »

Yes, it runs unrar in a incremental mode. But that shouldn't make a lot of difference..
Again, we did see this extra cpu usage before on VM hosts where waiting for disk IO was really causing cpu spikes for unrar. Something with the disk virtualization isn't playing well with unrar for some reason.
And indeed as far as I know unrar is not multi threaded, maybe only for compression, not unpacking.
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
FlangeMonkey
Newbie
Newbie
Posts: 11
Joined: July 6th, 2020, 2:27 pm

Re: Direct Unpacking RAR with passwords and slow download

Post by FlangeMonkey »

Thanks for the info, much appreciated.

I've been running this setup for years, which is why I'm not pointing at it, the VM's themselves are not using virtual disks they are NFS mounts. If it was an underlying disk issue, which I think is a valid consideration, it doesn't explain the oddity in speed when "direct unpacking" is enabled and disabled.

I'll see what I can do to change the storage side of things for validation.

Thanks for your help,
Post Reply