Page 1 of 1

Direct Unpacking RAR with passwords and slow download

Posted: April 25th, 2021, 2:37 pm
by FlangeMonkey
Hi,

I am downloading some password-protected rar files and I am getting a slow download when it starts to unpack directly.

I've noticed unrar hits 100% CPU on a single thread and I thought it was multi-threaded now. However, that's not the only odd issue:
  • if I disable "Direct Unpacking", it hits the same 100% load when extracting (and download is complete) but does not affect the speed of other downloads.
  • If I enable "Direct Unpacking" and pause a passworded download to download something without a password, download is slow until RAR is no longer at 100% load.
By my understanding, the loads are the same within the two examples above, but I get slow downloads.

The server is a ESXi VM and 6 cores from an E5-2680 v3, sab is in a docker container. Any thoughts?

Re: Direct Unpacking RAR with passwords and slow download

Posted: April 26th, 2021, 5:36 am
by safihre
The unrar process is out of our control, but it also depends a lot on the poster. Less experienced posters will forget to disable compression when packing the download, resulting in high cpu load when unpacking. It also has problems sometimes with VM mounted drives. Have you tried running it on the system directly? If that's an option?

You van also try to disable the Encrypted posts detection, it also uses lots of cpu sometimes to inspect the Rar.

Re: Direct Unpacking RAR with passwords and slow download

Posted: April 26th, 2021, 6:14 am
by FlangeMonkey
I think you may be right on the compression side of things, especially with the high CPU. Am I wrong in thinking the unrar is multi-threaded? I found some posts that refer to adding parameters to enable it on unrar, but I haven't been able to find rar parameters, and when checking syntax for unrar, it doesn't contain options for threading.

I don't get the feeling it's the container or VM, but saying that the docker container is out of my control and I will validate things locally on the VM.

The slow speeds appear to relate to "Direct Unpacking". It's odd, a single thread hitting 100% load, leaves plenty of room for other processors. I even changed the "direct_unpack_threads" to 1 for testing and get different results for speed while unpacking with "Direct Unpacking" enabled and disabled. If it was consistent I would be looking elsewhere.

Is "Direct Unpacking" doing something different under the hood?

Re: Direct Unpacking RAR with passwords and slow download

Posted: April 27th, 2021, 1:08 am
by safihre
Yes, it runs unrar in a incremental mode. But that shouldn't make a lot of difference..
Again, we did see this extra cpu usage before on VM hosts where waiting for disk IO was really causing cpu spikes for unrar. Something with the disk virtualization isn't playing well with unrar for some reason.
And indeed as far as I know unrar is not multi threaded, maybe only for compression, not unpacking.

Re: Direct Unpacking RAR with passwords and slow download

Posted: April 27th, 2021, 4:24 am
by FlangeMonkey
Thanks for the info, much appreciated.

I've been running this setup for years, which is why I'm not pointing at it, the VM's themselves are not using virtual disks they are NFS mounts. If it was an underlying disk issue, which I think is a valid consideration, it doesn't explain the oddity in speed when "direct unpacking" is enabled and disabled.

I'll see what I can do to change the storage side of things for validation.

Thanks for your help,