How to set article cache to more than 4G (or is there a better way)?

Get help with all aspects of SABnzbd
Forum rules
Help us help you:
  • Are you using the latest stable version of SABnzbd? Downloads page.
  • Tell us what system you run SABnzbd on.
  • Adhere to the forum rules.
  • Do you experience problems during downloading?
    Check your connection in Status and Interface settings window.
    Use Test Server in Config > Servers.
    We will probably ask you to do a test using only basic settings.
  • Do you experience problems during repair or unpacking?
    Enable +Debug logging in the Status and Interface settings window and share the relevant parts of the log here using [ code ] sections.
Post Reply
herrde
Newbie
Newbie
Posts: 13
Joined: May 17th, 2017, 3:50 pm

How to set article cache to more than 4G (or is there a better way)?

Post by herrde »

Hi everybody,

I have quite a fast internet connection (Gigabit speed) that gives me approx. 110 MB/s in SABnzbd (latest stable version on latest macOS 10.14.3, iMac late 2015 with i5 CPU and 24 GB RAM). I have set the article cache to 4G which I believe is maximum (higher inputs still end up with just 4G being used as shown in status window). All is fine when queuing up downloads with file sizes of 2-5 GB each. Trouble starts when I download large files ( > about 25 GB).
Then, the download starts at about 110 MB/s and stays there for a couple of minutes, all cool. Status windows shows that only a couple of hundred megabytes of the article cache are being used, so I believe that downloading, decoding, assembling and unpacking (yes, direct unpacking is enabled) are working just fine - for the first couple of minutes. Then, the article cache usage slowly but steadily rises (why?) until it maxes out at 4G (at this point, usually about 25 GB have been downloaded). Of course from that point on, download speed drops to something like 30-40 MB/s or lower and my machine becomes more and more unresponsive as it is busy with disk caching (I think).
I have found out that disabling direct unpacking actually doesn't improve the situation as a whole. It delays the point at which the article cache maxes out, sure, but on the other hand, I lose the advantage of already having a big part of my download unpacked when downloading finishes which is becoming more beneficial the larger the downloaded file is.

So my questions are:

1. Why is everything running fine (with nice and low article cache usage) for the first few minutes before things start to go sideways? What is changing after these first few minutes?
2. As I have 24 GB RAM installed in my machine, is there any way to increase the article cache size to more than 4G (I'd like to try 8G)?
3. What's the recommended number of connections with my setup? I don't know how many are actually set at the moment (as I am currently at work), but is there a general hint/formula?

Thanks a lot for your help in advance!
User avatar
safihre
Administrator
Administrator
Posts: 5338
Joined: April 30th, 2015, 7:35 am
Contact:

Re: How to set article cache to more than 4G (or is there a better way)?

Post by safihre »

The problem seems to be that SABnzbd can not keep up 100% with your speed, maybe only 95%. This also depends on article-size that the poster used, sometimes they use really small article-sizes which uses much more CPU-power to constantly switch connections.
Technically you could increase article cache more, but then when the download is near the end it just has to wait for the CPU to finish decoding, there's just no point to fill it more and more.

In general I would suggest to set your speed limit to something like 80MB/s and see if the article-cache stays nice and low.
I would also suggest to start with 20 connections to your server(s) and only add more if you can't hit that 80MB/s.
From there maybe increase it, see if 90MB/s is also possible without pushing it over the downloading/decoding limit.
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
herrde
Newbie
Newbie
Posts: 13
Joined: May 17th, 2017, 3:50 pm

Re: How to set article cache to more than 4G (or is there a better way)?

Post by herrde »

OK, thanks for your help, I'll try decreasing the number of connections first. I think I actually have 30 or even 50 connections set...

Say I wanted to play around with higher cache settings nevertheless, how do I get past that 4G limit?

Thanks again!
User avatar
safihre
Administrator
Administrator
Posts: 5338
Joined: April 30th, 2015, 7:35 am
Contact:

Re: How to set article cache to more than 4G (or is there a better way)?

Post by safihre »

You can't, it's a hard-coded limit because using more would just not make sense :)
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
herrde
Newbie
Newbie
Posts: 13
Joined: May 17th, 2017, 3:50 pm

Re: How to set article cache to more than 4G (or is there a better way)?

Post by herrde »

OK, after digging deeper into my problem over the weekend I was finally able to isolate it. In my case, SAB and its 50 connections to my Usenet provider was not the problem at all, neither is my CPU power (Intel i5 late 2015 which never went beyond 50-60% system load) or anything related - the bottleneck was simply my external USB3 SSD on which the temp folder was located initially. It seems that its read/write speeds of about 410 MB/s each (which I considered being totally sufficient) simply aren't enough for decoding, writing and assembling incoming Usenet files at a download rate of 110 MB/s (Gigabit connection).
I then placed the temp folder onto the SSD part (which is of course A LOT faster) of my internal iMac Fusion Drive (I separated the SSD drive from the HD drive a while ago) while keeping the target folder for the final unpacked files on the HD part et voilà - everyhing works like a charm now, even with the 50 connections that my Usenet provider allows and SABnzbd's Direct Unpacking feature enabled! And all of this at maxed out bit rate of the above mentioned 110 MB/s with target files of 60 GB and over, and without a single drop in download speed - article cache usage never goes beyond a few 100 Megabytes. Finally, my whole setup really is a lot of fun at last!
Again, thank you for your advice, I hope that my findings may help others with similar problems.

Best
User avatar
safihre
Administrator
Administrator
Posts: 5338
Joined: April 30th, 2015, 7:35 am
Contact:

Re: How to set article cache to more than 4G (or is there a better way)?

Post by safihre »

Aaaah, but you didn't tell us this important information ;)

I should check the website to put this more clearly somewhere. External drives are always bad for throughout, since the data is in small chunks and not the big-steady filetransfers that the advertise speeds with.
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
herrde
Newbie
Newbie
Posts: 13
Joined: May 17th, 2017, 3:50 pm

Re: How to set article cache to more than 4G (or is there a better way)?

Post by herrde »

Yes, I didn't specifically report that it was an external drive because I never would have thought that an external drive could be the problem just BECAUSE it is an external drive as long as its nominal speed (in my case read/write 410 MB/s each) is basically high enough....

It really seems one learns something new every day....!

So, does this apply to external drives connected via USB only or do e.g. external Thunderbolt drives have this problem as well? Do you know?
User avatar
safihre
Administrator
Administrator
Posts: 5338
Joined: April 30th, 2015, 7:35 am
Contact:

Re: How to set article cache to more than 4G (or is there a better way)?

Post by safihre »

Well I can't make it a rule, but the 4K read/write performance could be an identifier of the performance.
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
herrde
Newbie
Newbie
Posts: 13
Joined: May 17th, 2017, 3:50 pm

Re: How to set article cache to more than 4G (or is there a better way)?

Post by herrde »

Sorry, I'm not sure if I follow.... what do you mean by "4K read/write performance"?
User avatar
safihre
Administrator
Administrator
Posts: 5338
Joined: April 30th, 2015, 7:35 am
Contact:

Re: How to set article cache to more than 4G (or is there a better way)?

Post by safihre »

Its one of the standard speed-metrics when you run a disk-speed measurement: https://www.google.com/search?q=4K+read%2Fwrite
If you like our support, check our special newsserver deal or donate at: https://sabnzbd.org/donate
herrde
Newbie
Newbie
Posts: 13
Joined: May 17th, 2017, 3:50 pm

Re: How to set article cache to more than 4G (or is there a better way)?

Post by herrde »

Ah, ok, now I see what you mean... I'll check my drives about that some time. Thank you!
Post Reply