Completion checker need debug help?

[sab12397]

I have about 40 articles missing and it appears SABnzbd outputs a line to the log saying each article is missing during the checking phase but it still downloads the post even though they are too many missing articles.

Could you add some debug statements to the log file on what criteria it is using to decide if the post is going to download or not?

Perhaps in the log file (made up values)

Checker Result is Total Articles = 200, Missing Articles = 40, PAR2 Articles = 10, Ratio 85%, Paused as below 95%

Does the checker also check for missing articles in PAR2 files and excludes bad PAR2 files from any calculations?

Thanks

[shypike]

It checks all articles.
The algorithm is quite simple.
The total amount of downloadable bytes (rar+par) should be a little
above the amount of bytes of the actual download (just rar).
However, there is a catch.
Par2 works with blocks, not with indovidual bytes.
Even when enough bytes are available, it still can mean that not enough
complete blocks are available. Block size and borders cannot be predicted from the nzb file.
There is no way to predict with full certainty whether a download can succeed.
Percentages just above 100% give the least reliable prediction.

[sab12397]

Thanks for the explanation.

Any chance of you outputting the results of the completion check to the log file so I can see why it is still downloading even though it is incomplete?

Just total articles checked and total articles available will help out with testing?

Thanks

[shypike]

They are logged in debug mode.
Bytes, articles are par2-blocks are three different things with little relationship.
That's what makes predictions unreliable.

[sab12397]

Thanks can you tell me what I should search for as I can't find the line.

I can see some articles have failed but I can't see a line saying what the total number of articles were and how many were missing?

If it is "working" (without the totals I have no way of easily telling) perhaps we need the ability to set the failure percentage or have a selection in a dropdown?

Instead of the current Yes/No "Check before download" change to a selection list containing Off and some appropriate values so the user can decide?

I think it needs to be a bit more stringent for me again I can't give figures as I do not know the total article count and the total missing articles as I can't find the values in the log file.

[shypike]

If SABnzbd refuses the download, it already tells you in the History entry how many articles
are missing and what the predicted completion rate is.
After that it's your guess if it's worth trying anyway.
Like I said, no prediction can ever be accurate.
If you want to search the log file, look for "Download Quality:"

[sab12397]

Thanks for trying to help I have now found the Download Quality lines and also noticed it mentions the number of articles missing in the history.

It is saying ratio 1.02 but there were 108 articles missing. With this number of missing articles I can't see how it can have a ratio over 1!

Instead of trying to calculate the byte size of the articles (the Download Quality lines indicates bytes) would it not be better to base the decision on pausing the download on the actual number of articles that are missing regardless of byte sizes?

more than 5 articles missing for every 100 articles (5% of articles missing) then pause the download

missing articles / total number of articles * 100 = missing percentage rate

This missing percentage rate could be user configurable so it means everyone can be catered for?

[shypike]

Bytes nor articles are a good prediction.
Don't fuss about articles, they won't be a better indicator.
It's all about par2 blocks, unfortunately that's a unit that cannot be derived from the NZB file.

[sab12397]

Thanks for information and I get it that you are not going to change but I will put forward my argument anyway just in case I can win you over.

Currently the check is trying to be too clever - there are 108 articles missing when normally there are probably none or very few missing. I think the check that is being carried out to decide if the item should be paused needs some adjustment.

I think the PAR2 files should not even be considered in the calculation apart from a simple question is there par2 files or not. You say yourself its all about the par2 blocks and there is no reliable calculation to work out the completion.

If no par2 files are present the item would be paused if any articles are missing.
If there are par2 files and the completion is greater than 95% (excluding par2 files) then continue otherwise let me decide if I want to risk it or not by pausing the item in the queue for me.
The completion rate percent would be user configurable.

On average par2 files may allow recovery of at least 10% of the post - setting to 95% completion rate would mean most if not all posts would still be downloaded.

Problematic posts would be paused saving time and saving money (both ISP charges and Usenet charges).

If you have a more reliable server you could set a lower completion rate.

[shypike]

I don't think your proposal will be more successful.
I have seen lots of degenerated postings.
Also take into account that the way providers react to DMCA notifications
is problematic. Some delete one article of each file or just enough
to make the post unrepairable.
That doesn't mean that there's no room for improvement.
I think it might be better to pause when there's less than 102% of the
required 100%.

[sab12397]

I think it might be better to pause when there's less than 102% of the
required 100%.

Yes that might be better as you say there is always a bit of guess work needed.

Perhaps make this a configurable setting? Even if its a "Special" setting.

[sab12397]

Quick update - tried beta5 working great now. My previous test files no longer download automatically. Many thanks.

Even better able to adjust the completion rate up a bit via a Special option just to be a bit on the safe side.

Perfection if it included a "Download should succeed, 104.1% of required 102.0% available" line in all history items regardless not just the ones with possible incomplete problems.

If taken the time to do a check display the results somewhere.

This could perhaps be displayed under the average download rate in the Download heading in the History listing?

TIA

[shypike]

"Download should succeed"

That's an odd text, considering that SABnzbd isn't going to wait but will start downloading
At this point I'm not even sure if it will remain as it is, so I'm not adding anything until after 0.7.0Final.

[sab12397]

Thanks for Beta5 for me what is currently implemented in beta 5 is working great.

Yes the item will start downloading but as you display the average download rate in the history after the download completes (this is useful information) I also think displaying the completion rate that was calculated at the check phase would also be great information to display in the history for each item.

Perhaps add an entry like "Download will succeed, 104.1% of required 102.0% available" to the history.

This will mean for each queued item I can check what the completion rate was in the history just like I can check what the average speed was.

[shypike]

Ok, I get it.