Real time validation of post consistency & par2 check

[Anon_]

Sabnzbdplus has this somewhat.  In alt.binz, on the "download queue" tab you can see if posts are missing articles as you download.  In sabnzbd you can view similar information in the different skins (smpl "warnings" etc).  However, I am not aware of the ability to view a consistency check of the posts/files like you can view in alt.binz on the "par2" tab.

As alt.binz implements it, you have the ability to determine the degree of post consistency.   You can then make a judgement call as to whether or not there is a sufficient degree of par2 files to overcome poor USP service. This can save much unnecessary bandwidth should a USP have inconsistent or corrupt post retention. 

[shypike]

Alt.binz does this at the expense of the time it takes to ask the server whether it has each article.

SABnzbd is primarily a way of automatic your downloading as much as  possible.
The assumption is that the probability of successful download is very high (with or without repairs).

If you very often need to scrape your downloads together from unreliable sources,
I'm not sure if SABnzbd is the right solution.

Personally I don't bother: less than 1 of every 100 downloads fails for me.

Nevertheless, it's something for future attention.

[Anon_]

I'm not a developer, but in principle, shouldn't replicating alt.binz par2 tab be relatively easy?

If I understand correctly 0.5 release will include prioritizing the queue according to specific files?

sabnzbd downloads small par2 first.
small files next
and then larger files.

Every file after the small par2 is then checked by par2.exe as it completes and indicates how many blocks are intact.
sabnzbd could then reflect this information on a par2 tab/page.

[Anon_]

Here's a perfect example of wasted bandwidth... =/
This could have been avoided had there been a par2 check display feature.

History rss
Purge history | Hide details
Bytes Downloaded: 21.44 GB - Bytes in History: 18.06 GB
Finished Name Status
2008-12-08 02:55:05 ***** Completed


Stage Download
    [Time-Taken]: 3 hours 45 minutes 14 seconds
    [Avg-Speed]: 364kB/s


Stage Par2
    [PAR-INFO] ****: => Not enough repair blocks left (have: 0, need: 12)

[shypike]

The required info is not in the par2 files.
The only way you know whether a download is going to be successful is if you know
how many articles you will be able to get.
So the only way is to ask the server if it has the articles (without actually downloading).
After that you can relate that figure to the amount of (potential) par2 files.

It's a complicated feature.
It will be looked at in the future, but it's not high priority now.

Furthermore it would be an atypical SABnzbd feature, exactly because it requires making
a "judgement call".

BTW, does alt.binz actually check before downloading?
You say "you can see if posts are missing articles as you download", so that would be too late.

[Anon_]

The required info to rebuild is in the par2 files.  As each file completes it could be checked to show how may blocks are intact on the file.

You can do this by using par2.exe by itself

What I was trying to convey is say you download 10 files and notice they all have a few blocks missing.  It could be a helpful indicator of judging the probability of a wasted download by comparing the blocks missing to amount of par2 files.   I can do this with alt.binz.   I can't with sabnzbd.

I could run alt.binz in wine, but I'd prefer not to if possible which is why I thought I'd request the feature here.  I'm not trying to turn this into a thread of alt.binz vs sabnzbd, but there are some features included in alt, which would be nice to see in sab.

[shypike]

I see what you mean now.
It's not a very reliable method, but I understand that it can be useful sometimes.

Actually, using par2.exe is a very bad idea and I'm sure alt.binz doesn't do it this way.
What they must be doing and what we have implemented for release 0.5.0 is on-the-fly
par2 checking. What's missing now for you is a visual indication of the health of each
downloaded file.
This has been discussed earlier in the team, because now we only spit out a bunch of
warnings about missing articles, which isn't very useful for the user.

I'll see what can be done for release 0.5.0 (due in 1-2 months).
It will definitely not be in any 0.4.x release.

Thanks for your suggestion.

[MageMinds]

Allow me to add that it would be interesting to stop the job in case the on-the-fly par2 checking determine that there wouldn't be enough parity parts to repair the job, then pause the job and send an email of the new status.

Then we could determine why the parts are missing, maybe the posts are too young or any other reasons, and maybe later we could retry the entire job by retrying to get only the posts that the server reported missing earlier and add those parts to the incomplete file already downloaded.

MageMinds

[shypike]

The fast check we do is nothing more than a consistency check, which is very cheap (in CPU terms) to do.
You know whether the file is OK or not, nothing more.
Determining the amount of missing blocks would mean doing the work of a real PAR2 verification.
That would be too expensive to do (and way too complicated as well).

Counting missing articles may be a way to do some predictions, but by the time you see that
there won't be enough par2 files, it can be a bit late in the download.
It would only be useful for extreme cases (hardly any data or none at all).

When you are mostly downloading fringe cases, maybe SABnzbd isn't the right solution for you.

Anyway, we'll give it some thought, but don't expect artificial intelligence any time soon