Page 1 of 1

Option to fill missing parts with zeroes

Posted: January 21st, 2012, 3:12 pm
by kukushka
Right now missing parts are just getting skipped. The problem is that resulting filesize is different and if .rar files have recovery record - it's not possible to use it. so please-please, if it's possible, make that option so the resulting file could be restored.
Oh, btw, i saw a topic about recovery volumes - tbh, i didn't quite understood, last i checked, winrar was shareware, not payware. Anyway auto recovery using recovery record and/or volumes would be nice, but if it's not possible in auto mode - please make it possible for a future recovery in manual mode by giving the option of keeping the filesize intact (basing on the uniformity of size of posts). (probably in the case of missing par2) thank you

Re: Option to fill missing parts with zeroes

Posted: January 21st, 2012, 3:25 pm
by shypike
Why do you think that filling zeros has any actual use?
It isn't possible to insert zeros for missing parts, because the NZB format doesn't
specify the actual amount of useful data in an article (only the raw byte count of the article).
So only extending the file with zeros would be possible and I have serious doubt that will solve anything.

Re: Option to fill missing parts with zeroes

Posted: January 21st, 2012, 4:11 pm
by kukushka
shypike wrote:Why do you think that filling zeros has any actual use?
It isn't possible to insert zeros for missing parts, because the NZB format doesn't
specify the actual amount of useful data in an article (only the raw byte count of the article).
So only extending the file with zeros would be possible and I have serious doubt that will solve anything.
..actually there's no need to make a fake post, it's just for pure binary part that will come out of it..
just made a field test - uploaded intentionally something with a missing part, downloaded nzb, figured out the missing part, doubled the previous string (with a changed part number), downloaded "custom" nzb with sabnzbd, it gave crc error check (of course), then - rar r badarch.rar - and voila, worked like a charm. ..before this with skipped part rar got stucked.
..if it can be useful, i can pm you links to nzb's or it can be easily found by searching for mvo in .boneless . the missing part has a #24

Re: Option to fill missing parts with zeroes

Posted: January 21st, 2012, 6:08 pm
by shypike
So, you're asking us to compensate for a bad design of Winrar?
It's rather lame that rar would need fillers.
Are peope actual posting self-repairing rar files instead of par2 files?
I've never seen such a post.

Re: Option to fill missing parts with zeroes

Posted: January 22nd, 2012, 1:25 am
by kukushka
shypike wrote:So, you're asking us to compensate for a bad design of Winrar?
It's rather lame that rar would need fillers.
Are peope actual posting self-repairing rar files instead of par2 files?
I've never seen such a post.
nobody's perfect. the whole concept of putting binaries into something that was intended initially for text is far from being ideal as you may imagine too. rar recovery record scenerio works with corrupted bytes instead of skipped ones - so why not help him just a little bit to do its job if there is such a possibility. sometimes people post bad par2, bad archives, recovery record generally operates with smaller blocks than par2 does, files can come from ddl where rr is quite common and just dumped to newz..
another variant - the big set of archives with rr in every volume and par2 for the whole set. a good part of archive got lost so it's beyond repair with par2. on other parts - just a few blocks are missing. rr can deal perfectly with this situation.
so there are scenarios where par2 can't help while rr can. people use rr - jbinup, warezfaq guides recommend to use it (while, au contraire, quickpar guide discourage it).par2 should be present in normal situation - i agree on that, but if for whatever reason it fails - there is a relatively easy variant to have a second line of defense.
ps didn't experimented with it yet, but probably, even if there's no rr, rar will restore more files from a "full" broken archive than from a "short" variant.

Re: Option to fill missing parts with zeroes

Posted: July 6th, 2012, 12:47 pm
by joetilbone
Make use of rar recovery thanks to you are able to recover muli-part archives of more than 4GB

Re: Option to fill missing parts with zeroes

Posted: August 12th, 2013, 6:04 pm
by kenmacd
This is kind of an old feature request, but here's how I'd find the option useful:

Sometimes there are missing articles for some file, but there also exists a torrent file using the same set of RAR files. Currently running a Bittorrent hash-check it seems to reject the files and download a lot of data. If the file was zero-filled the data would be more usable.

Re: Option to fill missing parts with zeroes

Posted: January 14th, 2017, 3:12 pm
by funberry
I'd like to add myself to the long-ignored voices asking for this feature:

Please add the option to pad empty space when assembling downloads comprising missing articles. I've used this feature in other clients and it is essential for repairing downloads using torrents in the case that par2 files are absent or insufficient.

shypike mentions in this General Help thread https://forums.sabnzbd.org/viewtopic.ph ... 73&p=86808 where a user is trying to enable this feature:
shypike wrote:I have my doubts about whether there's a significant number of people actually repairing Usenet downloads using torrents.
I certainly have no doubt that very few users are repairing sabnzbd downloads using torrents, and that is because sabnzbd "mangles" the assembly, rendering this technique mostly useless. Please excuse my insulting terminology, I do appreciate that skipping missing articles is quicker and saves disk space, and would be the preferred option for some. But that is a reason the feature should be an optional switch, not a reason to reject the feature.

A search through these forums finds a significant number of users that want this feature. I can only guess how many more are looking for it but haven't bothered to register and make a forum post.

I've even found that one user wrote a patch implementing the feature: https://bugs.launchpad.net/sabnzbd/+bug/615535

There's an indication in 2010 that the patch will be added when the article cache is no longer needed. I don't think that ever happened. Users like myself run cache_limit = -1 to disable the cache. Perhaps that could be a prerequisite to enabling this feature, if necessary.

Anyways, I hope this gets some serious reconsideration. Thanks for reading.

Re: Option to fill missing parts with zeroes

Posted: January 16th, 2017, 5:23 am
by shypike
One big issue is that we would need to guess the amount of missing bytes from the context (so, the other articles).
The NZB files only says how many raw bytes an article contains.
The amount of actual data is different and cannot be determined beforehand.

Re: Option to fill missing parts with zeroes

Posted: January 16th, 2017, 2:11 pm
by funberry
Yes, I think you've mentioned this in older threads. And the implication is that the "guess" may be wrong? If that's the case, do you have an idea of how often it might be wrong? Let's say it's right 99% of the time, surely that would be acceptable? I will note that other newsreaders (if they are using downloaded articles to guess) do make it work in my experience.

Also, the patch I linked to (https://bugs.launchpad.net/sabnzbd/+bug/615535) looks to use a different technique? Namely, writing data to the correct offset in the file based upon the =ypart directive. Would that be another solution? The author claims that it works.

Re: Option to fill missing parts with zeroes

Posted: January 16th, 2017, 2:28 pm
by safihre
We have an idea how to implement it, however, we would like to combine it with more optimizations of the writing process so it might take a while!