series duplicate detection not working

[diamaunt]

Hi, I'm running sabnzbd 1.1.0 from source on Solaris. series Duplicate detection is not working for me, I mentioned this in another thread, https://forums.sabnzbd.org/viewtopic.ph ... 13#p108619 but that thread ended up being mostly about unwanted file extension filtering not working, (and was fixed! thanks again!) and was reminded that more than one issue per thread isn't a good idea... so here goes.

I'm getting multiple copies of tv episodes dropped into the same directory
I have detect series/season/episode duplicates set to 'pause' but this is often not happening.

right now, I have S01E02.1.mkv and S01E02.mkv in addition to a S01E02.mkv that I'd moved into a 'watched' directory a couple of weeks ago.

It would be really nice, when these are detected and paused, if it indicated in some way that it was a series duplicate but different nzb and not an identical duplicate nzb. perhaps DUPLICATE S/S/E instead of just DUPLICATE when paused, to differentiate between a reposted nzb, and a wholly different one that is a duplicate of the episode.

[shypike]

Duplicate detection depends on sensible conventions in the NZB names.
Could you PM some examples of NZB names that aren't treated properly?

[safihre]

He send them to me and they result both in the same dupe-check string, so it should work..

Did you maybe delete your history?

EDIT: Failed jobs that are deleted are gone from the history, so we can't check those. Only ones that are actually in the history.

[diamaunt]

Duplicate detection depends on sensible conventions in the NZB names.

well, I'm confused then, what's the difference in "detect duplicate downloads" and "detect duplicate episodes in series"?
the episodes are being dropped into the same final series/SeasonEpoisode directory.

Could you PM some examples of NZB names that aren't treated properly?

done!

[diamaunt]

He send them to me and they result both in the same dupe-check string, so it should work..

Did you maybe delete your history?

EDIT: Failed jobs that are deleted are gone from the history, so we can't check those. Only ones that are actually in the history.

When you say history... do you mean the displayed history on the web interface? yes, I clean that up on a regular basis, surely you mean some other history, like a file somewhere... I thought that sometimes even with a cleared display history I get things that go into paused status marked 'duplicate'.

[safihre]

What is displayed, is what is in history.. So that's why it's not working.
It can use the files in the NZB Backup-folder to duplicate matching of truly identical NZB's, but it needs the history to do series matching.

It's not on the Config page itself, but it's mentioned (briefly, I will admit) in the Wiki that it uses the history:
https://sabnzbd.org/wiki/configuration/ ... tches#toc2

[diamaunt]

What is displayed, is what is in history.. So that's why it's not working.
It can use the files in the NZB Backup-folder to duplicate matching of truly identical NZB's, but it needs the history to do series matching.

It's not on the Config page itself, but it's mentioned (briefly, I will admit) in the Wiki that it uses the history:
https://sabnzbd.org/wiki/configuration/ ... tches#toc2

well, with respect, that bites. I've got to look at all that mess on the web page to have a very useful feature work.

[safihre]

I will update the texts on the Config page within Sabnzbd to make the requirement of history items clearer

[diamaunt]

I will update the texts on the Config page within Sabnzbd to make the requirement of history items clearer

so, what, just never delete history? does it eventually go away on it's own?

wouldn't it be better to have a program history for the program to use and a displayed 'recently downloaded' history that can be cleared?

[diamaunt]

I got got three email notices in a row about downloading the same series/season/episode of a show. one .mkv, one .mp4, one .avi, all three are in history, of course, since it all happened before I even noticed.

showname S10E07 XviD-AFG
showname S10E07 AAC MP4-Mobile
showname S10E07 HDTV x264-LOL

all downloaded into the same directory, one after another at 17:08, 17:38 and 19:40.

there's holes in the duplicate detection.

[shypike]

Only after the first job completes successfully, there will be an entry in History.
So if new jobs come in before the first one is finished, they're not seen as duplicates.
This is good and bad.
You'd want a new job of the first one failed.
On the other hand, you do end up with duplicates if they come in too fast and the first one is OK.
We need to think this through again.

[diamaunt]

Only after the first job completes successfully, there will be an entry in History.
So if new jobs come in before the first one is finished, they're not seen as duplicates.
This is good and bad.
You'd want a new job of the first one failed.
On the other hand, you do end up with duplicates if they come in too fast and the first one is OK.
We need to think this through again.

well, there was 30 minutes between the first and second, and and two hours and two minutes between the second and third.

looking at the history, the first took a minute 23 seconds, the second 30 seconds, and the third 47 seconds to download.

so it looks like the first one should have been well done and finished by the time the second one entered the queue, and certainly before the third.

[shypike]

Are you sure that the show name was exactly the same in all posts ?
The issue I describved is real (and will be tackled), although it may not be your issue.

[diamaunt]

Are you sure that the show name was exactly the same in all posts ?
The issue I describved is real (and will be tackled), although it may not be your issue.

I'm not sure what you mean about the show name? sab picked three downloads out of an rss feed, downloaded all three, and dropped all three into the same series directory, with the same season/episode numbers, all automatically and unattended, over the span of two and a half hours, noting that it only takes a couple minutes to download and process each one.

[shypike]

Suppose a post is called "The Great Unknown S01E02 blabla".
Another one is called "Great Unknown S01E02 blabla".
SABnzbd will not detect duplicates in this case: "The Great Unknown" vs. "Great Unknown".