"Download speed: 0.18 KB/s" because of ... missing articles

[sander]

Hi,

Today, SAB was saying "Download speed: 0.18 KB/s" and ETA was thus sky high. I checked all things: server settings, network connecting, DSL-modem settings (I even rebooted it).

After a lot of checks, I discovered the low download speed (and high ETA) was because ... the articles were missing on the server:

"2008-01-23 17:05:54,450::WARNING::[downloader] Thread [email protected]:119: Article [email protected] missing"

So, questions:

How can I see that "missing articles" is causing a slow download speed? Can I only check that via opening the log files? Or could there be some traffic light / health indicator (=feature request)?

And why the high ETA? SAB should be able to see nothing is being downloaded, so the low download speed is fake and cannot be used.

Thanks

Sander

[switch]

The detection and handling (user feedback wise, not repairing wise) of incomplete files needs revamping.

The reported speed is actually the AVERAGE speed, not the current speed. Something I did not know either till recently. So it is not actually downloading, it is just looking for the articles, not finding them, then trying on backup servers ect. Same when starting a download, it will probably be going at full speed, but appear to ramp up in speed gradually.

A "health status" feature for items in the queue has been requested before, but may not be easy to implement without a lot of changes. It will have to be looked into further.

These features probably won't be looked into till we are making changes to the download code, which we haven't really touched so far.

[rascalli]

Health indicator might be a problem I guess

As this depends on the retention days the usenet provider has.
And sometimes it happens that even when retention is 200 days, that a file of 150 days is still missing on the servers

[sander]

Health indicator might be a problem I guess

As this depends on the retention days the usenet provider has.
And sometimes it happens that even when retention is 200 days, that a file of 150 days is still missing on the servers

I'm not asking for a *prediction* of the health, just a measurement how successful the download so far has been. Examples:

Of the 100 articles tried/downloaded so far, 99 were present, 1 was not. I would say: GREEN, or "99% healthy"
Of the 100 articles tried/downloaded so far, 90 were present, 10 were not. I would say: ORANGE, or "90% healthy"
Of the 100 articles tried/downloaded so far, 2 were present, 98 were not. I would say: RED, or "2% healthy"

As long as you which articles belong to a post / download (and SAB knows that), you can determine the health factor (I can from the log file)

Does this sound reasonable and feasible?

Sander

[sander]

The detection and handling (user feedback wise, not repairing wise) of incomplete files needs revamping.

The reported speed is actually the AVERAGE speed, not the current speed. Something I did not know either till recently. So it is not actually downloading, it is just looking for the articles, not finding them, then trying on backup servers ect.

OK, clear for the Download Speed. That's understandable. The real problem is then the sky high ETA, but is (probably) calculated by dividing the remaining MBs by the current average Download Speed. The problem is that the ETA is not true; instead of months or even years more download time, SAB will only take a few hours.

So, suggestion: calculate the ETA based on the number of *articles* handled per second. Or, mayby easier, calculate the ETA based on the amount of MBs 'handled' (not necessarily downloaded) per second.

How about this?

[sander]

As an example, here's some manual, reverse engineering Health Detection by inspecting ~/.sabnzbd/logs/sabnzbd.error.log

First example:

I'm interested in a post of which all articles have names with ...[email protected] in them. First step: I count all relevant messages: get all lines with the article name, then only with "decoder' in them, filter out "untested" (what does that mean?), and then count them:

sander@sander-amd64:~/.sabnzbd/logs$ cat sabnzbd.error.log | grep [email protected] | grep decoder | grep -vi untested | wc
  1560  18720  259606

OK: 1560 articles in the log file. Second step: how many of those articles are "missing from all":

sander@sander-amd64:~/.sabnzbd/logs$ cat sabnzbd.error.log | grep [email protected] | grep decoder | grep -vi untested | grep -i "missing from all" | wc
  1560  18720  259606
sander@sander-amd64:~/.sabnzbd/logs$

Oops: 1560 of the 1560 articles are missing, 0 are there. Health is 0/1560 or 0 %. Too bad.



Second example: a post / NZB of which all articles have ...[email protected] in their name. Same procedure:

sander@sander-amd64:~/.sabnzbd/logs$ cat sabnzbd.error.log.weg | grep [email protected] | grep decoder | grep -vi untested | wc
  5661  39632  738075
sander@sander-amd64:~/.sabnzbd/logs$ cat sabnzbd.error.log.weg | grep [email protected] | grep decoder | grep -vi untested | grep -i "missing from" | wc
      0      0      0
sander@sander-amd64:~/.sabnzbd/logs$

So: 5661 articles so far in the log files, of which 0 are missing. Health is 100%. Good!


Disclaimer: this is just a proof of concept, just to get an idea of Health Detection. HTH

[sander]

I made a script to automate the process. You should run the script from within the .sabnzbd/logs/ directory

Here's the script:

#!/bin/sh
echo "Inspecting:" $1
echo "Total number of Articles Handled, resp missing Articles:"
cat sabnzbd.error.log | grep $1 | grep decoder | grep -vi untested > kul.$$.$1
cat kul.$$.$1 | wc -l
grep -i "missing from" kul.$$.$1 | wc -l
rm kul.$$.$1
echo "Finished"

Usage: , for example './health [email protected]'. Output example:

sander@sander-amd64:~/.sabnzbd/logs$ ./health [email protected]
Inspecting: [email protected]
Total number of Articles Handled, resp missing Articles:
303
0
Finished
sander@sander-amd64:~/.sabnzbd/logs$

So, that's a healthy download so far: 303 articles, of which 0 are missing. Health 100%

HTH

[rascalli]

@Sander

Oke I see what yoiu mean now,
You want something after the download that shows the health , I was thinking of something during downloading

[sander]

@Sander

Oke I see what yoiu mean now,
You want something after the download that shows the health , I was thinking of something during downloading

Uhm, no, I mean *during* the download. The method / script I posted can be used at any moment during the download.

Suggestion for implementation:
1) Each moment SAB has handled one article (successfully or unsuccessfully) and writes to the sabnzbd.error.log, it will also increment the article-counter by one. And if the article was successful (thus not "missing from all") it will also increment the successful-article-counter.
2) SAB should continuously present "successful-article-counter / article-counter * 100%" next to RemainTotal, Age and ETA in the Queue overview.

This way, during download you can already guesstimate if the download is going to be successful.

Sander

[rascalli]

This way, during download you can already guesstimate if the download is going to be successful.

Sander

Yes & No.

As basicly you should have the PAR files to repair it , if Par files are missing than you have have a problem and is health not 100%

[switch]

It should actually be possible to check the health when importing the nzb through the use of the "HEAD" command.

Basically the server just responds with the header, not the full article. If the article is missing it gives the status code for a missing article.
Could use one connection to cycle through each of the articles and see if it is present on the server to give an fairly accurate prediction of the successfulness or "health" of a queue job.

I'm not sure how easy it would be to stick this in the code, I'm still getting to grips with the underlying download code.

[sander]

This way, during download you can already guesstimate if the download is going to be successful.

Sander

Yes & No.

As basicly you should have the PAR files to repair it , if Par files are missing than you have have a problem and is health not 100%

If the health indicator is 99%, I can expect that the PAR files will be able to repair it.

However, if the health indicator is 50%, I'm sure the PAR files won't help.


And: it's a *indicator*, not a guarantee

[switch]

It should actually be possible to check the health when importing the nzb through the use of the "HEAD" command.

Basically the server just responds with the header, not the full article. If the article is missing it gives the status code for a missing article.
Could use one connection to cycle through each of the articles and see if it is present on the server to give an fairly accurate prediction of the successfulness or "health" of a queue job.

I'm not sure how easy it would be to stick this in the code, I'm still getting to grips with the underlying download code.

Wow: the HEAD feature is cool; you can just grab the lines from the NZB and ask the newsserver with "HEAD " whether it's there. That won't take very long.

See example below.

sander@sander-amd64:~$ grep @news.tweaknews.nl /home/sander/NZBs/***\ \(2006\).imdb7.4.nzb | wc -l
7024
sander@sander-amd64:~$ telnet newszilla.xs4all.nl nntp
Trying 194.109.133.133...
Connected to newszilla.xs4all.nl.
Escape character is '^]'.
200 dreader32.news.xs4all.nl NNRP Service Ready - [email protected] (posting ok).
GROUP alt.binaries.boneless
211 2000001 1183669996 1185669996 alt.binaries.boneless
HEAD
221 0 head
Path: dreader32.news.xs4all.nl!binary6.news.xs4all.nl!transit1.news.xs4all.nl!newsgate.cistron.nl!xs4all!news2.euro.net!feeder1.cambrium.nl!feed.tweaknews.nl!not-for-mail
From: binaries4ever.net@Com (UnKnowns)
Sender: binaries4ever.net@Com
Newsgroups: alt.binaries.boneless
Subject: !  !  ! - ***.2006.LiMiTED.PAL.NL.DVDR-LoRD - !  !  ! - binaries4ever.net # 81184 "***.vol097+27.PAR2" UnKnowns (01/75)
X-Newsposter: Yenc-Power-Post-GB-12 Beta 7  12.2.7.0 (Modified POWER-POST)
Date: 22 Jan 2008 22:52:35 GMT
Lines: 5590
Message-ID:
Organization: TweakNews
X-Trace: DXC=GWjfPUo?B\jU\=7YLD[Fec
Xref: dreader32.news.xs4all.nl alt.binaries.boneless:267779716
.
QUIT
205 Transferred 1003 bytes in 0 articles, 1 group.  Disconnecting.
Connection closed by foreign host.
sander@sander-amd64:~$

[sander]

To test the HEAD option, I created a perl script that reads a NZB and with the HEAD command checks if the articles exist on the specified newsserver.

Example results of such a query:

Found: 20760 -- Not Found: 207
Results:
Articles found: 20760
Articles not found: 207
End of output

Of this post, only 207 articles are missing and 20760 articles are there, so I can be confident that the PAR2 will be able to repair.

Example of usage:

sander@sander-amd64:~/NZBs$ ./nzb-checker.pl newszilla.xs4all.nl EliGage.nzb
Start of program
newsserver is newszilla.xs4all.nl, nzbfile is EliGage.nzb
group is alt.binaries.x
...

One warning: checking of 20.000 articles does take some time, varying from five to tens of minutes.

I would like to post the perl script, but I got an error, maybe because of the square brackets in the perl code. Let me know if you would like to have the code.

[shypike]

Sander,

Looks interesting, good tip for later improvements in the downloader.
If want to, you can send the Perl code to [email protected].

Thanks for your efforts.