NZBMegasearch, opensource NZB search aggregator
NZBMegasearch, opensource NZB search aggregator
Dear all,
tired of searching on 5 different NZB indices for finding your content? Well, I present you a solution to this problem: NZBMegasearch, a meta-search index software.
It aggregates results from your favorite indices, it is configurable with N#wsnb-based, Spotw#b-based, and many others search sites. Already after the install you will find support for 6 free indices.
It natively supports CP and SB, retrieves trending movies and shows, gives appropriate search suggestions. Many configuration and security options are available.
It is written in Python and it works on linux, mac, win. Windows binaries are also available.
It is completely open source.
https://github.com/pillone/usntssearch
I hope you will like it,
0
tired of searching on 5 different NZB indices for finding your content? Well, I present you a solution to this problem: NZBMegasearch, a meta-search index software.
It aggregates results from your favorite indices, it is configurable with N#wsnb-based, Spotw#b-based, and many others search sites. Already after the install you will find support for 6 free indices.
It natively supports CP and SB, retrieves trending movies and shows, gives appropriate search suggestions. Many configuration and security options are available.
It is written in Python and it works on linux, mac, win. Windows binaries are also available.
It is completely open source.
https://github.com/pillone/usntssearch
I hope you will like it,
0
Re: NZBMegasearch, opensource NZB search aggregator
Wow ... impressive!
EDIT: It works for me (Ubuntu)
EDIT: It works for me (Ubuntu)
Re: NZBMegasearch, opensource NZB search aggregator
hey,
thanks!
I add a screenshot to make easier to comprehend what it does:
click -> http://i.imgur.com/m4A1nP2.jpg
(trends and suggestions intentionally blanked)
thanks!
I add a screenshot to make easier to comprehend what it does:
click -> http://i.imgur.com/m4A1nP2.jpg
(trends and suggestions intentionally blanked)
Re: NZBMegasearch, opensource NZB search aggregator
Loving it cant wait to test it out, any plans for adding support for headphones? also any chance of increasing the trends from 5 to 10?
Re: NZBMegasearch, opensource NZB search aggregator
thanks!
> Headphones
I see what I can do
> Trends
No problem, i put this in the wishlist for the next release
> Headphones
I see what I can do
> Trends
No problem, i put this in the wishlist for the next release
Re: NZBMegasearch, opensource NZB search aggregator
When I start a search, the page says "this may take up to 20s". I guess NZBmegasearch then searches the different search engines. Is it possible to show some progress? For example: the search engines being used, or the stuff already found?
Re: NZBMegasearch, opensource NZB search aggregator
FWIW:
In mega2.py, I changed this line:
... and now I can access NZBmegasearch's webinterface over IPv6 (and IPv4).
You can check NZBmegasearch is listening on IPv6:
In mega2.py, I changed this line:
Code: Select all
chost = '::' ### was: '0.0.0.0'
You can check NZBmegasearch is listening on IPv6:
Code: Select all
$ netstat -apon | grep 5000
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp6 0 0 :::5000 :::* LISTEN 31010/python off (0.00/0/0)
$
Re: NZBMegasearch, opensource NZB search aggregator
BTW: I keep getting
See below. Is there a way to get rid of it?
Code: Select all
>> Failure in suggestions time data '2013-09-00' does not match format '%Y-%m-%d'
Code: Select all
$ python mega2.py
~*~ ~*~ NZBMegasearcH ~*~ ~*~
>> This is the newest version available
>> version: 0.31
>> Bootstrapping...
>> Base domain and protocol: http://192.168.1.150:5000
>> Hash table has been initialized: 30515
>> Loading modules from: /home/sander/temp/usntssearch/NZBmegasearch/SearchModules/
>> Found 8 modules
>> Importing: Newznab, aa_NZBx, ab_Findnzb, ac_NZBcc, ad_NZBclub, ae_FTDworld, af_Fanzub, ah_WMBidx
2013-05-20 19:30 MVT 4
>> Failure in suggestions time data '2013-09-00' does not match format '%Y-%m-%d'
2013-05-20 19:30 SHT 5
>> Running on port 5000
Re: NZBMegasearch, opensource NZB search aggregator
stdout says:
So: is the "15" the timeout in seconds? If so, I guess it causes a big delay?
However: a manual wget to that URL only takes 0.053s. So what is going on? Is nzbclub refusing based on UserAgent?
EDIT:
Update: I did a wireshark analysis, and nzbclub does send back a HTTP/XML with useful content within a second: see screendump. So why is NZBMegasearch giving the time-out message?
Code: Select all
http://www.nzbclub.com/nzbfeed.aspx -- HTTPConnectionPool(host='www.nzbclub.com', port=80): Request timed out. (timeout=15)
However: a manual wget to that URL only takes 0.053s. So what is going on? Is nzbclub refusing based on UserAgent?
Code: Select all
sander@flappie:~$ time wget http://www.nzbclub.com/nzbfeed.aspx
--2013-05-20 19:41:44-- http://www.nzbclub.com/nzbfeed.aspx
Resolving http://www.nzbclub.com (http://www.nzbclub.com)... 94.75.212.200
Connecting to http://www.nzbclub.com (http://www.nzbclub.com)|94.75.212.200|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 476 [text/html]
Saving to: ‘nzbfeed.aspx.1’
100%[=====================================================================================================>] 476 --.-K/s in 0s
2013-05-20 19:41:44 (35.4 MB/s) - ‘nzbfeed.aspx.1’ saved [476/476]
real 0m0.053s
user 0m0.004s
sys 0m0.004s
sander@flappie:~$
Update: I did a wireshark analysis, and nzbclub does send back a HTTP/XML with useful content within a second: see screendump. So why is NZBMegasearch giving the time-out message?
Re: NZBMegasearch, opensource NZB search aggregator
Does nzbmegasearch.log tell the wait time of the results? If so: sometimes nzbclub is within 15 sec:
... and sometimes it's not:
Does this help?
Code: Select all
2013-05-20 20:13:18,639 - SearchModule - INFO - TS http://www.newshost.co.za/ 0.0395300388336
2013-05-20 20:13:18,734 - SearchModule - INFO - TS https://nzb.cc 0.135137081146
2013-05-20 20:13:18,830 - SearchModule - INFO - TS https://nzbx.co 0.240167856216
2013-05-20 20:13:23,786 - SearchModule - INFO - TS https://findnzbs.info 5.19303393364
2013-05-20 20:13:31,933 - SearchModule - INFO - TS http://www.nzbclub.com 13.3368968964
2013-05-20 20:13:31,972 - megasearch - INFO - [ubuntu] [ubuntu ] 146 61 118
Code: Select all
2013-05-20 20:15:22,708 - SearchModule - INFO - TS http://www.newshost.co.za/ 0.0404541492462
2013-05-20 20:15:22,771 - SearchModule - INFO - TS https://nzb.cc 0.108345031738
2013-05-20 20:15:22,855 - SearchModule - INFO - TS https://nzbx.co 0.195661783218
2013-05-20 20:15:31,484 - SearchModule - INFO - TS https://findnzbs.info 8.82390189171
2013-05-20 20:15:37,687 - SearchModule - CRITICAL - http://www.nzbclub.com/nzbfeed.aspx -- HTTPConnectionPool(host='www.nzbclub.com', port=80): Request timed out. (timeout=15)
2013-05-20 20:15:37,697 - megasearch - INFO - [dwdd] [dwdd ] 26 0 105
Re: NZBMegasearch, opensource NZB search aggregator
wow,
super thanks. awesome info.
No time now, reply in few hours
super thanks. awesome info.
No time now, reply in few hours
Re: NZBMegasearch, opensource NZB search aggregator
First of all thanks for the comments
> When I start a search, the page says "this may take up to 20s". I guess NZBmegasearch then searches the different search engines. Is it possible to show some progress?
Yes, correct.
Well, it would be possible. Though it would require async communication, sorting, retreiving etc
Given that is a purely aesthetic change, I am not sure I want to spend the efforts in that direction.
Maybe something can be done without excessive code restructuring. I'll see...
regarding timeouts:
sometimes indexers take time to respond due to traffic and firewall ddos rules. For an analysis of performance you have to look at the median reply time, not just single measurements.
yes, TS is the log is the response time.
Overall, no worries it is normal, it might happen with some indices.
> Failure in suggestions time data '2013-09-00' does not match format '%Y-%m-%d'
That's just a warning, you can ignore. It just means that the thetvdb.com for one show has a record with invalid date.
No harm.
> IPv6
thanks for the ipv6 tip. It will be updated in the next version.
thanks,
0
> When I start a search, the page says "this may take up to 20s". I guess NZBmegasearch then searches the different search engines. Is it possible to show some progress?
Yes, correct.
Well, it would be possible. Though it would require async communication, sorting, retreiving etc
Given that is a purely aesthetic change, I am not sure I want to spend the efforts in that direction.
Maybe something can be done without excessive code restructuring. I'll see...
regarding timeouts:
sometimes indexers take time to respond due to traffic and firewall ddos rules. For an analysis of performance you have to look at the median reply time, not just single measurements.
yes, TS is the log is the response time.
Overall, no worries it is normal, it might happen with some indices.
> Failure in suggestions time data '2013-09-00' does not match format '%Y-%m-%d'
That's just a warning, you can ignore. It just means that the thetvdb.com for one show has a record with invalid date.
No harm.
> IPv6
thanks for the ipv6 tip. It will be updated in the next version.
thanks,
0
Re: NZBMegasearch, opensource NZB search aggregator
time-out: my point is that nzbclub sends it's reply within a second, but NZBmegasearch says 13 - 15+ seconds. So ... what is going on?
Ipv6: Oh ... be careful with that; some systems don't like/understand "::" (for example IPv4-only devices, and even some modern full OSes). Better to leave it to IPv4 as default. That's safer.
Ipv6: Oh ... be careful with that; some systems don't like/understand "::" (for example IPv4-only devices, and even some modern full OSes). Better to leave it to IPv4 as default. That's safer.
Re: NZBMegasearch, opensource NZB search aggregator
Wait, was your wireshark log from a NZBmegasearch timeout event?