Newznab Installation Tutorial for Mac OSX

[rk360]

query sounds like a mysql command and also dump would go with mysql

But I have no idea how to do it :-/

[seybwob]

you need to type the query message into the main chat window. This will then open another chat window providing instructions on what to do.

[darkromz]

Great guide, and you right about that text with the email from newznabplus lol, learn something new everyday. Im just at the point now where i have run

/applications/xampp/xamppfiles/bin/php-5.3.1 update_binaries.php

I've only activated the alt.binaries.teevee for 1 day any idea how long this process normally takes?

cheers

[darkromz]

Anyone have any info on how to backfill?

[Tinology]

De5perate I'm having issues with my couch potato as well.

Under Nzb providers I've put

Host - http://localhost/
Api key - ?apikey=(then api key under my profile)

Is this correct because mine isn't fetching any releases even the newest ones I have in my index.

[bwhocking]

Anyone have any info on how to backfill?

(Assuming you don't know how...)

Select from the admin menu View Groups



Then show all active groups



Then select the group you wish to expand



Then make the changes



NOTES:
-Don't add to much at once bit by bit work better (I have stuffed this up).
-Stop normal updates while doing this (Update scripts mentioned earlier)
-Don't update to many at once I do one pernight expanding a week at a time

Open Terminal

Code: Select all

cd /Applications/xampp/xamppfiles/htdocs/nnplus/misc/update_scripts

then

Code: Select all

/applications/xampp/xamppfiles/bin/php-5.3.1 backfill.php

then

Run you normal update script

[De5perate]

you need to type the query message into the main chat window. This will then open another chat window providing instructions on what to do.

Thanks very much - worked perfectly.

I would warn people to be careful pasting their newznab ID into the chat window - if you get the code wrong then everyone will be able to see your ID. I nearly made that mistake with a rogue space at the beginning of my code from copying the text.

[De5perate]

Further progress - I haven't started backfilling yet - thought I would get all the dumped NZBs from Newznab into my collection first then fill in the gap.

So that was a painful experience as every step seems to be.

So I downloaded the torrents containing the zipped NZBs for a number of collections (Movies, TV, PC, etc) and I had read that importing more than about 100 NZBs at a time can cause problems. With over 17,000 in some folders I wasn't about to start moving 100 out at a time so I Googled and via the Newznab documentation I found the following link to some useful code:

https://gist.github.com/998386

Now the first problem is it requires Ruby to run - yet another programming language - for someone who has a minimal programming knowledge I have a vast collection of computer programming tools at my disposal now.

So to find Ruby:

Google again and I found this link:

https://rvm.io/rvm/install/

I simply typed the following line of code into terminal and it all seemed to install - I received a few error messages but it seems to work so I shall ignore those:

Code: Select all

\curl -L https://get.rvm.io | bash -s stable --ruby

Then I had to edit the ruby code to provide the correct paths for my setup. I was using a folder in my Documents folder /Documents/nnplus1 to hold all my NZBs in the folder structure as unzipped:

-PC
---PCGAMES
---PCISO
---etc
-TV
---TVDocumentary
---TVForeign
---etc

and the nzb_import.rb file was located in the nnplus1 directory too.

In the code pasted below I have edited lines 43, 44, 45, added line 46 and edited lines 93, 94, 96 and 98. The code I ended up after editing the path and providing a PHP_PATH was:

Code: Select all

#!/usr/bin/env ruby
=begin 
  Created by keyvan
  #newznab on irc.synirc.net
 
  This script is for importing a giant dump of NZB files into newznab. Be warned, it's HACKY!
  1) I expect you have downloaded a big archive full of nzbs and have extracted it.
      The directory structure should be like this:
        dump/
          MoviesHD/
            blablabla.nzb
            blablbalbalb.nzb
            fawefawef.nzb
          Music/
            blukguykg.nzb
          ... and so on ...
 
  2) You need to set a few constants: ADMINPATH, SCRIPTPATH, and IMPORTPATH
    You will find below right after this documentation.
    Just replace the strings with the paths that reflect your setup, instead of mine.
    
  3) Now you are ready to actually use the script. To simply begin importing do this:
    ./nzb_import.rb import
    
    BUT WAIT... The script has another feature, which is the ability to check its own status.
    To use this feature, you need to run the above command within screen, with logging enabled.
    When you enable logging in screen, it creates a file screenlog.0 in the current dir.
      
        Keep in mind, the script looks for screenlog.0 in your home directory ~/screenlog.0
        So if it doesn't work, that's why, and you can go change it in the script.
    
    Now when you run ./nzb_import.rb status from that dir, it will compare the log to the
    directories inside the import path, and tell you how much has been done vs how much is left.
  
                                      => CAUTION <= 
    There are parts of this script where I am using sudo. This may be undesirable for you!
    As such, please make sure to go in and remove the sudo. The only reason I have it there
    is because I imported with sudo because I didn't want to hassle with permissions.
  
  TODO: Make the script not suck. Add documentation to the methods.
=end

ADMINPATH  = "/Applications/XAMPP/xamppfiles/htdocs/nnplus/www/admin" # replace with yours
SCRIPTPATH = "/Applications/XAMPP/xamppfiles/htdocs/nnplus/misc/update_scripts" # replace with yours
IMPORTPATH = "/Users/peter/Documents/nnplus1" # replace with yours
PHP_PATH = "/Applications/XAMPP/xamppfiles/bin"

@all = []
def fill_all(path=IMPORTPATH)
  Dir.foreach(path) do |f|
    next if ['.','..'].include?f
    if File.directory? "#{path}/#{f}"
      fill_all "#{path}/#{f}"
    elsif f.include? ".nzb"
      @all << path.match(/#{IMPORTPATH.split('/').last}\/(.*)/)[1]
      break
    end
  end
end

def get_complete
  complete = []
  `cat ~/screenlog.0 | grep complete!`.split("\n").each do |f|
    complete << f.match(/#{IMPORTPATH.split('/').last}\/(.*) is/)[1]
  end
  return complete.uniq
end

def show_diff
  @all = [] ; fill_all
  complete = get_complete
  diff = @all - complete
  if complete.any?
    puts "Completed items:"
    complete.each do |f|
      puts "    #{f}"
    end
  end
  if diff.any?
    puts "Remaining items left to import:"
    diff.each do |f|
      puts "    #{f}"
    end
  else
    puts "Import appears to have completed successfully!"
  end
end

def import!(path)
  puts %{
    There are NZB's here: #{path}\n
    Will execute:\n
      #{PHP_PATH}/php-5.3.1 #{ADMINPATH}/nzb-import.php
      #{PHP_PATH}/php-5.3.1 #{SCRIPTPATH}/update_releases.php
  }
  `cd #{ADMINPATH} && sudo #{PHP_PATH}/php-5.3.1 nzb-import.php #{path}`
  puts "nzb-import.php has completed, now moving on to update_releases.php"
  `cd #{SCRIPTPATH} && sudo #{PHP_PATH}/bin/php-5.3.1 update_releases.php`
  puts "Import of #{path} is complete!"
end

def import?(path=IMPORTPATH)
  "Searching for NZB files in #{path}..."
  Dir.foreach(path) do |f|
    next if ['.','..'].include?f
    if File.directory? "#{path}/#{f}"
      import? "#{path}/#{f}"
    elsif f.include? ".nzb"
      import! path
      break
    end
  end
end

if ARGV[0] == "status"
  show_diff
elsif ARGV[0] == "import"
  if File.directory? IMPORTPATH
    puts "Beginning import of #{IMPORTPATH}"
    import? IMPORTPATH
  else
    puts "Invalid import path - must be a directory"
  end
else
  puts "Arguments: import, status"
end

Now I tried running that and received a large number of mySQL errors referring to a db.php file (I have pasted the error below):

Code: Select all

PHP Warning:  mysql_connect(): [2002] No such file or directory (trying to connect via unix:///var/mysql/mysql.sock) in /Applications/XAMPP/xamppfiles/htdocs/nnplus/www/lib/framework/db.php on line 21

So I opened the file db.php (see the path from the error above) in Textwrangler and found line 21 which is:

Code: Select all

DB::$conn = mysql_connect(DB_HOST, DB_USER, DB_PASSWORD)

So it seemed to me that I needed to replace the DB_HOST, DB_USER and DB_PASSWORD with the actual values.

If you have installed using Xampp as shown in the initial tutorial then the host should be the computer you're running on so use 127.0.0.1, the DB_USER is root and the password is blank so the line should look like this:

Code: Select all

DB::$conn = mysql_connect('127.0.0.1', 'root', '')

Save the file and that is it done.

To run the script, open terminal change directory to the location you have your nab_import.rb file and all the folders containing the nzbs to be imported and just type:

Code: Select all

./nzb_import.rb import

It should go off and start populating your nzbs.

I would recommend only putting one or two folders of NZBs in the location specified in the code as if it does go wrong it will take less time. I doesn't take that long to process the first 100 nzbs so you can check in your newznab browser window in the section you're importing to see how it is progressing. It will take a long time however to process the thousands of NZBs downloaded. You can also see progress by looking at the number of NZBs in the folder being processed - the number drops as the nzbs are moved - it takes a few seconds to process each one.

UPDATE

I have tried deleting the sudo in front of both lines that initiate the nzb import and the release update as every time I ran the script it would keep asking for a password and if I wasn't here to provide one it would just progress to the next step without doing anything. I have just run it and it didn't ask for a password so we shall see whether it will progress to the next step without wanting a password - although I have 72,000 nzbs to process so I could be in for a long wait.

The revised code:

Code: Select all

#!/usr/bin/env ruby
=begin 
  Created by keyvan
  #newznab on irc.synirc.net
 
  This script is for importing a giant dump of NZB files into newznab. Be warned, it's HACKY!
  1) I expect you have downloaded a big archive full of nzbs and have extracted it.
      The directory structure should be like this:
        dump/
          MoviesHD/
            blablabla.nzb
            blablbalbalb.nzb
            fawefawef.nzb
          Music/
            blukguykg.nzb
          ... and so on ...
 
  2) You need to set a few constants: ADMINPATH, SCRIPTPATH, and IMPORTPATH
    You will find below right after this documentation.
    Just replace the strings with the paths that reflect your setup, instead of mine.
    
  3) Now you are ready to actually use the script. To simply begin importing do this:
    ./nzb_import.rb import
    
    BUT WAIT... The script has another feature, which is the ability to check its own status.
    To use this feature, you need to run the above command within screen, with logging enabled.
    When you enable logging in screen, it creates a file screenlog.0 in the current dir.
      
        Keep in mind, the script looks for screenlog.0 in your home directory ~/screenlog.0
        So if it doesn't work, that's why, and you can go change it in the script.
    
    Now when you run ./nzb_import.rb status from that dir, it will compare the log to the
    directories inside the import path, and tell you how much has been done vs how much is left.
  
                                      => CAUTION <= 
    There are parts of this script where I am using sudo. This may be undesirable for you!
    As such, please make sure to go in and remove the sudo. The only reason I have it there
    is because I imported with sudo because I didn't want to hassle with permissions.
  
  TODO: Make the script not suck. Add documentation to the methods.
=end

ADMINPATH  = "/Applications/XAMPP/xamppfiles/htdocs/nnplus/www/admin" # replace with yours
SCRIPTPATH = "/Applications/XAMPP/xamppfiles/htdocs/nnplus/misc/update_scripts" # replace with yours
IMPORTPATH = "/Users/peter/Documents/nnplus1" # replace with yours
PHPPATH = "/Applications/XAMPP/xamppfiles/bin"

@all = []
def fill_all(path=IMPORTPATH)
  Dir.foreach(path) do |f|
    next if ['.','..'].include?f
    if File.directory? "#{path}/#{f}"
      fill_all "#{path}/#{f}"
    elsif f.include? ".nzb"
      @all << path.match(/#{IMPORTPATH.split('/').last}\/(.*)/)[1]
      break
    end
  end
end

def get_complete
  complete = []
  `cat ~/screenlog.0 | grep complete!`.split("\n").each do |f|
    complete << f.match(/#{IMPORTPATH.split('/').last}\/(.*) is/)[1]
  end
  return complete.uniq
end

def show_diff
  @all = [] ; fill_all
  complete = get_complete
  diff = @all - complete
  if complete.any?
    puts "Completed items:"
    complete.each do |f|
      puts "    #{f}"
    end
  end
  if diff.any?
    puts "Remaining items left to import:"
    diff.each do |f|
      puts "    #{f}"
    end
  else
    puts "Import appears to have completed successfully!"
  end
end

def import!(path)
  puts %{
    There are NZB's here: #{path}\n
    Will execute:\n
      #{PHPPATH}/php-5.3.1 #{ADMINPATH}/nzb-import.php
      #{PHPPATH}/php-5.3.1 #{SCRIPTPATH}/update_releases.php
  }
  `cd #{ADMINPATH} && #{PHPPATH}/php-5.3.1 nzb-import.php #{path}`
  puts "nzb-import.php has completed, now moving on to update_releases.php"
  `cd #{SCRIPTPATH} && #{PHPPATH}/php-5.3.1 update_releases.php`
  puts "Import of #{path} is complete!"
end

def import?(path=IMPORTPATH)
  "Searching for NZB files in #{path}..."
  Dir.foreach(path) do |f|
    next if ['.','..'].include?f
    if File.directory? "#{path}/#{f}"
      import? "#{path}/#{f}"
    elsif f.include? ".nzb"
      import! path
      break
    end
  end
end

if ARGV[0] == "status"
  show_diff
elsif ARGV[0] == "import"
  if File.directory? IMPORTPATH
    puts "Beginning import of #{IMPORTPATH}"
    import? IMPORTPATH
  else
    puts "Invalid import path - must be a directory"
  end
else
  puts "Arguments: import, status"
end

[De5perate]

De5perate I'm having issues with my couch potato as well.

Under Nzb providers I've put

Host - http://localhost/
Api key - ?apikey=(then api key under my profile)

Is this correct because mine isn't fetching any releases even the newest ones I have in my index.

I am not sure exactly what you have entered in your Api Key box but you should not have all that ?apikey= stuff, it should just contain a 32 digit alphanumeric code.

As for your host - mine points to another PC on my network and so I have entered it's IP address. I have not entered the final backslash (/) at the end of the address (not sure if that makes any difference). You might want to try replacing localhost with 127.0.0.1 instead.

Good luck.

De5.

[confusedxx]

What size HDD space do you need to run Newznab site and mysql db with 300 days of backfill on 12 groups? I am afraid that my hdd is too small. If it is, is there a way of adding an external Hdd that will be recognized where I can add the apps onto the external hdd?

[drummer1785]

Hey De5perate, again thank you so much for doing this. You claim to be a noob, but compared to me you're a master programmer. I followed your tutorials and everything seems to be working alright. I was trying to do the nzb dump as well. I was able to find the torrents and download just the TV and Movies files, I placed them into the same file path as you, under Documents>nnplus1. I then made two new folders, Movies and TV, and placed the zip files in the corresponding folders. Now do I unzip them into those folders? Then I tried downloading ruby and I keep getting these errors:

Libraries missing for ruby-1.9.3-p327: env. Refer to your system manual for installing libraries
Mounting remote ruby failed, trying to compile.

Error running './configure --prefix=/Users/my username/.rvm/usr', please read /Users/my username/.rvm/log/ruby-1.9.3-p327/yaml/configure.log

Error running 'make -j2', please read /Users/my username/.rvm/log/ruby-1.9.3-p327/yaml/make.log

Error running './configure --enable-shared --disable-install-doc --prefix=/Users/my username/.rvm/rubies/ruby-1.9.3-p327 --with-opt-dir=/Users/my username/.rvm/usr', please read /Users/my username/.rvm/log/ruby-1.9.3-p327/configure.log
There has been an error while running configure. Halting the installation.
ruby-1.9.3-p327 is not installed.

ruby-1.9.3-p327 is not installed.
Could not load ruby ruby-1.9.3-p327.
/Users/my username/.rvm/scripts/alias: line 111: /Users/my username/.rvm/rubies/ruby-1.9.3-p327/bin/ruby: No such file or directory

* To start using RVM you need to run `source /Users/my username/.rvm/scripts/rvm`
in all your open shell windows, in rare cases you need to reopen all shell windows.

Any ideas what I do now? I also don't know where to find the nab_import.rb file you've talked about. Any help would be greatly appreciated. Like I said I was able to do it up to this point with no problem, but I need a step by step with this stuff because I don't know what's going on. Could you help fill in the gaps? Sorry, I don't know how to do those nice code windows either.

[Ouboet]

What size HDD space do you need to run Newznab site and mysql db with 300 days of backfill on 12 groups? I am afraid that my hdd is too small. If it is, is there a way of adding an external Hdd that will be recognized where I can add the apps onto the external hdd?

I have the same question, especially now that I am looking to backfill dump archive. No way my internal hard drive will manage. Where is the sql database housed?

I downloaded the torrents containing the zipped NZBs for a number of collections (Movies, TV, PC) but pretty much filled the internal drive up when unzipping, so have moved it to external drive. Now waiting before backfilling as I am anxious that it will overload the internal drive again?

Agree that De5 is a master!

[confusedxx]

What torrents did you search for to get movie and tv collections? I am searching for nzb on isohunt, but not finding anything.

[Ouboet]

[1] for access to the dump of nzbs, type /query newznab !nzbdump YOURNEWZNABID

Got them with the instructions on the newznab chat room with above command. Follow De5's instructions earlier in this tutorial...
Cheers

[drummer1785]

Alright, I was able to figure out how to install Ruby. I had installed xcode awhile ago, probably following someone else's tutorial to do some other thing I don't understand, and if you've done that you need to do a couple more steps. I googled it and found out how.

I've extracted a couple of the files in the "movie" folder and I'm trying to run the import code and I keep getting "permission denied". I'm assuming that's because I copied your edited code where you took out the "sudo" commands. De5perate, you said that you ran the revised code without sudo and that it didn't prompt you for a password. Any idea how I get around the permission denied without the sudo commands, or should I just use the sudo command? Also should I extract everything I want uploaded to my index before running the command or can I extract some, then run it and then extract some more and run it again, etc.?