a Ruby Web Crawler, Scraper & File Harvester
[EDIT — 2023 — Unfortunately, RubyRetriever is very out-of-date and does not currently work on the latest Ruby versions, etc.]
RubyRetriever is a Web Crawler, Scraper & File Harvester. Available as a command-line executable and as a crawling framework.
URL: https://github.com/joenorton/rubyretriever
RubyRetriever (RR) uses asynchronous HTTP requests via Eventmachine & Synchrony to crawl webpages very quickly. RR also uses a Ruby implementation of the bloomfilter in order to keep track of pages it has already crawled in a memory efficient manner.
v1.3.0 Update (6/22/2014) – The major change in this update is the new PageIterator class which adds functionality for library/script usage. Now you can run custom blocks against each page during the crawl. This update also includes more tests, and other code improvements to improve modularity and testability.
v1.0 Update (6/07/2014) – Includes major code changes and a lot of bug fixes. It’s now much better in dealing with redirects, issues with the host changing, etc. Also added the SEO mode, which grabs a number of key SEO components from every page on a site. Lastly, this update was so extensive that I could not ensure backward compatibility; thus, this was update 1.0!
Mission
RubyRetriever aims to be the best command-line crawling and scraping package written in Ruby and a replacement for paid software such as Screaming Frog SEO Spider.
Roadmap?
Not sure. Feel free to offer your thoughts.
Some Potential Ideas:
- ‘freeroam mode’ – to go on cruising the net endlessly in fileharvest mode
- ‘dead-link finder’ mode
- ‘validate robots.txt’ mode – outputs the bot-exposed sitemap of your site
- more sophisticated SEO analysis? replace screaming frog?
Features
- Asynchronous HTTP Requests thru EM & Synchrony
- Bloom filter for tracking visited pages
- Supports HTTPS
- Follows 301 redirects (if to same host)
- 3 CLI modes
- Sitemap – Find all links on a website, output a valid XML sitemap, or just a CSV
- File Harvest – find all files linked to on a website, option to autodownload
- SEO – collect important SEO info from every page, output to a CSV (or STDOUT)
- Run a Custom Block on a Per-Page basis (PageIterator)
Use cases
As an Executable
With a single command at the terminal, RR can:
- Crawl your website and output a valid XML sitemap based on what it found.
- Crawl a target website and download all files of a given filetype.
- Crawl a target website, collect important SEO information such as page titles, meta descriptions and h1 tags, and write it to CSV.
Used in Custom scripts
As of version 1.3.0, with the PageIterator class you can pass a custom block that will get run against each page during a crawl, and collect the results in an array. This means you can define for yourself whatever it is you want to collect from each page during the crawl.
Help & Forks Welcome!
Getting started
Install the gem {% highlight sh %} $ gem install rubyretriever {% endhighlight %}
Using the Executable
Example: Sitemap mode
{% highlight sh %} $ rr –sitemap CSV –progress –limit 10 http://www.cnet.com {% endhighlight %} OR — SAME COMMAND
{% highlight sh %} $ rr -s csv -p -l 10 http://www.cnet.com {% endhighlight %}
This would map http://www.cnet.com until it crawled a max of 10 pages, then write the results to a CSV named cnet. Optionally, you could also use the format XML and RR would output the same URL list into a valid XML sitemap that could be submitted to Google.
Example: File Harvesting mode
{% highlight sh %} $ rr –files txt –verbose –limit 1 http://textfiles.com/programming/ {% endhighlight %} OR — SAME COMMAND
{% highlight sh %} $ rr -f txt -v -l 1 http://textfiles.com/programming/ {% endhighlight %}
This would crawl http://textfiles.com/programming/ looking for txt files for only a single page, then write out a list of filepaths to txt files to the terminal. Optionally, you could have the script autodownload all the files by adding the -a/–auto flag.
Example: SEO mode
{% highlight sh %} $ rr –seo –progress –limit 10 –out cnet-seo http://www.cnet.com {% endhighlight %} OR — SAME COMMAND
{% highlight sh %} $ rr -e -p -l 10 -o cnet-seo http://www.cnet.com {% endhighlight %}
This would go to http://www.cnet.com and crawl a max of 10 pages, during which it would collect the SEO fields on those pages – this currently means [url, page title, meta description, h1 text, h2 text]. It would then write the fields to a csv named cnet-seo.
command-line arguments
{% highlight sh %} Usage: rr [MODE FLAG] [OPTIONS] Target_URL
{% endhighlight %}
Where MODE FLAG is required, and is either:
-s, –sitemap FORMAT (only accepts CSV or XML atm)
-f, –files FILETYPE
-e, –seo
and OPTIONS is the applicable:
-o, –out FILENAME Dump fetch data as CSV
-p, –progress Outputs a progressbar
-v, –verbose Output more information
-l, –limit PAGE_LIMIT_# set a max on the total number of crawled pages
-h, –help Display this screen
Using as a Library
If you want to collect something, other than that which the executable allows, on a ‘per page’ basis then you want to use the PageIterator class. Then you can run whatever block you want against each individual page’s source code located during the crawl.
Sample Script using PageIterator
{% highlight ruby %} require ‘retriever’
opts = { ‘maxpages’ => 1 } t = Retriever::PageIterator.new(‘http://www.basecamp.com‘, opts) do |page| [page.url, page.title] end puts t.result.to_s {% endhighlight %}
{% highlight sh %}
[[“http://www.basecamp.com“, “Basecamp is everyone’s favorite project management app.”]]
{% endhighlight %} Available methods on the page iterator:
- #url – returns full URL of current page
- #source – returns raw page source code
- #title – returns html decoded verson of curent page title
- #desc – returns html decoded verson of curent page meta description
- #h1 – returns html decoded verson of current page’s h1 tag
- #h2 – returns html decoded verson of current page’s h2 tag
- #links – returns array of all links on the page
- #parse_internal – returns array of current page’s internal (same host) links
- #parse_internal_visitable – returns #parse_internal plus added filtering of only links that are visitable
- #parse_seo – returns array of current page’s html decoded title, desc, h1 and h2
- #parse_files – returns array of downloaded files of type supplied as RR options (fileharvest options)
Current Requirements
em-synchrony
ruby-progressbar
bloomfilter-rb
addressable
htmlentities
License
See included ‘LICENSE’ file. It’s the MIT license.