Can you scrape a website with Ruby?

Yes. Ruby's open-uri (in the standard library) downloads a page in one line, and Nokogiri parses the HTML with CSS or XPath selectors. That pairing scrapes any static, server-rendered site cleanly. For pages that build their content with JavaScript, you add a headless-browser gem like Ferrum or Selenium, but for most pages Nokogiri alone is enough.

What is Nokogiri and do I need it?

Nokogiri is Ruby's standard HTML and XML parser — you install it with gem install nokogiri, then query the parsed document with the same CSS selectors you use in the browser. It's the backbone of almost every Ruby scraper, including higher-level tools built on top of it. For anything beyond trivial string matching, yes, you want it.

How do I fetch a web page in Ruby?

The simplest way is open-uri, part of the standard library: URI.open(url).read returns the page as a string, and you can pass headers like a User-Agent as arguments. For more control — custom methods, timeouts, retries — use Net::HTTP or a gem like HTTParty, but for reading a public page, open-uri is a single clean line.

Why does my Ruby scraper return nothing on a page that clearly has data?

The page is rendering its content with JavaScript after the initial HTML loads, and open-uri only fetches that initial HTML — it never runs scripts. Nokogiri then parses a page that genuinely doesn't contain your data yet. View the page source: if the values aren't there, you need a headless browser gem like Ferrum or Selenium, not more selectors.

Is Ruby good for web scraping?

For readability and quick scripts, Ruby is a joy — Nokogiri's selector API is clean and the code reads almost like the plan in your head. It's a natural fit inside a Rails app that needs to pull in outside data. Its limits are the usual ones: JavaScript-rendered pages need a browser driver, and very large concurrent crawls favor languages with lighter concurrency.

Ruby Web Scraping Tutorial (2026): Nokogiri, Step by Step

By Ashesh Dhakal · Updated July 5, 2026

Ruby has a reputation for reading like plain intent, and scraping is where that pays off — the code to turn a page into a list of records is short enough to hold in your head. The tool that makes it possible is Nokogiri, Ruby's HTML parser, paired with open-uri from the standard library for fetching. That's the entire stack for a static site. I ran every snippet against the live site on Ruby 3.3; the output you see is the output I got.

We'll scrape books.toscrape.com, a sandbox built for practice: 1,000 fictional books over 50 pages, no terms-of-service worries, no bot detection. Prefer no code? An AI website scraper does this from a pasted URL — compared honestly at the end.

What do I need to scrape a website with Ruby?

Ruby (I'm on 3.3) and one gem. Fetching is built in through open-uri; parsing needs Nokogiri:

ruby --version
gem install nokogiri

Nokogiri compiles a native extension, so the first install takes a moment, but modern versions ship precompiled binaries for common platforms and land in seconds. Once it's in, you have CSS and XPath selectors over any HTML document — the whole toolkit for static pages, no API key, no browser.

How do I download a page in Ruby?

open-uri extends URI so you can open a URL like a file. One line reads the page, and you pass headers as keyword-ish arguments:

require "open-uri"

html = URI.open(
  "https://books.toscrape.com/",
  "User-Agent" => "Mozilla/5.0 (compatible; MyScraper/1.0)"
).read

puts "#{html.length} characters"

The User-Agent header is worth including from the start — plenty of sites reject Ruby's default agent, and a browser-like string clears the common blocks. For heavier needs — retries, fine-grained timeouts, POST — reach for Net::HTTP or the HTTParty gem, but for reading a public page this is all it takes.

How do I parse HTML with Nokogiri?

Pass the HTML to Nokogiri::HTML and query with CSS selectors. Look at the page first: each book here is an <article class="product_pod">, with its title in h3 a (the title attribute), price in .price_color, and stock in .instock.availability. css returns all matches; at_css returns the first:

require "nokogiri"

doc = Nokogiri::HTML(html)

books = doc.css("article.product_pod").map do |book|
  {
    title: book.at_css("h3 a")["title"],
    price: book.at_css(".price_color").text,
    availability: book.at_css(".instock.availability").text.strip
  }
end

puts "Found #{books.length} books"

The map block reads like a description of the data: for each book, take these three fields. Reading the title from the ["title"] attribute rather than the link text sidesteps the ellipsis truncation, and .text.strip trims the whitespace padding the stock label. That found all twenty books on the page.

How do I scrape multiple pages?

Move the logic into a method returning an array of hashes, then loop the numbered pages, stopping at the first empty one:

require "open-uri"
require "nokogiri"

def scrape_page(url)
  html = URI.open(url, "User-Agent" => "Mozilla/5.0 (compatible; MyScraper/1.0)").read
  doc = Nokogiri::HTML(html)

  doc.css("article.product_pod").map do |book|
    {
      title: book.at_css("h3 a")["title"],
      price: book.at_css(".price_color").text,
      availability: book.at_css(".instock.availability").text.strip
    }
  end
end

all = []
(1..2).each do |page|
  rows = scrape_page("https://books.toscrape.com/catalogue/page-#{page}.html")
  break if rows.empty?                # ran past the last page
  puts "page #{page}: #{rows.length} books"
  all.concat(rows)
  sleep 1                             # be a polite guest
end
puts "TOTAL: #{all.length} books"

Two pages, kept light:

page 1: 20 books
page 2: 20 books
TOTAL: 40 books

Raise the range to (1..50) and you'd gather all 1,000 books — but keep the sleep 1. That one line is the line between a scraper and a nuisance; it stops you from firing requests faster than any human could, which is both impolite and the quickest route to a block.

How do I export scraped data to CSV?

Ruby's standard library includes CSV, which quotes and escapes fields for you:

require "csv"

CSV.open("books.csv", "w") do |csv|
  csv << ["title", "price", "availability"]
  all.each { |b| csv << [b[:title], b[:price], b[:availability]] }
end

The file opens cleanly in Excel or any importer:

title,price,availability
A Light in the Attic,£51.77,In stock
Tipping the Velvet,£53.74,In stock
Soumission,£50.10,In stock

Forty books, three columns, one gem — and the whole program reads top to bottom like the steps you'd describe out loud.

When does Nokogiri stop working?

The usual wall: JavaScript. open-uri fetches the HTML the server sends and never executes a script, so a page that renders its content in the browser hands Nokogiri a document that genuinely doesn't contain your data yet. Confirm by viewing source — if the values aren't there, no selector will conjure them. For those pages you bring in a headless browser through a gem like Ferrum (which drives Chrome directly) or Selenium, at a real cost in speed and setup. The other wall is serious anti-bot protection, which a user agent won't defeat.

Do you actually need to write Ruby for this?

For a static site, Nokogiri is genuinely pleasant, and inside a Rails app that needs outside data it's the obvious tool. But a scraper's cost isn't the first script — it's the upkeep. Selectors break when the markup changes. JavaScript pages pull in a browser driver you now maintain. And "re-run this every morning and tell me what changed" means building scheduling, storage, and change detection, which is the genuinely hard part.

That's the work an AI scraper absorbs. Website Scraper renders the page (JavaScript and all), extracts by understanding the content instead of a product_pod selector you babysit, and turns any scrape into a monitored alert with no infrastructure on your side. If you're writing Ruby because you like it or live in Rails, this tutorial is the right path. If you just want the data, paste the URL into the no-code scraper. The how to scrape data from a website guide covers the workflow, and the Python, PHP, and JavaScript tutorials show the same job in other languages.