Go Web Scraping Tutorial (2026): net/http + goquery, Step by Step

Go earns its place in scraping the moment volume enters the picture. A single-threaded script fetching pages one at a time is fine for forty books and painful for forty thousand; Go's goroutines turn that second case from an overnight job into a coffee break. This tutorial starts with the fundamentals — net/http from the standard library to fetch, goquery to parse — because you should understand the sequential version before you make it concurrent. I ran every snippet against the live site; the output is what I got.

We'll scrape books.toscrape.com, a sandbox built for practice: 1,000 fictional books over 50 pages, no terms-of-service worries, no bot detection. Prefer no code? An AI website scraper does this from a pasted URL, compared honestly at the end.

What do I need to scrape a website in Go?

Go itself (I'm on a recent 1.x) and one dependency for parsing. Fetching is standard library — net/http is built in. For turning HTML into something you can query with CSS selectors, add goquery:

go mod init myscraper
go get github.com/PuerkitoBio/goquery

goquery wraps Go's golang.org/x/net/html parser in a jQuery-like API, so doc.Find(".price_color") works the way you'd expect from the browser. That's the whole toolkit for static pages — no API key, no browser.

How do I fetch a page in Go?

Build a request so you can set headers, send it, and always check the status and close the body. Go is explicit about all three, which feels verbose but keeps you honest:

req, _ := http.NewRequest("GET", "https://books.toscrape.com/", nil)
req.Header.Set("User-Agent", "Mozilla/5.0 (compatible; MyScraper/1.0)")

resp, err := http.DefaultClient.Do(req)
if err != nil {
    log.Fatal(err)
}
defer resp.Body.Close()

if resp.StatusCode != 200 {
    log.Fatalf("HTTP %d", resp.StatusCode)
}

The User-Agent header heads off the easy blocks — Go's default agent is a common denylist entry. The defer resp.Body.Close() prevents a resource leak that will bite you the moment you fetch in a loop.

How do I parse HTML with goquery?

Hand the response body to goquery and select with CSS. Inspect the page first: each book on this site is an <article class="product_pod">, with its title in h3 a (in the title attribute), its price in .price_color, and stock in .instock.availability. A struct holds each record:

type Book struct {
    Title, Price, Availability string
}

doc, err := goquery.NewDocumentFromReader(resp.Body)
if err != nil {
    log.Fatal(err)
}

var books []Book
doc.Find("article.product_pod").Each(func(i int, s *goquery.Selection) {
    title, _ := s.Find("h3 a").Attr("title")
    books = append(books, Book{
        Title:        title,
        Price:        s.Find(".price_color").Text(),
        Availability: strings.TrimSpace(s.Find(".instock.availability").Text()),
    })
})

fmt.Printf("Found %d books\n", len(books))

Reading the title from the title attribute rather than the visible text avoids the display truncation, and Attr returns a second boolean for "did it exist" that you can check when a field is optional. That found all twenty books on the page.

How do I scrape multiple pages?

Move fetch-and-parse into a function returning ([]Book, error), then loop the numbered pages, stopping when one comes back empty:

func scrapePage(url string) ([]Book, error) {
    req, _ := http.NewRequest("GET", url, nil)
    req.Header.Set("User-Agent", "Mozilla/5.0 (compatible; MyScraper/1.0)")
    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()
    if resp.StatusCode != 200 {
        return nil, fmt.Errorf("HTTP %d", resp.StatusCode)
    }

    doc, err := goquery.NewDocumentFromReader(resp.Body)
    if err != nil {
        return nil, err
    }

    var books []Book
    doc.Find("article.product_pod").Each(func(i int, s *goquery.Selection) {
        title, _ := s.Find("h3 a").Attr("title")
        books = append(books, Book{
            Title:        title,
            Price:        s.Find(".price_color").Text(),
            Availability: strings.TrimSpace(s.Find(".instock.availability").Text()),
        })
    })
    return books, nil
}

func main() {
    var all []Book
    for page := 1; page <= 2; page++ {
        url := fmt.Sprintf("https://books.toscrape.com/catalogue/page-%d.html", page)
        books, err := scrapePage(url)
        if err != nil || len(books) == 0 {
            break
        }
        fmt.Printf("page %d: %d books\n", page, len(books))
        all = append(all, books...)
        time.Sleep(time.Second)
    }
    fmt.Printf("TOTAL: %d books\n", len(all))
}

Two pages, kept light:

page 1: 20 books
page 2: 20 books
TOTAL: 40 books

What about Go's concurrency — isn't that the point?

It is, and it's where Go pulls ahead, but it comes with a responsibility. You could launch a goroutine per page and fetch all fifty at once. On a real site that's indistinguishable from an attack. The disciplined pattern is a bounded worker pool — a buffered channel as a semaphore that caps how many requests run at a time:

sem := make(chan struct{}, 5)   // at most 5 concurrent requests
var wg sync.WaitGroup
var mu sync.Mutex
var all []Book

for page := 1; page <= 50; page++ {
    wg.Add(1)
    go func(p int) {
        defer wg.Done()
        sem <- struct{}{}          // acquire a slot
        defer func() { <-sem }()   // release it

        url := fmt.Sprintf("https://books.toscrape.com/catalogue/page-%d.html", p)
        books, err := scrapePage(url)
        if err == nil {
            mu.Lock()
            all = append(all, books...)
            mu.Unlock()
        }
    }(page)
}
wg.Wait()

Five at a time is fast without being abusive; the sync.Mutex guards the shared slice from concurrent writes. This is the concurrency that makes Go worth reaching for — controlled, not unleashed.

How do I export to CSV?

Go's standard library includes encoding/csv, which escapes fields correctly so you don't hand-roll the format:

f, _ := os.Create("books.csv")
defer f.Close()

w := csv.NewWriter(f)
w.Write([]string{"title", "price", "availability"})
for _, b := range all {
    w.Write([]string{b.Title, b.Price, b.Availability})
}
w.Flush()

The resulting file opens in Excel or any database import:

title,price,availability
A Light in the Attic,£51.77,In stock
Tipping the Velvet,£53.74,In stock
Soumission,£50.10,In stock

When does net/http and goquery stop working?

The familiar wall: JavaScript. net/http fetches the server's HTML and never runs a line of the page's own scripts, so a site that renders its content in the browser hands goquery an empty shell. Check by viewing source — if the data isn't there, no selector will find it. For those pages you need chromedp, which drives a headless Chrome from Go, at a real cost in speed and memory. The other wall is serious anti-bot protection, which a user agent won't get you past.

Do you actually need to write Go for this?

Go is a genuinely strong choice for large, concurrent, static-site crawls, and if you're already in the ecosystem this is the right approach. But weigh the full lifetime of a scraper: the selectors break when markup changes, JavaScript pages pull in chromedp and browser infrastructure, and turning a one-off crawl into "watch this and tell me what changed" means building scheduling, storage, and diffing — the part that's genuinely hard, no matter how fast your fetching is.

An AI scraper hands you that for free. Website Scraper renders the page (JavaScript included), extracts by reading the content rather than a product_pod selector you maintain, and turns any scrape into a monitored alert with zero infrastructure. If you're writing Go for the speed or because it's your stack, follow this tutorial. If you only want the data out, paste the URL into the no-code scraper. The how to scrape data from a website guide covers the workflow, and the Python, PHP, and JavaScript tutorials show the same job elsewhere.