Skip to content

About

A simple web crawler built using go

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

2 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Web Crawler in Go

A simple, from-scratch web crawler built in Go as a learning project. It starts from a seed URL, fetches HTML pages, extracts links, and recursively visits new links up to a maximum depth while staying on the same host.

Purpose

This project is intentionally small and simple as this is mainly a learning project.

The project focuses on a few core ideas:

  • Parsing CLI flags for configuration.
  • Fetching pages with net/http.
  • Parsing HTML into a DOM-like tree.
  • Extracting and resolving links.
  • Launching concurrent crawl tasks with goroutines.
  • Tracking completion with sync.WaitGroup.
  • Limiting active concurrent work with a semaphore pattern built from a buffered channel.
  • Preventing duplicate visits with a thread-safe store.

What the crawler does

The crawler keeps the behavior deliberately narrow:

  • Accepts a seed URL, max depth, and max workers from CLI flags.
  • Starts crawling from the seed URL at depth 0.
  • Fetches each page and parses its HTML.
  • Extracts links from <a href="..."> elements.
  • Resolves relative links into absolute URLs with Go's net/url package.[cite:86]
  • Skips URLs that exceed the allowed depth.
  • Skips URLs that were already visited.
  • Skips URLs that leave the original host.
  • Prints visited URLs and their depth.

Project structure

go-crawler/
├── main.go        # entry point and CLI flags
├── crawler/
│   └── crawler.go     # orchestration, recursion, goroutines, waitgroup, semaphore
├── fetcher/
│   └── fetcher.go     # HTTP GET logic
├── parser/
│   └── parser.go      # HTML parsing and link extraction
├── store/
│   └── store.go       # thread-safe visited URL tracking
├── go.mod
└── go.sum

How it works

1. Read configuration

The program reads three CLI flags:

  • -url for the seed URL.
  • -maxDepth for the maximum crawl depth.
  • -workers for the maximum number of active concurrent fetches.

These values are used to initialize the crawler and its shared dependencies.

2. Initialize the crawler

The crawler is created with:

  • The seed host, used to keep crawling on the same site.
  • The maximum depth.
  • A fetcher for HTTP requests.
  • A visited URL store.
  • A semaphore channel to limit active work.
  • A sync.WaitGroup to track running goroutines.

3. Start the first crawl task

The Crawl(seedURL) method adds the first task to the waitgroup and launches the first goroutine for the seed page.

That goroutine begins with crawl(seedURL, 0).

4. Run safety checks

Each crawl(url, depth) call begins with a few safety checks:

  1. Stop if the current depth is greater than the configured max depth.
  2. Stop if the URL was already visited.
  3. Stop if the URL is outside the original host.
  4. Stop if the URL is malformed or cannot be parsed.

These checks keep the crawler small, correct, and safe from obvious infinite loops.

5. Limit concurrent work

Before fetching a page, the goroutine acquires a slot from the semaphore channel. If the semaphore is full, that goroutine blocks until another goroutine releases a slot.

This does not prevent new goroutines from being created. It limits how many goroutines can be actively doing the protected work at the same time, which in this project is the fetch stage.

6. Fetch the page

The fetcher performs an HTTP GET request using net/http and returns the raw response body. A timeout should always be configured so a slow server does not block a goroutine forever.

7. Parse HTML and extract links

The parser converts raw HTML into a tree structure using golang.org/x/net/html, then walks that tree looking for anchor tags and their href attributes.

Each discovered link is resolved relative to the current page URL so that paths like /about become full absolute URLs.

8. Spawn child crawl tasks

For every extracted link, the crawler increments the waitgroup and launches another goroutine with depth + 1.

This is the recursive part of the crawler. Each child goroutine repeats the same process for its own URL.

9. Wait for all goroutines to finish

Every goroutine calls Done() when it finishes. The top-level crawl waits until the waitgroup counter returns to zero, which means all spawned crawl tasks have completed.

Concurrency model

This project uses a simple and useful Go concurrency pattern:

  • Goroutines run crawl tasks concurrently.
  • A waitgroup tracks when all crawl tasks are finished.
  • A buffered channel used as a semaphore limits active concurrent work.
  • A thread-safe store prevents duplicate visits.

A useful mental model is:

main -> start first goroutine -> fetch page -> parse links -> spawn child goroutines
                                   |                              |
                                   v                              v
                             semaphore limit                 waitgroup tracking

Why this project is good for learning

This crawler is intentionally not too featureful. It avoids framework abstractions so the main building blocks remain visible:

  • Networking with net/http.
  • URL handling with net/url.
  • Recursive HTML traversal.
  • Basic bounded concurrency with goroutines, channels, and waitgroups.
  • Shared-state safety when multiple goroutines work at once.

That makes it a strong first crawler project before moving on to more advanced tools like Colly.

Example run

go run cmd/main.go -url=https://golang.org -maxDepth=2 -workers=5

Example output:

Starting crawl: https://golang.org (depth=2, workers=5)

[depth 0] https://golang.org
[depth 1] https://golang.org/doc/
[depth 1] https://golang.org/pkg/
[depth 2] https://golang.org/pkg/fmt/

Done.

Notes and limitations

This is a learning crawler, so it intentionally leaves out many production features:

  • No robots.txt handling.
  • No retry logic.
  • No rate limiting per host beyond the semaphore.
  • No persistence layer.
  • No structured result export.
  • No JavaScript rendering.
  • No tests yet.

Those omissions are intentional. The value of this project is that the crawler flow remains easy to trace and reason about.

About

A simple web crawler built using go

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages