A simple, from-scratch web crawler built in Go as a learning project. It starts from a seed URL, fetches HTML pages, extracts links, and recursively visits new links up to a maximum depth while staying on the same host.
This project is intentionally small and simple as this is mainly a learning project.
The project focuses on a few core ideas:
- Parsing CLI flags for configuration.
- Fetching pages with
net/http. - Parsing HTML into a DOM-like tree.
- Extracting and resolving links.
- Launching concurrent crawl tasks with goroutines.
- Tracking completion with
sync.WaitGroup. - Limiting active concurrent work with a semaphore pattern built from a buffered channel.
- Preventing duplicate visits with a thread-safe store.
The crawler keeps the behavior deliberately narrow:
- Accepts a seed URL, max depth, and max workers from CLI flags.
- Starts crawling from the seed URL at depth 0.
- Fetches each page and parses its HTML.
- Extracts links from
<a href="...">elements. - Resolves relative links into absolute URLs with Go's
net/urlpackage.[cite:86] - Skips URLs that exceed the allowed depth.
- Skips URLs that were already visited.
- Skips URLs that leave the original host.
- Prints visited URLs and their depth.
go-crawler/
├── main.go # entry point and CLI flags
├── crawler/
│ └── crawler.go # orchestration, recursion, goroutines, waitgroup, semaphore
├── fetcher/
│ └── fetcher.go # HTTP GET logic
├── parser/
│ └── parser.go # HTML parsing and link extraction
├── store/
│ └── store.go # thread-safe visited URL tracking
├── go.mod
└── go.sum
The program reads three CLI flags:
-urlfor the seed URL.-maxDepthfor the maximum crawl depth.-workersfor the maximum number of active concurrent fetches.
These values are used to initialize the crawler and its shared dependencies.
The crawler is created with:
- The seed host, used to keep crawling on the same site.
- The maximum depth.
- A fetcher for HTTP requests.
- A visited URL store.
- A semaphore channel to limit active work.
- A
sync.WaitGroupto track running goroutines.
The Crawl(seedURL) method adds the first task to the waitgroup and launches the first goroutine for the seed page.
That goroutine begins with crawl(seedURL, 0).
Each crawl(url, depth) call begins with a few safety checks:
- Stop if the current depth is greater than the configured max depth.
- Stop if the URL was already visited.
- Stop if the URL is outside the original host.
- Stop if the URL is malformed or cannot be parsed.
These checks keep the crawler small, correct, and safe from obvious infinite loops.
Before fetching a page, the goroutine acquires a slot from the semaphore channel. If the semaphore is full, that goroutine blocks until another goroutine releases a slot.
This does not prevent new goroutines from being created. It limits how many goroutines can be actively doing the protected work at the same time, which in this project is the fetch stage.
The fetcher performs an HTTP GET request using net/http and returns the raw response body. A timeout should always be configured so a slow server does not block a goroutine forever.
The parser converts raw HTML into a tree structure using golang.org/x/net/html, then walks that tree looking for anchor tags and their href attributes.
Each discovered link is resolved relative to the current page URL so that paths like /about become full absolute URLs.
For every extracted link, the crawler increments the waitgroup and launches another goroutine with depth + 1.
This is the recursive part of the crawler. Each child goroutine repeats the same process for its own URL.
Every goroutine calls Done() when it finishes. The top-level crawl waits until the waitgroup counter returns to zero, which means all spawned crawl tasks have completed.
This project uses a simple and useful Go concurrency pattern:
- Goroutines run crawl tasks concurrently.
- A waitgroup tracks when all crawl tasks are finished.
- A buffered channel used as a semaphore limits active concurrent work.
- A thread-safe store prevents duplicate visits.
A useful mental model is:
main -> start first goroutine -> fetch page -> parse links -> spawn child goroutines
| |
v v
semaphore limit waitgroup tracking
This crawler is intentionally not too featureful. It avoids framework abstractions so the main building blocks remain visible:
- Networking with
net/http. - URL handling with
net/url. - Recursive HTML traversal.
- Basic bounded concurrency with goroutines, channels, and waitgroups.
- Shared-state safety when multiple goroutines work at once.
That makes it a strong first crawler project before moving on to more advanced tools like Colly.
go run cmd/main.go -url=https://golang.org -maxDepth=2 -workers=5Example output:
Starting crawl: https://golang.org (depth=2, workers=5)
[depth 0] https://golang.org
[depth 1] https://golang.org/doc/
[depth 1] https://golang.org/pkg/
[depth 2] https://golang.org/pkg/fmt/
Done.
This is a learning crawler, so it intentionally leaves out many production features:
- No
robots.txthandling. - No retry logic.
- No rate limiting per host beyond the semaphore.
- No persistence layer.
- No structured result export.
- No JavaScript rendering.
- No tests yet.
Those omissions are intentional. The value of this project is that the crawler flow remains easy to trace and reason about.