cap redirect

[Mzack9999]

redirect callback now caps body reads at BodyReadSize instead of reading the whole response

Closes #1696

Summary by CodeRabbit

• Bug Fixes

  • Limited how much of an HTTP response body is read before parsing, preventing hangs on very large or streaming redirect responses.
  • Improved redirect handling so requests can complete reliably even when the intermediate response never finishes normally.

• Tests

  • Added coverage for redirects with unbounded response bodies to confirm the response size limit is respected and the final request completes successfully.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: e8e1dded-a306-4261-8a17-05c96d63922c

📥 Commits

Reviewing files that changed from the base of the PR and between 3ae3c6d and 765a1a3.

📒 Files selected for processing (2)

• pkg/engine/common/base.go
• pkg/engine/common/redirect_bodycap_test.go

---

Walkthrough

The redirect callback in NewCrawlSessionWithURL now limits response-body reads with BodyReadSize before parsing. A new test streams an unbounded redirect response and verifies the crawl session still completes with the capped read.

Changes

Redirect response body cap

Layer / File(s)	Summary
Cap redirect body reads pkg/engine/common/base.go, pkg/engine/common/redirect_bodycap_test.go	The redirect response callback reads only BodyReadSize bytes before goquery parsing, and the new test drives a streaming redirect server to confirm the crawl session completes with the capped read.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

A bunny hopped and gave a grin,
“Less body bytes now flow within!”
🐇✨ Redirects no longer swell,
The capped reads work and pass quite well,
And final-ok hops safely in.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is concise and points to the redirect-body capping change.
Linked Issues check	✅ Passed	The redirect callback now honors BodyReadSize, matching the issue requirement, and the test covers capped redirect bodies.
Out of Scope Changes check	✅ Passed	The PR stays focused on redirect body capping and the related regression test.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch redirect-bodycap

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}