Skip to content

docs: security features sweep — IP Restrictions page, threat protection billing fix, OpenAPI override schema - #1114

Merged
mogery merged 4 commits into
mainfrom
mogery/security-docs-sweep
Jul 7, 2026
Merged

docs: security features sweep — IP Restrictions page, threat protection billing fix, OpenAPI override schema#1114
mogery merged 4 commits into
mainfrom
mogery/security-docs-sweep

Conversation

@mogery

@mogery mogery commented Jul 7, 2026

Copy link
Copy Markdown
Member

Sweep of the docs against the recently shipped security/enterprise API features. Three gaps found and fixed:

1. IP Restrictions had no documentation

The per-team API key IP allowlist shipped with only a one-line bullet on the enterprise page. Adds features/ip-restrictions.mdx covering:

  • IPv4 / IPv6 / CIDR entries (and IPv4-mapped-IPv6 normalization)
  • enforcement at authentication across every API surface, with the exact 403 body
  • fail-closed 500 behavior, empty-list-means-unrestricted rule, ~1 min propagation
  • inbound vs. outbound (static egress IPs) disambiguation

Linked from enterprise.mdx and the Key Restrictions page; registered in nav (English blocks only — locadex handles locales).

2. Threat protection billing docs were wrong for crawls

The feature page claimed crawl/search/map scan fees scale with unique domains, not pages. Per the billing implementation and its snips (scrape-billing.ts, threat-protection-billing.test.ts: "crawl bills base + 2 per scraped document", "batch scrape bills base + 2 per document (each job scans afresh)"), crawls and batch scrapes scan every page independently — verdicts are never reused across page jobs (ZDR, no verdict cache). Only search and map dedupe by domain within a request. Rewrote the billing bullets accordingly, and documented the redirect dedup rule.

Also documents the forced mode rule (overrides may not set mode: "off") with its error-reference row.

3. Per-request threatProtection override missing from OpenAPI

The feature page describes the per-request override object, but no request schema defined it. Adds a ThreatProtectionOverride component to both specs and references it exactly where the API accepts it:

  • v2: shared ScrapeOptions (covers scrape, batch scrape, crawl/search scrapeOptions) + top-level on /map, /extract, /agent, /search
  • v1: BaseScrapeOptions + top-level on /map, /extract, /search

v1 spec edited via Prettier-clean surgery (60 pure insertions, no reflow).

Explicitly not covered

  • Search highlights: already in flight in docs(search): document the highlights parameter (on by default) #1112.
  • Admin cache-clear endpoints (ip-restriction-cache-clear, key-restriction-cache-clear): internal, BULL_AUTH_KEY-gated — intentionally undocumented.
  • Key Restrictions page: verified against the merged implementation, already accurate.
  • Per-key concurrency limits (api_keys.concurrency): just merged, no public config surface yet — skipped for now.

🤖 Generated with Claude Code

mogery and others added 4 commits July 7, 2026 15:01
Documents the per-team API key IP allowlist: IPv4/IPv6/CIDR entries,
enforcement at authentication across all API surfaces, the 403/500 error
behavior, and the empty-list = unrestricted rule. Linked from the
enterprise page and the Key Restrictions page, registered in nav.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…de rule

Crawls and batch scrapes bill +2 per scraped page (each page job scans
afresh; verdicts are never reused across jobs), not per unique domain —
only search and map dedupe by domain within a request. Also documents
that enforced (forced) threat protection rejects overrides that set
mode to off, and adds the matching error-reference row.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…OpenAPI

The override object was described in prose on the feature page but
missing from the request schemas. Adds a ThreatProtectionOverride
component to both specs and references it where the API accepts it:
shared scrape options (scrape, batch scrape, crawl/search scrapeOptions)
plus top-level on map, extract, search, and agent (v2).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The allowlist itself is team-scoped; the enterprise feature flag lives
on the organization, matching the Key Restrictions page wording.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mintlify

mintlify Bot commented Jul 7, 2026

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
firecrawl 🟢 Ready View Preview Jul 7, 2026, 1:09 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@mogery
mogery merged commit 6299902 into main Jul 7, 2026
2 checks passed
@mogery
mogery deleted the mogery/security-docs-sweep branch July 7, 2026 16:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant