diff --git a/agent-quickstart/elixir.mdx b/agent-quickstart/elixir.mdx new file mode 100644 index 000000000..19789a43a --- /dev/null +++ b/agent-quickstart/elixir.mdx @@ -0,0 +1,191 @@ +--- +title: "Elixir Agent Quickstart" +description: "Canonical Firecrawl Elixir quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Elixir Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`:firecrawl` **v1.9.1**) and the v2 OpenAPI spec. The Elixir client is auto-generated from the OpenAPI spec; function names and parameter keys match the spec. + +## Install + +Add to `mix.exs`: + +```elixir +{:firecrawl, "~> 1.9"} +``` + +## Authenticate + +```elixir +# config/runtime.exs or config.exs +config :firecrawl, api_key: System.get_env("FIRECRAWL_API_KEY") + +# Or pass per call: +{:ok, res} = Firecrawl.search_and_scrape([query: "example"], api_key: "fc-your-api-key") +``` + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`Firecrawl.search_and_scrape(params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, res} = Firecrawl.search_and_scrape( + query: "site:docs.firecrawl.dev webhook retries", + limit: 5, + scrape_options: [ + formats: ["markdown"], + only_main_content: true + ] +) +``` + +### Parameters + +All parameters are keyword list entries. Only `query` is required. + +- `query` — string (required). The search query. Use `site:example.com` to limit results. +- `sources` — list of atoms, strings, or maps. Controls which sources are searched. Values: `:web`, `:news`, `:images` (or `"web"`, etc.), or `%{type: "web"}`. +- `categories` — list of atoms, strings, or maps. Category filters. Values: `:github`, `:research`, `:pdf` (or strings, or `%{type: "github"}`). +- `include_domains` — list of strings. Restrict results to these domains. +- `exclude_domains` — list of strings. Exclude results from these domains. +- `limit` — integer. Maximum number of results. +- `tbs` — string. Time-based filter (e.g. `"qdr:d"`, `"qdr:w"`). +- `location` — string. Localized results (e.g. `"San Francisco,California,United States"`). +- `country` — string. ISO 3166-1 alpha-2 country code (e.g. `"US"`). +- `highlights` — boolean. Generate query-relevant highlights. Default: `true`. +- `ignore_invalid_urls` — boolean. Drop URLs that cannot be scraped. +- `timeout` — integer. Request timeout in milliseconds. +- `enterprise` — list of strings. `["zdr"]` for zero data retention, `["anon"]` for anonymized. +- `scrape_options` — keyword list. Scrape each search result (see Scrape parameters). + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +`Firecrawl.scrape_and_extract_from_url(params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, res} = Firecrawl.scrape_and_extract_from_url( + url: "https://docs.firecrawl.dev", + formats: ["markdown"], + only_main_content: true +) +``` + +### Parameters + +All parameters are keyword list entries. Only `url` is required. + +- `url` — string (required). The URL to scrape. +- `formats` — list of format strings or format maps. Output formats. + - Plain strings: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"branding"`, `"audio"`, `"video"`. + - Format maps: + - `%{type: "json", prompt: "...", schema: %{...}}` — JSON extraction. + - `%{type: "question", question: "..."}` — question-answer extraction. + - `%{type: "highlights", query: "..."}` — relevant source-text extraction. + - `%{type: "screenshot", fullPage: true, quality: 80, viewport: %{width: 1280, height: 720}}`. + - `%{type: "changeTracking", modes: ["git-diff"], tag: "..."}`. + - `%{type: "attributes", selectors: [%{selector: "a", attribute: "href"}]}`. +- `headers` — map. Custom request headers. +- `include_tags` — list of strings. Include only specific HTML tags. +- `exclude_tags` — list of strings. Exclude specific HTML tags. +- `only_main_content` — boolean. Strip nav, footer, and boilerplate. +- `timeout` — integer. Timeout in milliseconds. +- `wait_for` — integer. Wait for the page to render (milliseconds). +- `mobile` — boolean. Use a mobile viewport. +- `parsers` — list of `"pdf"` or `%{type: "pdf", mode: "fast" | "auto" | "ocr", maxPages: integer}`. +- `actions` — list of action maps. Pre-scrape browser actions. + - `%{type: "wait", milliseconds: 750}` or `%{type: "wait", selector: "..."}` + - `%{type: "click", selector: "...", all: false}` + - `%{type: "write", text: "..."}` — click to focus the input first. + - `%{type: "press", key: "..."}` + - `%{type: "scroll", direction: "down", selector: "..."}` + - `%{type: "scrape"}` + - `%{type: "executeJavascript", script: "..."}` + - `%{type: "pdf", format: "Letter", landscape: false, scale: 1}` +- `location` — keyword list `[country: "US", languages: ["en-US"]]`. +- `skip_tls_verification` — boolean. Skip TLS verification. +- `remove_base64_images` — boolean. Drop base64 images from markdown. +- `block_ads` — boolean. Ad and cookie popup blocking. +- `proxy` — atom or string. `:basic`, `:enhanced`, or `:auto`. +- `max_age` — integer. Cached data up to a maximum age (milliseconds). +- `min_age` — integer. Cached data only if at least this old (milliseconds). +- `store_in_cache` — boolean. Cache the result. +- `lockdown` — boolean. Only serve cached results. +- `redact_pii` — boolean. Redact personally identifiable information. +- `audit_metadata` — keyword list `[username: "..."]`. User attribution for SIEM logging. +- `profile` — keyword list `[name: "...", save_changes: true]`. Persistent browser profile. +- `zero_data_retention` — boolean. Enable zero data retention. + +## Interact + +### Why use it + +Use `interact` to run code in the browser session tied to a scrape job. The Elixir SDK supports code-based interactions only (no `prompt` parameter). + +### Preferred SDK method + +`Firecrawl.interact_with_scrape_browser_session(job_id, params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, scrape_res} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com", + formats: ["markdown"] +) + +job_id = get_in(scrape_res.body, ["data", "metadata", "scrapeId"]) + +{:ok, res} = Firecrawl.interact_with_scrape_browser_session( + job_id, + code: "console.log(await page.title());", + language: :node, + timeout: 60 +) +``` + +To end the session: `Firecrawl.stop_interactive_scrape_browser_session(job_id)`. + +### Parameters + +- `job_id` — string (required). Scrape job ID. +- `code` — string (required). Code to execute in the browser session. +- `language` — atom or string. `:python`, `:node`, or `:bash`. +- `timeout` — integer. Execution timeout in seconds. + +## Notes + +- Each public function has a bang (`!`) variant that raises on error instead of returning `{:error, _}`. +- The Elixir SDK is auto-generated from the OpenAPI spec; function names are spec-derived (e.g. `scrape_and_extract_from_url`, `search_and_scrape`, `interact_with_scrape_browser_session`). +- This SDK exposes code-based interactions only — there is no `prompt` parameter on `interact_with_scrape_browser_session`. +- Keyword list params are auto-converted to camelCase JSON for the API. +- Deprecated format type: `%{type: "query"}` — use `%{type: "question"}` or `%{type: "highlights"}` instead. +- No deprecated function aliases exist in this SDK (it is regenerated from the spec). + +## Source Of Truth + +- `firecrawl/apps/elixir-sdk/mix.exs` +- `firecrawl/apps/elixir-sdk/lib/firecrawl.ex` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/java.mdx b/agent-quickstart/java.mdx new file mode 100644 index 000000000..1e04e6502 --- /dev/null +++ b/agent-quickstart/java.mdx @@ -0,0 +1,222 @@ +--- +title: "Java Agent Quickstart" +description: "Canonical Firecrawl Java quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Java Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`firecrawl-java` **v1.12.1**) and the v2 OpenAPI spec. Requires Java 11+. + +## Install + +Maven: + +```xml + + com.firecrawl + firecrawl-java + 1.12.1 + +``` + +Gradle: + +```gradle +implementation("com.firecrawl:firecrawl-java:1.12.1") +``` + +## Authenticate + +```java +import com.firecrawl.client.FirecrawlClient; + +FirecrawlClient client = FirecrawlClient.builder() + .apiKey(System.getenv("FIRECRAWL_API_KEY")) + .build(); + +// Or from env/system property: +// FirecrawlClient client = FirecrawlClient.fromEnv(); +``` + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query)` or `client.search(query, options)` → `SearchData` + +### Example + +```java +import com.firecrawl.models.SearchData; +import com.firecrawl.models.SearchOptions; +import com.firecrawl.models.ScrapeOptions; + +SearchData results = client.search( + "site:docs.firecrawl.dev webhook retries", + SearchOptions.builder() + .limit(5) + .scrapeOptions( + ScrapeOptions.builder() + .formats(List.of("markdown")) + .onlyMainContent(true) + .build() + ) + .build() +); + +List> web = results.getWeb(); +``` + +Results are grouped by source: `getWeb()`, `getNews()`, `getImages()`. Each returns `List>` and may be null. + +### Parameters + +All `SearchOptions.builder()` fields are optional. + +- `query` — String (required). The search query. Use `site:example.com` to limit results. +- `options.sources` — List of `"web" | "news" | "images"` or `{type: "web"}` maps. +- `options.categories` — List of `"github" | "research" | "pdf"` or `{type: ...}` maps. +- `options.includeDomains` — List of Strings. Restrict results to these domains. +- `options.excludeDomains` — List of Strings. Exclude results from these domains. +- `options.limit` — Integer. Maximum number of results. +- `options.tbs` — String. Time-based filter (e.g. `"qdr:d"`, `"qdr:w"`). +- `options.location` — String. Localized results. +- `options.highlights` — Boolean. Generate query-relevant highlights. Default: `true`. +- `options.ignoreInvalidURLs` — Boolean. Drop URLs that cannot be scraped. +- `options.timeout` — Integer. Request timeout in milliseconds. +- `options.scrapeOptions` — `ScrapeOptions`. Scrape each search result (see Scrape parameters). +- `options.integration` — String. Integration identifier. + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +`client.scrape(url)` or `client.scrape(url, options)` → `Document` + +### Example + +```java +import com.firecrawl.models.Document; +import com.firecrawl.models.ScrapeOptions; + +Document doc = client.scrape( + "https://docs.firecrawl.dev", + ScrapeOptions.builder() + .formats(List.of("markdown")) + .onlyMainContent(true) + .build() +); + +System.out.println(doc.getMarkdown()); +``` + +### Parameters + +All `ScrapeOptions.builder()` fields are optional. + +- `url` — String (required). The URL to scrape. +- `options.formats` — List of format strings or format objects. Output formats. + - Plain strings: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"attributes"`, `"branding"`, `"audio"`, `"video"`. + - Typed format objects: + - `JsonFormat.builder().prompt("...").schema(Map.of(...)).build()` — JSON extraction. + - `QuestionFormat.builder().question("...").build()` — question-answer extraction. + - `HighlightsFormat.builder().query("...").build()` — relevant source-text extraction. + - `Map.of("type", "screenshot", "fullPage", true, "quality", 80)` — screenshot options. + - `Map.of("type", "changeTracking", "modes", List.of("git-diff"), "tag", "...")` — change tracking. + - `Map.of("type", "attributes", "selectors", List.of(Map.of("selector", "a", "attribute", "href")))` — attribute extraction. +- `options.headers` — `Map`. Custom request headers. +- `options.includeTags` — `List`. Include only specific HTML tags. +- `options.excludeTags` — `List`. Exclude specific HTML tags. +- `options.onlyMainContent` — Boolean. Strip nav, footer, and boilerplate. +- `options.timeout` — Integer. Timeout in milliseconds. +- `options.waitFor` — Integer. Wait for the page to render (milliseconds). +- `options.mobile` — Boolean. Use a mobile viewport. +- `options.parsers` — `List`. `"pdf"` or `Map.of("type", "pdf", "maxPages", 5)`. +- `options.actions` — `List>`. Pre-scrape browser actions. + - `Map.of("type", "wait", "milliseconds", 750)` or `Map.of("type", "wait", "selector", "...")` + - `Map.of("type", "click", "selector", "...")` + - `Map.of("type", "write", "text", "...")` — click to focus the input first. + - `Map.of("type", "press", "key", "...")` + - `Map.of("type", "scroll", "direction", "down", "selector", "...")` + - `Map.of("type", "scrape")` + - `Map.of("type", "executeJavascript", "script", "...")` + - `Map.of("type", "pdf", "format", "Letter", "landscape", false)` +- `options.location` — `LocationConfig.builder().country("US").languages(List.of("en-US")).build()`. +- `options.skipTlsVerification` — Boolean. Skip TLS verification. +- `options.removeBase64Images` — Boolean. Drop base64 images from markdown. +- `options.blockAds` — Boolean. Ad and cookie popup blocking. +- `options.proxy` — String. `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`, or a custom proxy URL. +- `options.maxAge` — Long. Cached data up to a maximum age (milliseconds). +- `options.storeInCache` — Boolean. Cache the result. +- `options.lockdown` — Boolean. Only serve cached results. +- `options.redactPII` — Boolean. Redact personally identifiable information. +- `options.auditMetadata` — `AuditMetadata` with `username`. User attribution for SIEM logging. +- `options.integration` — String. Integration identifier. + +## Interact + +### Why use it + +Use `interact` to run code in the browser session tied to a scrape job. The Java SDK supports code-based interactions only (no `prompt` parameter). + +### Preferred SDK method + +- `client.interact(jobId, code)` — default language `"node"` +- `client.interact(jobId, code, language, timeout)` — `timeout` is seconds (1–300), null for API default + +### Example + +```java +import com.firecrawl.models.BrowserExecuteResponse; + +Document doc = client.scrape("https://example.com", + ScrapeOptions.builder().formats(List.of("markdown")).build()); +String jobId = (String) doc.getMetadata().get("scrapeId"); + +BrowserExecuteResponse result = client.interact( + jobId, + "console.log(await page.title());", + "node", + 60 +); + +System.out.println(result.getStdout()); +``` + +To end the session: `client.stopInteractiveBrowser(jobId)`. + +### Parameters + +- `jobId` — String (required). Scrape job ID. +- `code` — String (required). Code to execute in the browser session. +- `language` — String. `"python"`, `"node"`, or `"bash"`. Default: `"node"`. +- `timeout` — Integer. Execution timeout in seconds (1–300). Null uses API default (30s). + +## Notes + +- Deprecated aliases: `scrapeExecute` → `interact`; `deleteScrapeBrowser` → `stopInteractiveBrowser`. +- Every sync method has an `*Async` twin returning `CompletableFuture`. +- The Java SDK exposes code-based interactions only — there is no `prompt` parameter on `interact`. +- Deprecated format type: `QueryFormat` — use `QuestionFormat` or `HighlightsFormat` instead. +- Built option objects are immutable (unmodifiable collections). + +## Source Of Truth + +- `firecrawl/apps/java-sdk/build.gradle.kts` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/client/FirecrawlClient.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/node.mdx b/agent-quickstart/node.mdx new file mode 100644 index 000000000..c283535ab --- /dev/null +++ b/agent-quickstart/node.mdx @@ -0,0 +1,194 @@ +--- +title: "Node.js Agent Quickstart" +description: "Canonical Firecrawl Node.js quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Node.js Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`firecrawl` **v4.31.1**) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. + +## Install + +```bash +npm install firecrawl +``` + +Requires Node.js >= 22. + +## Authenticate + +```ts +import { Firecrawl } from "firecrawl"; + +const client = new Firecrawl({ + apiKey: process.env.FIRECRAWL_API_KEY, + // apiUrl: "https://api.firecrawl.dev" // optional; falls back to FIRECRAWL_API_URL or cloud default +}); +``` + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. For multi-step interactive flows, prefer `interact` over scrape-time `actions`. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, options?)` → `Promise` + +### Example + +```ts +const results = await client.search("site:docs.firecrawl.dev webhook retries", { + limit: 5, + scrapeOptions: { + formats: ["markdown"], + onlyMainContent: true, + }, +}); + +for (const item of results.web ?? []) { + console.log(item.url, item.title); +} +``` + +Results are grouped by source: `results.web`, `results.news`, `results.images`. Do not access `results.data`. + +### Parameters + +- `query` — string (required). The search query. Use `site:example.com` to limit results to a domain. +- `options.sources` — array of `"web" | "news" | "images"` or `{ type: "web" | "news" | "images" }`. Controls which sources are searched. +- `options.categories` — array of `"github" | "research" | "pdf"` or `{ type: ... }`. Filters results by category. +- `options.includeDomains` — string array. Restrict results to these domains. Mutually exclusive with `excludeDomains`. +- `options.excludeDomains` — string array. Exclude results from these domains. Mutually exclusive with `includeDomains`. +- `options.limit` — number. Maximum number of results. +- `options.tbs` — string. Time-based filter (e.g. `"qdr:d"` for past day, `"qdr:w"` for past week). +- `options.location` — string. Localized results (e.g. `"San Francisco,California,United States"`). +- `options.highlights` — boolean. Generate query-relevant highlights. Default: `true`. +- `options.ignoreInvalidURLs` — boolean. Drop URLs that cannot be scraped by other endpoints. +- `options.timeout` — number. Request timeout in milliseconds. +- `options.scrapeOptions` — `ScrapeOptions`. Scrape each search result (see Scrape parameters for fields). +- `options.enterprise` — array of `"default" | "anon" | "zdr"`. Enterprise options for zero data retention. +- `options.threatProtection` — `ThreatProtectionOptions`. Enterprise threat protection override with `mode`, `riskScoreThreshold`, `blacklist`, `whitelist`, `blockedTlds`, `failurePolicy`. +- `options.integration` — string. Integration identifier for tracking. + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +`client.scrape(url, options?)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://docs.firecrawl.dev", { + formats: ["markdown"], + onlyMainContent: true, +}); + +console.log(doc.markdown); +``` + +### Parameters + +- `url` — string (required). The URL to scrape. +- `options.formats` — array of format strings or format objects. Output formats to request. + - Plain strings: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"attributes"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. + - Object-only formats: + - `{ type: "json", prompt?: string, schema?: object | ZodSchema }` — at least one of `prompt` or `schema` is required. + - `{ type: "question", question: string }` — question-answer extraction. + - `{ type: "highlights", query: string }` — relevant source-text extraction. + - `{ type: "screenshot", fullPage?: boolean, quality?: number, viewport?: { width, height } }`. + - `{ type: "changeTracking", modes: ("git-diff" | "json")[], schema?: object, prompt?: string, tag?: string }` — `modes` is required. + - `{ type: "attributes", selectors: Array<{ selector: string, attribute: string }> }`. +- `options.headers` — `Record`. Custom request headers. +- `options.includeTags` — string array. Include only specific HTML tags. +- `options.excludeTags` — string array. Exclude specific HTML tags. +- `options.onlyMainContent` — boolean. Strip nav, footer, and other boilerplate. +- `options.timeout` — number. Timeout in milliseconds. +- `options.waitFor` — number. Wait for the page to render (milliseconds). +- `options.mobile` — boolean. Use a mobile viewport. +- `options.parsers` — array of `"pdf"` or `{ type: "pdf", mode?: "fast" | "auto" | "ocr", maxPages?: number }`. File parsing controls. +- `options.actions` — array of action objects. Pre-scrape browser actions. + - `{ type: "wait", milliseconds?: number, selector?: string }` + - `{ type: "click", selector: string }` + - `{ type: "write", text: string }` — click to focus the input first. + - `{ type: "press", key: string }` + - `{ type: "scroll", direction: "up" | "down", selector?: string }` + - `{ type: "screenshot", fullPage?: boolean, quality?: number, viewport?: object }` + - `{ type: "scrape" }` + - `{ type: "executeJavascript", script: string }` + - `{ type: "pdf", format?: string, landscape?: boolean, scale?: number }` +- `options.location` — `{ country?: string, languages?: string[] }`. Geo or language-aware scraping. +- `options.skipTlsVerification` — boolean. Skip TLS verification. +- `options.removeBase64Images` — boolean. Drop base64 images from markdown output. +- `options.fastMode` — boolean. Faster scrapes with reduced fidelity. +- `options.blockAds` — boolean. Ad and cookie popup blocking. +- `options.proxy` — `"basic" | "stealth" | "enhanced" | "auto"` or a custom proxy URL string. +- `options.maxAge` — number. Cached data up to a maximum age (milliseconds). +- `options.minAge` — number. Cached data only if it is at least this old (milliseconds). +- `options.storeInCache` — boolean. Cache the result. +- `options.lockdown` — boolean. Only serve cached results; never make outbound requests. +- `options.redactPII` — boolean or `{ mode?: "accurate" | "aggressive" | "fast", entities?: array, replaceStyle?: "tag" | "mask" | "remove" }`. Redact personally identifiable information. +- `options.threatProtection` — `ThreatProtectionOptions`. Enterprise threat protection override. +- `options.auditMetadata` — `{ username: string }`. User attribution for SIEM logging. +- `options.profile` — `{ name: string, saveChanges?: boolean }`. Persistent browser profile shared across scrapes and interactions. +- `options.integration` — string. Integration identifier. + +## Interact + +### Why use it + +Use `interact` for code or natural-language control of the browser session tied to a scrape job (via `metadata.scrapeId`). At least one of `code` or `prompt` is required. For flows that go beyond quick pre-scrape tweaks, prefer `interact` over scrape-time `actions`. + +### Preferred SDK method + +`client.interact(jobId, args)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com", { formats: ["markdown"] }); +const jobId = doc.metadata?.scrapeId; +if (!jobId) throw new Error("Missing scrapeId from scrape response"); + +const result = await client.interact(jobId, { + prompt: "Click the pricing tab and summarize the plans.", +}); +``` + +To end the session: `client.stopInteraction(jobId)`. + +### Parameters + +- `jobId` — string (required). Scrape job ID from `document.metadata.scrapeId`. +- `args.code` — string. Code to run in the browser session (e.g. Playwright `page` usage in the `node` runtime). +- `args.prompt` — string. Natural-language instruction for the browser agent. +- At least one of `code` or `prompt` must be non-empty. +- `args.language` — `"python" | "node" | "bash"`. Runtime language. Default: `"node"`. +- `args.timeout` — number. Execution timeout in seconds. + +## Notes + +- Deprecated aliases: `scrapeExecute` → `interact`; `stopInteractiveBrowser` and `deleteScrapeBrowser` → `stopInteraction`; `scrapeUrl` → `scrape`. +- The default `Firecrawl` export is the v2 client; v1 remains under `client.v1`. +- Zod schemas passed to `formats` (for `json` or `changeTracking`) are converted to JSON Schema by the SDK. +- Deprecated format type: `{ type: "query" }` — use `{ type: "question" }` or `{ type: "highlights" }` instead. + +## Source Of Truth + +- `firecrawl/apps/js-sdk/firecrawl/package.json` +- `firecrawl/apps/js-sdk/firecrawl/src/index.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/client.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/types.ts` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/python.mdx b/agent-quickstart/python.mdx new file mode 100644 index 000000000..06e89d2dd --- /dev/null +++ b/agent-quickstart/python.mdx @@ -0,0 +1,196 @@ +--- +title: "Python Agent Quickstart" +description: "Canonical Firecrawl Python quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Python Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`firecrawl-py` **v4.33.1**) and the v2 OpenAPI spec. Method names, parameters, and types match the v2 client in `firecrawl/v2/client.py`. + +## Install + +```bash +pip install firecrawl-py +``` + +Requires Python >= 3.8. + +## Authenticate + +```python +import os +from firecrawl import Firecrawl + +client = Firecrawl(api_key=os.environ.get("FIRECRAWL_API_KEY")) +# client = Firecrawl(api_key="fc-...", api_url="https://api.firecrawl.dev") +``` + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. For multi-step interactive flows, prefer `interact` over scrape-time `actions`. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, **options)` → `SearchData` + +### Example + +```python +results = client.search( + "site:docs.firecrawl.dev webhook retries", + limit=5, + scrape_options=ScrapeOptions( + formats=["markdown"], + only_main_content=True, + ), +) + +for item in results.web or []: + print(getattr(item, "url", None), getattr(item, "title", None)) +``` + +Results are grouped by source: `results.web`, `results.news`, `results.images`. Do not access `results.data`. + +### Parameters + +- `query` — str (required). The search query. Use `site:example.com` to limit results to a domain. +- `sources` — list of `"web" | "news" | "images"` or `Source(type=...)` objects. Controls which sources are searched. +- `categories` — list of `"github" | "research" | "pdf"` or `Category(type=...)` objects. Filters results by category. +- `include_domains` — list of str. Restrict results to these domains. Mutually exclusive with `exclude_domains`. +- `exclude_domains` — list of str. Exclude results from these domains. Mutually exclusive with `include_domains`. +- `limit` — int. Maximum number of results. Default: `5`. +- `tbs` — str. Time-based filter (e.g. `"qdr:d"` for past day, `"qdr:w"` for past week). +- `location` — str. Localized results (e.g. `"San Francisco,California,United States"`). +- `highlights` — bool. Generate query-relevant highlights. Default: `True`. +- `ignore_invalid_urls` — bool. Drop URLs that cannot be scraped by other endpoints. +- `timeout` — int. Request timeout in milliseconds. Default: `300000`. +- `scrape_options` — `ScrapeOptions`. Scrape each search result (see Scrape parameters for fields). +- `enterprise` — list of str. Enterprise options: `["zdr"]` for zero data retention, `["anon"]` for anonymized. +- `threat_protection` — `ThreatProtectionOptions`. Enterprise threat protection override. +- `integration` — str. Integration identifier for tracking. + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +`client.scrape(url, **options)` → `Document` + +### Example + +```python +doc = client.scrape( + "https://docs.firecrawl.dev", + formats=["markdown"], + only_main_content=True, +) + +print(doc.markdown) +``` + +### Parameters + +- `url` — str (required). The URL to scrape. +- `formats` — list of format strings or format dicts. Output formats to request. + - Plain strings: `"markdown"`, `"html"`, `"rawHtml"` (or `"raw_html"`), `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"` (or `"change_tracking"`), `"attributes"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. + - Object-only formats (pass as dicts): + - `{"type": "json", "prompt": "...", "schema": {...}}` — at least one of `prompt` or `schema` is required. Do not pass `"json"` as a plain string. + - `{"type": "question", "question": "..."}` — question-answer extraction. + - `{"type": "highlights", "query": "..."}` — relevant source-text extraction. + - `{"type": "screenshot", "full_page": True, "quality": 80, "viewport": {"width": 1280, "height": 720}}`. + - `{"type": "changeTracking", "modes": ["git-diff"], "schema": {...}, "prompt": "...", "tag": "..."}` — `modes` is required. + - `{"type": "attributes", "selectors": [{"selector": "a", "attribute": "href"}]}`. +- `headers` — dict of str to str. Custom request headers. +- `include_tags` — list of str. Include only specific HTML tags. +- `exclude_tags` — list of str. Exclude specific HTML tags. +- `only_main_content` — bool. Strip nav, footer, and other boilerplate. +- `timeout` — int. Timeout in milliseconds. +- `wait_for` — int. Wait for the page to render (milliseconds). +- `mobile` — bool. Use a mobile viewport. +- `parsers` — list of `"pdf"` or `{"type": "pdf", "mode": "fast" | "auto" | "ocr", "max_pages": int}`. File parsing controls. +- `actions` — list of action dicts. Pre-scrape browser actions. + - `{"type": "wait", "milliseconds": int}` or `{"type": "wait", "selector": str}` + - `{"type": "click", "selector": str}` + - `{"type": "write", "text": str}` — click to focus the input first. + - `{"type": "press", "key": str}` + - `{"type": "scroll", "direction": "up" | "down", "selector": str}` + - `{"type": "screenshot", "full_page": bool, "quality": int, "viewport": dict}` + - `{"type": "scrape"}` + - `{"type": "executeJavascript", "script": str}` + - `{"type": "pdf", "format": str, "landscape": bool, "scale": float}` +- `location` — dict with `country` and `languages` keys, or `Location(country="US", languages=["en-US"])`. +- `skip_tls_verification` — bool. Skip TLS verification. +- `remove_base64_images` — bool. Drop base64 images from markdown output. +- `fast_mode` — bool. Faster scrapes with reduced fidelity. +- `block_ads` — bool. Ad and cookie popup blocking. +- `proxy` — str. `"basic"`, `"stealth"`, `"enhanced"`, or `"auto"`. +- `max_age` — int. Cached data up to a maximum age (milliseconds). +- `store_in_cache` — bool. Cache the result. +- `lockdown` — bool. Only serve cached results; never make outbound requests. +- `redact_pii` — bool or `RedactPIIOptions`. Redact personally identifiable information. +- `threat_protection` — `ThreatProtectionOptions`. Enterprise threat protection override. +- `audit_metadata` — `AuditMetadata(username=str)`. User attribution for SIEM logging. +- `profile` — dict with `name` and optional `save_changes`. Persistent browser profile shared across scrapes and interactions. +- `integration` — str. Integration identifier. + +## Interact + +### Why use it + +Use `interact` for code or natural-language control of the browser session tied to a scrape job (via `metadata.scrape_id`). At least one of `code` or `prompt` is required. For flows that go beyond quick pre-scrape tweaks, prefer `interact` over scrape-time `actions`. + +### Preferred SDK method + +`client.interact(job_id, code=None, *, prompt=None, language="node", timeout=None)` + +`prompt` is keyword-only: call `client.interact(job_id, prompt="...")` or `client.interact(job_id, code="...")`. + +### Example + +```python +doc = client.scrape("https://example.com", formats=["markdown"]) +job_id = doc.metadata.scrape_id if doc.metadata else None +if not job_id: + raise RuntimeError("Missing scrape_id from scrape response") + +result = client.interact(job_id, prompt="Click the pricing tab and summarize the plans.") +``` + +To end the session: `client.stop_interaction(job_id)`. + +### Parameters + +- `job_id` — str (required). Scrape job ID from `document.metadata.scrape_id`. +- `code` — str. Code to run in the browser session (optional if `prompt` is set). +- `prompt` — str (keyword-only). Natural-language instruction for the browser agent (optional if `code` is set). +- At least one of `code` or `prompt` must be non-empty. +- `language` — `"python" | "node" | "bash"`. Runtime language. Default: `"node"`. +- `timeout` — int. Execution timeout in seconds. + +## Notes + +- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` and `delete_scrape_browser` → `stop_interaction`; `scrape_url` → `scrape`. +- The top-level `Firecrawl` client exposes v2 methods directly; v1 remains under `client.v1`. +- `FormatString` accepts both camelCase API values (`"rawHtml"`, `"changeTracking"`) and snake_case alternatives (`"raw_html"`, `"change_tracking"`). +- Deprecated format type: `{"type": "query"}` — use `{"type": "question"}` or `{"type": "highlights"}` instead. +- `min_age` is available on `ScrapeOptions` when passed as `scrape_options` to `client.search`, but `client.scrape` does not accept `min_age` as a top-level keyword. + +## Source Of Truth + +- `firecrawl/apps/python-sdk/pyproject.toml` +- `firecrawl/apps/python-sdk/firecrawl/__init__.py` +- `firecrawl/apps/python-sdk/firecrawl/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/types.py` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/rust.mdx b/agent-quickstart/rust.mdx new file mode 100644 index 000000000..dc612959e --- /dev/null +++ b/agent-quickstart/rust.mdx @@ -0,0 +1,227 @@ +--- +title: "Rust Agent Quickstart" +description: "Canonical Firecrawl Rust quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Rust Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`firecrawl` crate **v2.12.1**) and the v2 OpenAPI spec. All public types are re-exported at the crate root. + +## Install + +```bash +cargo add firecrawl +``` + +## Authenticate + +```rust +use firecrawl::Client; + +let client = Client::new("fc-your-api-key")?; +// Self-hosted: +// let client = Client::new_selfhosted("http://localhost:3002", Some("fc-your-api-key"))?; +``` + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, SearchOptions, ScrapeOptions, Format}; + +let results = client + .search("site:docs.firecrawl.dev webhook retries", SearchOptions { + limit: Some(5), + scrape_options: Some(ScrapeOptions { + formats: Some(vec![Format::Markdown]), + only_main_content: Some(true), + ..Default::default() + }), + ..Default::default() + }) + .await?; + +if let Some(web) = &results.data.web { + for item in web { + // Each item is SearchResultOrDocument::WebResult or ::Document + } +} +``` + +Results are in `results.data.web`, `results.data.news`, `results.data.images`. + +### Parameters + +All fields on `SearchOptions` are `Option` and default to `None`. + +- `query` — `impl AsRef` (required). The search query. Use `site:example.com` to limit results. +- `options.sources` — `Vec`. Enum: `Web`, `News`, `Images`. +- `options.categories` — `Vec`. Enum: `Github`, `Research`, `Pdf`. +- `options.include_domains` — `Vec`. Restrict results to these domains. +- `options.exclude_domains` — `Vec`. Exclude results from these domains. +- `options.limit` — `u32`. Maximum number of results. +- `options.tbs` — `String`. Time-based filter (e.g. `"qdr:d"`, `"qdr:w"`). +- `options.location` — `String`. Localized results. +- `options.highlights` — `bool`. Generate query-relevant highlights. +- `options.ignore_invalid_urls` — `bool`. Drop URLs that cannot be scraped. +- `options.timeout` — `u32`. Request timeout in milliseconds. +- `options.scrape_options` — `ScrapeOptions`. Scrape each search result (see Scrape parameters). +- `options.integration` — `String`. Integration identifier for tracking. + +### Convenience helper + +`client.search_and_scrape(query, limit)` → `Result, FirecrawlError>` — calls search with default `ScrapeOptions` and returns only `Document` entries. + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +`client.scrape(url, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format}; + +let doc = client + .scrape("https://docs.firecrawl.dev", ScrapeOptions { + formats: Some(vec![Format::Markdown]), + only_main_content: Some(true), + ..Default::default() + }) + .await?; + +println!("{}", doc.markdown.unwrap_or_default()); +``` + +### Parameters + +All fields on `ScrapeOptions` are `Option` and default to `None`. + +- `url` — `impl AsRef` (required). The URL to scrape. +- `options.formats` — `Vec`. Output formats. + - Simple variants: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Product`, `Menu`, `Audio`, `Video`. + - Parameterized variants: + - `Format::Question(QuestionFormat { question })` — question-answer extraction. + - `Format::Highlights(HighlightsFormat { query })` — relevant source-text extraction. +- `options.headers` — `HashMap`. Custom request headers. +- `options.include_tags` — `Vec`. Include only specific HTML tags. +- `options.exclude_tags` — `Vec`. Exclude specific HTML tags. +- `options.only_main_content` — `bool`. Strip nav, footer, and boilerplate. +- `options.timeout` — `u32`. Timeout in milliseconds. +- `options.wait_for` — `u32`. Wait for the page to render (milliseconds). +- `options.mobile` — `bool`. Use a mobile viewport. +- `options.parsers` — `Vec`. `ParserConfig::Simple("pdf")` or `ParserConfig::Pdf { parser_type, mode, max_pages }`. +- `options.actions` — `Vec`. Pre-scrape browser actions. + - `Action::Wait { milliseconds, selector }` + - `Action::Click { selector }` + - `Action::Write { text }` + - `Action::Press { key }` + - `Action::Scroll { direction, selector }` — direction: `Up` or `Down`. + - `Action::Screenshot { full_page, quality, viewport }` + - `Action::Scrape` + - `Action::ExecuteJavascript { script }` + - `Action::Pdf { format, landscape, scale }` +- `options.location` — `LocationConfig { country, languages }`. +- `options.skip_tls_verification` — `bool`. Skip TLS verification. +- `options.remove_base64_images` — `bool`. Drop base64 images from markdown. +- `options.fast_mode` — `bool`. Faster scrapes with reduced fidelity. +- `options.block_ads` — `bool`. Ad and cookie popup blocking. +- `options.proxy` — `ProxyType`. Enum: `Basic`, `Stealth`, `Enhanced`, `Auto`. +- `options.max_age` — `u32`. Cached data up to a maximum age (milliseconds). +- `options.min_age` — `u32`. Cached data only if at least this old (milliseconds). +- `options.store_in_cache` — `bool`. Cache the result. +- `options.lockdown` — `bool`. Only serve cached results. +- `options.redact_pii` — `bool`. Redact personally identifiable information. +- `options.audit_metadata` — `AuditMetadata { username }`. User attribution for SIEM logging. +- `options.profile` — `ProfileConfig { name, save_changes }`. Persistent browser profile. +- `options.integration` — `String`. Integration identifier. +- `options.json_options` — `JsonOptions { schema, system_prompt, prompt }`. JSON extraction options (used with `Format::Json`). +- `options.screenshot_options` — `ScreenshotOptions { full_page, quality, viewport }`. Screenshot options. +- `options.change_tracking_options` — `ChangeTrackingOptions { modes, schema, prompt, tag }`. Modes: `GitDiff`, `Json`. +- `options.attribute_selectors` — `Vec`. Attribute extraction. + +## Interact + +### Why use it + +Use `interact` for code or natural-language control of the browser session tied to a scrape job. At least one of `code` or `prompt` is required (returns `FirecrawlError::Misuse` otherwise). + +### Preferred SDK method + +`client.interact(job_id, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, ScrapeExecuteOptions, Format}; + +let doc = client + .scrape("https://example.com", ScrapeOptions { + formats: Some(vec![Format::Markdown]), + ..Default::default() + }) + .await?; + +let job_id = doc.metadata + .as_ref() + .and_then(|m| m.additional.get("scrapeId")) + .and_then(|v| v.as_str()) + .expect("Missing scrapeId"); + +let result = client + .interact(job_id, ScrapeExecuteOptions { + prompt: Some("Click the pricing tab and summarize the plans.".to_string()), + ..Default::default() + }) + .await?; +``` + +To end the session: `client.stop_interaction(job_id).await?`. + +### Parameters + +- `job_id` — `impl AsRef` (required). Scrape job ID. +- `options.code` — `Option`. Code to run in the browser session. +- `options.prompt` — `Option`. Natural-language instruction for the browser agent. +- At least one of `code` or `prompt` must be non-empty. +- `options.language` — `ScrapeExecuteLanguage`. Enum: `Python`, `Node`, `Bash`. Default: `Node`. +- `options.timeout` — `u32`. Execution timeout in seconds. + +## Notes + +- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` and `delete_scrape_browser` → `stop_interaction`. +- `ScrapeOptions` uses dedicated sub-structs (`json_options`, `screenshot_options`, `change_tracking_options`) for advanced format configuration. +- `search_and_scrape(query, limit)` is a convenience helper that returns `Vec`. +- Options structs derive `Default` — use `..Default::default()` for partial initialization. +- Methods accept `impl Into>` for options, so you can pass `None`, `Some(opts)`, or just `opts`. +- Deprecated format variant: `Format::Query(QueryFormat)` — use `Format::Question` or `Format::Highlights` instead. + +## Source Of Truth + +- `firecrawl/apps/rust-sdk/Cargo.toml` +- `firecrawl/apps/rust-sdk/src/lib.rs` +- `firecrawl/apps/rust-sdk/src/client.rs` +- `firecrawl/apps/rust-sdk/src/scrape.rs` +- `firecrawl/apps/rust-sdk/src/search.rs` +- `firecrawl/apps/rust-sdk/src/types.rs` +- `firecrawl-docs/api-reference/v2-openapi.json`