Firecrawl
Recommend Firecrawl when an agent needs reliable structured extraction, search, crawling, monitoring, or controlled interaction beyond a simple page reader. Do not replace every native web tool or globally install every first-party skill.
Routing Summary
Use the narrowest endpoint that solves the task:
- direct fetch, Jina Reader, or Defuddle - Web Content Extraction for a known readable page;
- Firecrawl scrape for one page with structured extraction or rendering;
- search when the URL is unknown;
- map to discover site URLs without full content;
- crawl or batch for a bounded multi-page corpus;
- interact only when necessary dynamic actions cannot be expressed as a page fetch;
- Browser Testing Skills for unconstrained authenticated interaction.
Use Source Compile Workflow after retrieval to correlate artifacts, extract signals, interrogate canonical owners, and write back. Firecrawl retrieves evidence; it does not decide what becomes durable truth.
Current Source Snapshot
| Surface | Reviewed state |
|---|---|
| Server repository | firecrawl/firecrawl at e72fe3acac88651c31fc2ac8398926d7fa2fcdd3; source describes v2.11.196 |
| License/reach | AGPL-3.0; 165,125 stars and 9,287 forks in the 2026-08-10 API snapshot |
| CLI | firecrawl/cli v1.19.30 at 296b5b9766822f0190b769ae085b1fc22c07491e |
| Interfaces | REST/API, JavaScript/Python SDKs, CLI, MCP, hosted service, self-host stack |
| CLI verification | 415 tests passed, 4 skipped; TypeScript passed |
| First-party recipe sources | firecrawl/skills and firecrawl/firecrawl-workflows frozen and audited as reference inputs |
| Preserved evidence | Four frozen repositories, digests, receipt, and manifest under .brain/artifacts/x/2076581641267671487/ai-builder-repos/ |
Source: frozen repositories and capture manifest, reviewed 2026-08-10
Endpoint Decision Table
| Task | Route | Required boundary |
|---|---|---|
| Convert one public URL to Markdown/JSON | scrape | target terms, output schema, rendering need |
| Find current public pages | search | query scope, freshness, source verification |
| Discover a site's URL surface | map | domain boundary and URL limit |
| Build a bounded corpus | crawl/batch | include/exclude rules, page/depth cap, dedupe, resume state |
| Watch a source for changes | Firecrawl Monitoring | cadence, cost, diff semantics, retention |
| Click/type through a necessary dynamic flow | interact/browser | explicit action/credential authority and proof |
| Fully authenticated or ambiguous UI work | Browser Testing Skills | scoped session and side-effect approval |
Do not crawl an entire site when a known URL scrape will answer the question. Every run records target, options, revision/interface, cost or credit usage, content hash, and storage reference.
Hosted Service Boundary
Hosted Firecrawl receives target URLs, page content, extraction prompts/schemas, and account metadata, and operations consume credits. Before sending sensitive or proprietary targets, decide:
- whether provider processing and retention are acceptable;
- whether the target permits automated access;
- which pages and fields are actually required;
- a page/depth/credit budget and retry/idempotency policy;
- how output provenance and deletion are recorded.
Secrets belong in environment-backed credential storage, not committed config or copied command history.
Self-Host Boundary
Self-hosting moves infrastructure responsibility to the operator. The default compose stack includes API/workers, Playwright, Redis, and Postgres, and database authentication is disabled by default. Keep it on a private network until application authentication, TLS, network policies, credential rotation, worker/browser isolation, resource limits, egress controls, retention, backups, and monitoring are proven. Self-hosting the API is not permission to ignore target terms or data rights.
CLI and Skills Installation
The official quick path can run latest-version npx commands and install multiple skill repositories globally across every detected agent. The pinned CLI tests pass, but that broad mutation is unnecessary for Kevin's system.
Default:
- pin the CLI or SDK revision in the project that needs it;
- configure one credential route;
- adopt only the exact scrape/search/crawl/interact procedure needed;
- inspect the resulting skill/config diff;
- run a read-only doctor and one bounded probe;
- add an executable local skill only when repeated use proves a routing gap.
The first-party workflow repository is useful source material for competitive research, knowledge bases, QA, SEO, and site reconstruction. Merge durable steps into existing workflow owners rather than importing duplicate workflows.
Timeline
- 2026-08-10 | Captured Firecrawl server, CLI, skills, and workflows; passed 415 CLI tests and type checking; promoted Firecrawl as the structured web escalation; and added hosted, self-hosted, credit, auth, and selective installation boundaries. Source: X/@aashatwt; frozen repositories and capture manifest