Computer Use and Browser Automation Patterns
Agents need to use computers. The approaches range from pixel-level screenshots to turning websites into structured APIs. The right tool depends on the task shape. Source: compiled from wiki, 2026-05-19
The Spectrum
Browser automation for agents exists on a spectrum from low-level (pixels) to high-level (structured APIs):
| Approach | How it works | Best for | Fragility |
|---|---|---|---|
| Screenshot + vision | Take screenshot, use vision model to decide clicks | Universal fallback, visual verification | High (layout changes break it) |
| DOM/accessibility tree | Parse page structure, use refs to interact | Structured interaction with known pages | Medium (selector changes) |
| CDP snapshots | Chrome DevTools Protocol, ref-based element targeting | Agent-native browsing, React introspection | Low-medium |
| Site-specific skills | Pre-researched playbooks per website | Reliable repeated tasks on known sites | Low (maintained playbooks) |
| Websites as APIs | Convert web content to structured data (Reflex, Jina Reader) | Data extraction, content parsing | Low (content-focused) |
| Web directories | Curated catalogs of site capabilities | Agent discovery of what sites can do | Very low |
Key Players
| Tool | Approach | Role |
|---|---|---|
| agent-browser | Compact browser-command CLI/MCP, ref snapshots, sessions/profiles/auth, Chrome/Lightpanda, screenshots/diffs, React/perf commands | Default agent browser substrate |
| Playwright | Programmatic browser control, E2E testing | Committed regression tests and CI gates |
| browser-harness (browser-use) | Self-healing CDP harness, screenshot + coordinate-first, real Chrome, domain skills | Specialist fallback for screenshot-first real-Chrome flows |
| Stagehand (Browserbase) | AI browser automation, natural language | Cloud browser / Browserbase workflows |
| Browse.sh (Browserbase) | Site-specific skill catalog | Persistent site playbooks |
| Lightpanda | Fast headless browser, CDP-based | agent-browser engine option |
| Jina Reader | r.jina.ai/URL turns any page to markdown |
Free, zero-config |
| Defuddle | npx defuddle parse URL --markdown local extraction |
No API, offline capable |
| Autobrowse | Persistent browser playbooks (learn site, save SKILL.md) | Amortized discovery |
| Anthropic Computer Use | Full desktop automation via screenshots + vision | General computer tasks |
| cursor-ide-browser | Cursor's built-in vision pipeline | IDE-integrated browsing |
| Chrome DevTools MCP | Inspect user's existing browser session | Live debugging |
| Hermes creative GUI loop | Desktop navigation + visual/reference iteration inside a creative app | Learning tools such as TouchDesigner with a human reviewing output |
Architecture Decision
Kevin's browser automation ranking (from tool-hierarchy.mdc):
- agent-browser: default for browsing, scraping, form filling, screenshots, logged-in Chrome, UI QA, React/perf, and compact batch command loops.
- agent-browser MCP: same substrate when the active harness is MCP-first.
- browser-harness: specialist fallback for screenshot-first self-healing real-Chrome flows.
- Chrome DevTools MCP: computed styles, box model, DOM traversal, and exact device emulation.
- Playwright: deterministic E2E in CI and committed regression tests only.
For persistent playbooks: Autobrowse / Browse.sh (learn a site once, reuse forever).
For content extraction (not interaction): Jina Reader > Defuddle > WebFetch.
The Insight
Most browser automation fails because it treats every page visit as exploration from scratch. The compounding approach: research a site once, encode the navigation into a skill or playbook, reuse on every subsequent visit. Browse.sh and Autobrowse both implement this pattern. The agent investment shifts from per-task browsing to per-site onboarding. Source: compiled from wiki, 2026-05-19
The same compounding rule applies outside browsers. Amber Shen's Hermes/TouchDesigner demo shows a GUI creative-app version: the agent navigates the desktop, discovers how to connect to TouchDesigner, reads reference images, iterates with the human, and saves output. Treat this as "learn a tool, then crystallize the workflow" rather than as permission to let a screenshot agent freestyle production creative work without artifact review. Source: X/@whosamberella, 2026-06-13; Source: local video contact sheet, 2026-07-02
Testing: Route by Test Type
The same principle applies to browser testing. Instead of ranking tools monolithically, split by test type: agent-browser for exploratory rendered proof and logged-in interaction; Playwright for deterministic CI gates after behavior is worth locking. Findings from the agent loop feed back into Playwright as codified assertions. See Complementary Browser Testing for the full framework. Source: User, 2026-05-19; User request, 2026-07-04
Timeline
- 2026-07-02 | Added Hermes + TouchDesigner as a creative-GUI computer-use pattern: desktop navigation, tool-connection discovery, reference-image reading, human iteration, and saved output. Source: X/@whosamberella, 2026-06-13
- 2026-05-19 | Page created. Compiled from wiki tool pages, Browse.sh launch, and Kevin's browser automation stack. Source: User + wiki + web, 2026-05-19
- 2026-05-19 | Added testing section. Kevin articulated the complementary browser testing pattern: Playwright for determinism, Chrome MCP + agent loop for visual judgment. Cross-linked to Complementary Browser Testing. Source: User, 2026-05-19