AutoResearch (Karpathy) - Applied to Marketing

Karpathy's open-source autonomous-ML-research loop — and the community insight that the same propose → test → keep-if-better "ratchet" transfers directly to non-ML domains like marketing copy.

What AutoResearch actually is

karpathy/autoresearch is an open-source tool Andrej Karpathy published on GitHub in March 2026. It hands an AI coding agent (Claude Code, Codex, or equivalent) a minimal single-GPU LLM training setup — a simplified single-file nanochat core small enough to fit in an LLM context window — and lets it run experiments overnight with no human in the loop. Source: GitHub, https://github.com/karpathy/autoresearch, 2026-07-03 Source: DataCamp, https://www.datacamp.com/tutorial/guide-to-autoresearch, 2026-06-13

Three files split ownership: prepare.py (immutable — data, tokenization, evaluation), train.py (the only file the agent edits — full GPT model, Muon + AdamW optimizer, training loop), and program.md (human-authored instructions; you "program" the research org in Markdown, not Python). The loop: propose a code change → train for a fixed 5 minutes → measure validation bits-per-byte → git commit if it improved, git reset if not → repeat (~12 experiments/hour, ~100 overnight). The git ratchet keeps only validated wins. Source: GitHub, https://github.com/karpathy/autoresearch, 2026-06-13

Unlike AutoML / NAS (Optuna, Ray Tune), which search a fixed configuration space, AutoResearch lets the agent modify arbitrary code from its own ML knowledge — an open-ended search in code rather than over hyperparameters. Fortune dubbed the pattern "the Karpathy Loop." Source: Verdent, https://www.verdent.ai/guides/what-is-autoresearch-karpathy, 2026-06-13

Cross-domain transfer (the marketing angle)

@shannholmberg surfaced applying the same loop to marketing campaign optimization — Ole Lehmann ran the ratchet over landing-page copy and went from a 56% to a 92% pass rate overnight. 3,792 likes, 10,402 bookmarks. Source: X/@shannholmberg, 2026-03-24

The reviewed diagram translates the ML loop into a skill-improvement loop: write a 3-6 item checklist, run the skill once to get a baseline score, then let the loop change one thing in the skill prompt, rerun, score, keep if better, revert if worse, and save a changelog. The diagram explicitly generalizes the scorer to cold outreach variants, page speed, newsletter intros, and "any prompt you use repeatedly." That is the useful part for Kevin: AutoResearch is a ratchet pattern for repeatable agent work, not only an ML repo. Source: local X image artifact, 2026-07-03

The takeaway: a research-automation loop built for ML has immediate second-order uses anywhere you can define a scorer plus a candidate-edit step — copy, content, competitive intel. Kevin's Security and Review Skills is structurally similar (automated multi-source research synthesized into a brief); the difference is only the objective (campaign metrics or signal quality vs. validation loss). Both automate "propose, score, keep the winner."

Stack fit (Kevin)

• The harness IS the program — AutoResearch's "you program program.md, not the code" thesis is the same bet as kevin-wiki: the agent does the work; the human curates the instructions. Direct lineage with Karpathy LLM Wiki Setup Guide.
• Ratchet = eval loop — the keep-if-better git mechanism is an The Eval Loop (Slop Is an Output Problem) with a hard scorer; reuse it for any agent task with a 0–1 metric (humanizer scores, content rubrics, skill trigger-evals).
• Runs on Kevin's existing agents — it is agent-agnostic (Claude Code / Codex), so it slots into the harness Kevin already runs without a new runtime.
• Watch, don't adopt for marketing yet — the 56%→92% marketing result is a single viral anecdote; pilot it behind a real scorer before trusting it as a workflow.

Source Snapshot

---

Timeline

• 2026-07-03 | Deep-reviewed the local AutoResearch marketing diagram and refreshed GitHub source authority. Added the scorer/checklist skill-loop interpretation and current untagged master snapshot. Source: X/@shannholmberg, 2026-03-24; Source: GitHub karpathy/autoresearch, 2026-07-03
• 2026-03-24 | @shannholmberg on AutoResearch for marketing — Ole Lehmann landing-page test 56%→92% pass rate. 3,792 likes, 10,402 bookmarks. Source: X/@shannholmberg, 2026-03-24
• 2026-06-13 | Quality remediation: grounded the page in the real karpathy/autoresearch tool (~86K stars, 3-file ratchet loop, 5-minute val-bpb cycles) rather than only the marketing tweet; added eval-loop framing and stack fit; tier 3 → 2. Source: https://github.com/karpathy/autoresearch