workflowresearchexperimentsevaluationablationreproducibilitymlUpdated 2026-08-11

Research Experiment Workflow

Convert a question into a reproducible result: research the strongest prior recipes, freeze a baseline and evaluator, test one exact implementation, diagnose and ablate, then keep only improvements that survive held-out proof.

Use this workflow for recurring empirical work whose result can change a model, agent harness, retrieval policy, prompt, dataset, or product behavior. It is not the route for a literature summary alone; use the research tools in Capability Routing Map when no experiment will run.

Trigger and required inputs

Start only when all of these can be named:

  • question, target outcome, and explicit non-goals
  • current baseline and the exact artifact/revision that produced it
  • evaluation dataset or cases, metric, threshold, and held-out split
  • controllable variables and the first falsifiable hypothesis
  • time, compute, and money budget
  • execution environment, credentials, data rights, and approval owner
  • result destination and the canonical owner that will change if the result wins

If no evaluator exists, build the smallest discriminating evaluator before optimizing. If the benchmark cannot distinguish a real improvement from noise, the experiment is blocked rather than merely inconvenient.

The loop

  1. Freeze the claim and baseline. Record the hypothesis, primary metric, guardrail metrics, baseline command/config/revision, random seeds, evaluator version, stop condition, and what would falsify the claim. Keep tuning cases separate from held-out confirmation cases.
  2. Research attributed recipes. Start with primary papers and current implementation/docs. Follow citations downstream, read methodology and experiment sections rather than abstracts, and bind each proposed recipe to the result, dataset, configuration, and limitations that produced it.
  3. Audit the data. Verify source, license/consent, revision, schema, splits, duplicates, missing values, class balance, contamination, sample rows, and compatibility with the intended method. Synthetic or transformed data must retain generator, prompt/config, provenance, and a separate quality check.
  4. Write the experiment ledger. Rank candidate recipes by evidence and feasibility. Choose one change, state its expected effect, and record all fixed controls. Do not silently substitute a model, dataset, sequence length, method, or evaluator when the chosen recipe fails.
  5. Preflight the exact source. Use an isolated, replaceable environment. Print dependency/hardware versions, assert inputs and output destinations, run syntax/import checks, then execute a tiny path through the same entrypoint and dependencies. GPU work gets a representative GPU smoke test when feasible. Persist the exact tested source rather than reconstructing it later.
  6. Run one pilot. Submit one real run and verify startup, data loading, logging, evaluation, checkpoint/output persistence, and cost visibility. Only then launch a sweep or independent batch; one broken pilot is cheaper and more informative than many identical failures.
  7. Observe and diagnose. Capture configs, commits, data/evaluator revisions, logs, metrics, alerts, cost, duration, artifacts, and failure category. Read the actual result before choosing the next action. Retry only after the diagnosis identifies a meaningful change.
  8. Ablate and ratchet. Change one causal variable where possible. Use negative controls and repeated seeds when variance matters. Compare to the frozen baseline, keep an improvement only when the primary metric clears the threshold without violating guardrails, and revert or quarantine regressions.
  9. Confirm out of sample. Run the selected candidate against the untouched held-out set and, for consequential claims, an independent evaluator or reproduction. Treat launch charts, best-of-many runs, and self-authored judge scores as leads until this gate passes.
  10. Write back the result. Save the winning artifact, rejected alternatives, complete receipt, residual uncertainty, and next question. Update the canonical workflow/skill/design/tool/project owner only when the result changes future behavior. Add failed cases to the evaluator when they expose a reusable failure mode.

Curriculum and recipe-corpus intake

Teaching repositories, notebooks, slide decks, and “awesome” experiment lists are recipe sources—not executable authority. Retain the complete pinned corpus, then select one chapter because a named question needs it. Before running:

  1. Map the lesson to its primary paper, current upstream implementation, model, dataset, evaluator, and claimed result. Preserve the lesson as secondary explanation rather than silently promoting it to primary evidence.
  2. Resolve the corpus and every imported code/data/model license. Free access, a public GitHub repository, and stars do not grant reuse or redistribution.
  3. Inspect the actual notebook cells, saved outputs, install commands, mutable clones, remote downloads, API keys, GPU/disk assumptions, and environment metadata. Convert prose-only notebooks into explicit run plans rather than calling them executable.
  4. Pin all upstream revisions and build an isolated environment from a clean dependency lock. Replace embedded credentials with scoped environment variables; do not inherit unreviewed mirrors or shell commands.
  5. Run one tiny representative case through the current evaluator and record every repair. A repaired teaching exercise becomes a local derivative with its own receipt; it does not retroactively prove the source corpus current.

Dive into LLMs is the first retained fixture for this lane: 11 README/PDF/notebook chapters covering fine-tuning, prompting, knowledge editing, math distillation, watermarking, jailbreaks, steganography, multimodality, GUI agents, agent safety, and RLHF. The full corpus is useful, but seven notebooks have no code cells, only one chapter has a local requirements file, the repository has no detected license, and source head is 2025-10-10. Use it to find and teach candidate experiments, then rebuild the selected recipe against current primary sources. High-risk security and covert-channel chapters require isolated defensive scope and explicit output and publication boundaries. Source: Lordog/dive-into-llms@f84c042; current replay 2026-08-11

Sutskever's 30 papers implemented is the second fixture. Its NumPy notebooks are a useful paper-to-code curriculum, but the saved "100% completed" claim is not experiment proof: the pinned 225cba7b tree has no detected license and no test files. Admit one paper at a time. Bind the primary paper and exact theorem/algorithm, dataset or synthetic generator, expected intermediate values, reference implementation, numerical tolerances, known simplifications, dependency revision, and executable output. Add focused tests and compare the notebook against both the paper and a current trusted implementation before calling it reproduced. Preserve teaching value when a notebook is incomplete; do not backfill missing math or results with a model and then attribute them to the source. Source: X 2077261375567745064; pageman/sutskever-30-implementations@225cba7b, reviewed 2026-08-12

Authority and privacy gates

  • Prefer a task-scoped sandbox or HF Job over local checkout access. A sandbox limits blast radius; it does not grant permission to publish data, spend money, create recurring jobs, or mutate durable repositories.
  • Headless or YOLO modes are disallowed for normal runs unless their exact tool, spend, data-egress, and cancellation limits have been independently enforced.
  • Scheduled jobs, public uploads, model/dataset publication, paid compute, destructive repository actions, and scope-changing substitutions require explicit owner approval.
  • Use least-privilege, task-scoped credentials. Never put secrets in prompts, source files, traces, dashboards, or experiment outputs.
  • Trace and telemetry upload is opt-in after inspecting the actual payload and destination. “Private,” “scrubbed,” and “anonymous” are separate claims and each needs proof.

Tool routing

Experiment shape First route Boundary
HF-native model/data/training research ML Intern - Hugging Face Research Agent as an evaluated candidate Use interactive sandbox mode; active use is held until local-only trace handling is proven.
Practical LLM curriculum or teaching-recipe discovery Pinned Dive into LLMs corpus Retain all 11 themes; select one by the question, then trace current primary sources, resolve licenses, pin dependencies, sandbox, pilot, and evaluate. Never assume every notebook is executable.
Paper-to-NumPy implementation curriculum Pinned Sutskever 30 implementations Select one paper; verify math, data, expected intermediates, current reference, license, tests, and numerical output. A completed notebook or social count is not a reproduced result.
Small code-level optimizer with a fixed scorer AutoResearch (Karpathy) - Applied to Marketing pattern Immutable evaluator, bounded edit surface, hard keep/revert ratchet.
Agent-orchestration variable research CoTCodec harness + Harness Ablation Design Hold benchmark and significance method constant; ablate one orchestration variable.
Planner/worker swarm economics Harness Ablation Design with one owned task graph Freeze repository, task tree, held-out tests, context, budgets, merge policy, and reviewer lenses; measure worker-token share, conflicts, reconciliation cost, time, quality, and variance before changing topology.
General stochastic product/content output The Eval Loop (Slop Is an Output Problem) Use saved cases, rubric/validator, threshold, runtime/production sampling.

No tool owns the scientific claim. The frozen evaluator, provenance, exact artifact, and replay receipt do.

Required run receipt

Write an evidence-heavy run to outputs/<YYYY-MM-DD>/research-experiment/<experiment-id>/ with:

  • question.md: hypothesis, baseline, controls, budget, approval boundary
  • sources.md: papers, citation paths, docs, code, datasets, licenses
  • experiments.jsonl: one immutable row per run/config/revision/result
  • artifacts/: exact source, logs, metrics, charts, checkpoints or durable links
  • decision.md: keep/revert/blocked, held-out result, rejected alternatives, uncertainty, and owner writeback

Stable experiment IDs and content hashes make retries idempotent. A resumed run must load the ledger and verify remote job/artifact state before resubmitting.

Failure handling

Failure Response
Dataset or evaluator changed mid-run Invalidate comparison, version the new input, and rerun baseline.
Pilot fails Diagnose and repair the same recipe; do not fan out a batch.
OOM or infrastructure failure Preserve the method; adjust implementation-neutral resources first. Ask before changing the scientific scope.
Metric improves only on tuning cases Reject as overfit; add harder held-out cases if the evaluator is saturated.
Result depends on one seed or best-of-many selection Repeat and report variance/distribution, not only the maximum.
Trace/privacy boundary is unclear Disable external run or use a reviewed local-only wrapper; do not infer safety from “private.”
Budget or time expires Save the exact resumable state and report partial evidence without upgrading the claim.

Completion gate

The run follows Workflow Run Contract and is complete only when a future agent can replay the baseline and winning candidate, locate every source and artifact, distinguish tuning from held-out evidence, see authority/cost/data decisions, and explain why the result was kept, reverted, held, or blocked.

Timeline

Related Pages

The Eval Loop (Slop Is an Output Problem)Empirical VerificationHarness Ablation DesignML Intern - Hugging Face Research AgentAutoResearch (Karpathy) - Applied to MarketingWorkflow Run ContractHuman-in-the-Loop Control

5 pages link here

Actionable Tools IndexToolsCapability Routing MapMetaML Intern - Hugging Face Research AgentToolsThe Eval Loop (Slop Is an Output Problem)ConceptsX Bookmarks: AI Agents & Tools (Jan 2025 – Jun 2026)Tools