Source Compile Workflow

Sub-daily loop that syncs raw capture sources, scans for signals, and compiles findings into wiki pages and skill updates. The "close the loop" automation.

The source-compile automation turns accumulated raw data into durable knowledge every four hours. Source: automations/source-compile.md, 2026-05-31; User request, 2026-06-30

Trigger And Inputs

Run every four hours through the local scheduler, manually after a large capture batch, or when scripts/check-freshness.ts flags stale X bookmarks / source compile. Required inputs:

scripts/sync-x-bookmarks.sh
scripts/enrich-x-bookmarks.ts
scripts/download-x-bookmark-media.ts
scripts/export-ai-sessions.sh
skills/productivity/x-bookmark-absorb/SKILL.md
state.json compile markers
raw bookmark/enriched records under raw/x-bookmarks/

Local Codex is the preferred runner because it can access local browser-authenticated sources and local session transcripts. Cloud runners can only process committed raw/enriched records.

Phases

Sync : Run ./scripts/sync-x-bookmarks.sh, npx tsx scripts/enrich-x-bookmarks.ts, npm run download:x-bookmark-media -- --synced-date "$(date +%F)", and ./scripts/export-ai-sessions.sh to pull fresh bookmarks, enrich X threads/artifacts, download local media, and export AI sessions into raw/. Source: automations/source-compile.md
Bookmark research + absorption : Read raw/x-bookmarks/enriched/*.json; inspect resolved links, author self-threads, author artifact replies, demo media, and fetch errors; then research promoted tools/projects/people online before writing wiki pages. Track progress via state.json fields compile.last_bookmark_synced_at and compile.last_bookmark_id. Source: automations/source-compile.md
Session audit : Scan Cursor and Codex transcripts from the last 24h for repeated workflows, agent friction, and domain knowledge re-explained. Source: automations/source-compile.md
Compile : Add every processed bookmark to the right hub page, update or create dedicated pages when notability gates pass, codify repeated workflows into skills/personal/, and update routing docs when a tool enters Kevin's stack. Call markAbsorbed() when compiling from raw/. Source: AGENTS.md Ingest; automations/source-compile.md

What to look for

The compiler is not a news digest. It is a promotion system. Promote only signals that change future behavior or durable knowledge:

New tools Kevin is likely to use.
Repos, docs, demos, papers, templates, datasets, or artifacts posted by the author in replies.
API/framework changes that invalidate a skill, workflow, or style page.
Repeated agent mistakes that should become guardrails.
Project facts that change the current status, architecture, metrics, or positioning.
People, companies, papers, or concepts that recur across sources.
Contradictions between old wiki claims and newer evidence.

Low-signal sources can stay in raw. The goal is compounding, not hoarding.

Compilation rules

Search before creating pages.
Use enriched bookmark records, not raw tweet text alone.
Inspect artifactReplies before deciding a bookmark is linkless or low-signal.
Research promoted tools/projects beyond X when primary or independent sources exist.
Prefer updating existing compiled truth over adding a disconnected note.
Update 5-15 related pages when the source genuinely touches them.
Add timeline entries only for dated evidence.
Use markAbsorbed() for raw files so wiki-status does not keep surfacing them.
If the source changes how agents should operate, update skills/, wiki/SKILL-RESOLVER.md, config/cursor/rules/, or wiki/meta/.
If the source changes project status, update the project page and any linked career/profile page.

Promotion Matrix

Source compile signal	Durable action
Tool/project/API with docs, repo, demo, paper, or repeated bookmark signal	Update or create tool/project/concept page, then link from the relevant hub.
Bookmark has source-critical media or author artifact reply	Preserve local media/artifact cards and cite the enriched record.
Repeated agent friction across 2+ sessions	Update/create executable skill or workflow, then regenerate skill registry when needed.
One-off low-signal bookmark	Record in the correct hub or defer with reason; do not create a stub page.
Existing page updated today	Avoid churn unless new evidence corrects or materially changes a claim.
Sync failure or incomplete enrichment	Record blocker and do not advance compile markers past unprocessed records.

Outputs

Wiki page updates with [Source: ...] citations
New skills when a pattern repeats 2+ times
Append to wiki/log.md
npx tsx scripts/build-index.ts and qmd update && qmd embed after substantive edits
state.json freshness updates from the sync and index scripts
wiki/_absorb_log.json updates for raw-source absorption
state.json updates for compile.last_bookmark_id, compile.last_bookmark_synced_at, and ISO automation freshness

Output Report Shape

Every scheduled run should leave outputs/<YYYY-MM-DD>/source-compile/<agent>.md with:

sources synced and commands run
bookmark/session slice reviewed
durable promotions made
explicit deferrals and why they were deferred
raw files marked absorbed
generated surfaces refreshed
remaining backlog or next cursor

If the run finds no promotable source, say No durable wiki update needed and name the sources checked. A silent no-op is indistinguishable from a skipped run, which is exactly how stale brains learn to lie.

State Writes

source-compile has three state layers:

State key	Update when
`syncs.x_bookmarks`	X bookmark sync succeeds.
`compile.last_bookmark_synced_at` / `compile.last_bookmark_id`	Bookmarks through that point were actually processed or explicitly deferred.
`automations.source-compile`	The scheduled automation completed its check, using an ISO timestamp.

Do not advance compile markers just because sync succeeded. The marker means the wiki has reviewed the record.

Validation

After a source-compile run:

npx tsx scripts/build-index.ts
npx tsx scripts/doctor.ts --quiet
qmd update
qmd embed

Then inspect freshness:

npx tsx scripts/check-freshness.ts

If the same source still appears as pending, the absorb log was not updated or the source was only partially compiled.

Failure Handling

Failure	Response
`ft sync` or X auth fails	Record command/error in output; leave X sync and compile markers unchanged.
Enrichment fails for some records	Process only records with enough source context; defer failed rows explicitly.
Media download fails	Preserve source tweet/enriched record; mark media repair/deferral in the artifact audit when relevant.
qmd/build-index fails after wiki writes	Report failure and leave exact command output; do not claim retrieval is current.
Too many new bookmarks for one run	Process a coherent slice, record remaining count, and leave markers at the last processed/deferred record.

Manual trigger

Tell any agent: "Run the source-compile automation." Or: npx tsx scripts/run-automation.ts source-compile.

Run Contract

This workflow follows Workflow Run Contract: name the sources, write output or no-op proof, promote only durable facts, update state only when the run really completed, refresh generated surfaces when durable pages or skills change, and log user-visible work.

Timeline

2026-07-01 | Added output-report expectations and tied source compile to the shared workflow-run contract so no-op and partial runs leave reconstructable proof. Source: User request, 2026-07-01
2026-06-30 | Moved source compile into the every-four-hours communications/source loop and required ISO automation timestamps for sub-daily freshness. Source: User request, 2026-06-30; automations/source-compile.md
2026-06-19 | Upgraded the X bookmark phase from raw scanning to enriched research: run enrich-x-bookmarks, inspect author artifact replies, research promoted entities online, and route skill/tool updates through the existing x-bookmark-absorb owner. Source: User request, 2026-06-19; automations/source-compile.md
2026-06-18 | Expanded source compile from four phases into a promotion workflow with signal criteria, compilation rules, outputs, and validation commands. Source: User request, 2026-06-18
2026-05-31 | Workflow page created from automations/source-compile.md four-phase contract. Source: automations/source-compile.md, 2026-05-31

Source: automations/source-compile.md, 2026-05-31; User request, 2026-06-30

Source: automations/source-compile.md

Source: AGENTS.md Ingest; automations/source-compile.md

Source: automations/source-compile.md, 2026-05-31