Realtime Voice Agent Workflow

Build or evaluate a duplex voice agent as a measured stateful system—not a model demo. Freeze the complete path, prove each latency/quality/privacy boundary, and promote only a receipt-bearing route.

Trigger

Use this workflow when a product needs live microphone input, turn detection, speech recognition or native audio understanding, model/tool execution, and streamed speech output. Do not use it for ordinary file transcription, prerecorded voice generation, or interface sound effects.

Output

A completed run produces:

1. a feasibility classification, route decision, and frozen component manifest;
2. an authority/privacy/security manifest;
3. a representative evaluation corpus and network/device matrix;
4. per-stage and end-to-end traces, quality scores, tool receipts, cost, and failure taxonomy;
5. a UI state-machine and accessibility proof;
6. a promote, hold, replace, or reject decision with rollback/fallback.

Route Before Building

Do not choose a stack because one demo feels fast. The selected route must match language/runtime, client, hosting, privacy, hardware, model/provider, tools, and operator constraints.

Capability Signal → Product Route

A compelling use-case list is discovery input, not a build decision. Before a voice idea enters implementation, classify and prove five independent layers:

Failure in one layer cannot be repaired by more model reasoning. “The model can call a tool” does not prove the tool is authorized, accurate, fast, affordable, or allowed.

Opportunity lanes

The 17 GPT-Realtime idea signals compile into reusable lanes rather than 17 pages:

This preserves the post's valuable services-as-software map while rejecting its unsupported “silent mode,” “price of an API call,” 128K-equals-one-hour, and “impossible six months ago” claims. Source: X 2061129813750915508; audited against current OpenAI primary sources in .brain/artifacts/x/2061129813750915508/gpt-realtime-startup-ideas-review/source-audit.md

Current OpenAI Route

Freeze these as separate products and endpoints; do not treat “Realtime” as one interchangeable mode:

Use WebRTC for browser/mobile capture and playback, WebSocket for server-owned media pipelines, and SIP for supported telephony. Mint ephemeral client credentials on the server. Keep secrets, privileged tools, changing instructions, and business logic on a server/sideband control channel connected to the same session. Send a stable, privacy-preserving safety identifier for identified end users. Source: OpenAI Realtime overview and server-controls docs, captured 2026-08-11

For SDK function tools, set needsApproval on consequential calls, run deterministic input guardrails before the approval request and again before execution, validate the arguments independently because the model lacks Structured Outputs, and execute the frozen proposal idempotently. A Realtime session cannot process a new user request while it waits on the SDK approval event, so announce the pause, provide visible approve/reject controls, define expiry/cancel behavior, and move long approvals into durable job state. Output-transcript guardrails may interrupt speech, but they do not replace pre-tool authority checks. Source: OpenAI Agents SDK voice build guide, captured 2026-08-11

Frozen Experiment Contract

Record one manifest before the first benchmark:

Changing any verdict-bearing field creates a new experiment revision.

Runbook

1. Define the user job and authority. State who is speaking, what the agent may hear/see/store, what tools it may call, which actions require confirmation, and what silent/text fallback exists.
2. Select the narrowest architecture. Prefer a cascaded VAD → STT → LLM → TTS path when inspectability, replaceability, transcripts, or local components matter. Consider native audio only when it wins the same task/latency/privacy evaluation.
3. Freeze and isolate. Use a project environment, exact dependencies/model revisions, loopback by default, and a test identity/provider project. Do not globally install or expose an unauthenticated realtime server.
4. Draw two loops before one agent. Record the sensor → transport → realtime-model → playback perception loop separately from the task → action-agent → tool → receipt loop. Name every provider, model, gateway, vault, search, app, and persistence hop; never let “the agent” hide different authority or data processors.
5. Build the state machine first. Define signed-out/ready/connecting/connected-without-worker/model-starting/listening/user-speaking/processing/search/action-awaiting-confirmation/action-running/assistant-speaking/interrupted/degraded/error/stopped states and legal transitions. One state owner drives transport, transcript, controls, audio, animation, and recovery.
6. Bind identity and capabilities. Derive user identity server-side; use short-lived session/room tokens; scope room, publish source, subscribe, model, search, tool, app, and storage capabilities separately. A service credential plus caller-supplied user header is not identity proof.
7. Define visual reference. State whether “this” means the latest live frame or an explicitly pinned frame. Preserve orientation/crop/zoom metadata, show the pinned state, and attach the authoritative image to downstream work instead of relying on another model's lossy description.
8. Make confirmation a human state. Side-effecting or policy-gated work announces the proposed action, target, material parameters, and consequence; accepts an explicit approve/deny; expires on silence, hangup, session change, or input-hash change; and emits a receipt. The model, transport, recovery loop, or tool gateway may never answer its own always_ask/confirmation event.
9. Instrument every boundary. Emit monotonic timestamps for input capture, speech start/stop, final transcript, request start, first model token, first TTS chunk, first playback, local buffer clear, server cancellation, search/action dispatch, confirmation request/answer, tool start/end, late-result enqueue/delivery, response completion, and disconnect.
10. Design long work as a durable job. Assign an idempotency key and user/session owner, acknowledge after a bounded synchronous wait, report coarse progress without fabricated completion, persist through worker restart, and deliver the final result at most once in the current or next authorized session. A detached in-process promise is not durable execution.
11. Budget context explicitly. Cap realtime-session duration, accumulated audio/video context, action-history briefing, frame size/rate, proactive listening, and search/tool spend. Prefer a bounded recent-task ledger over replaying unbounded history; keep memory writes off the latency-critical path until measured value justifies them.
12. Run deterministic component checks. Test audio encoding/resampling, VAD boundaries, stale transcript rejection, event ordering, cancellation, tool argument assembly, playback buffer clearing, session/room recreation, token expiry, pin/unpin semantics, confirmation deny/timeout/reconnect, job dedupe/restart/single delivery, and empty/error cases without provider calls.
13. Run the representative corpus. Cover target accents/languages, background noise, echo, short/long turns, pauses, corrections, overlap, interruption, tool calls, dense visual text, moving/pinned frames, camera loss, worker absence, provider errors, mobile/wearable devices, and cold/warm starts.
14. Compare routes under one contract. Hold prompt, corpus, network, tools, concurrency, and evaluator constant. Report component and end-to-end results; do not compare a local cold run with a hosted warm demo or merge platform results from divergent architectures.
15. Review privacy and security. Trace every byte of audio, transcript, camera image, search query, tool result, prompt, vault credential, analytics/crash event, and log. Verify consent/bystander indicators, mute/stop, retention/deletion, secret isolation, auth/rate limits, TLS/origin/egress policy, tool approval, OAuth revoke, tenant isolation, and impersonation resistance.
16. Review the interface at normal and degraded states. Prove silent operation, visible transcript/state, keyboard focus, 44 px targets, 360 px layout, reduced motion, mute, stop, pin/unpin, voice-only fallback, reconnect, absent worker, provider failure, acoustic-echo-cancellation versus half-duplex state, and slow-network behavior.
17. Canary with real users only after synthetic proof. Start with short, non-sensitive, low-authority sessions. Preserve anonymized aggregate traces and explicit user feedback without retaining raw audio or camera data by default.
18. Promote or hold. Bind the accepted route to the project/tool owner, preserve the manifest and receipts, document fallback/rollback, and reopen review on any client, model, provider, transport, permission, processor, storage, or data-policy revision.

Multimodal Processor Map

For voice-plus-camera systems, complete this table rather than writing “uses AI”:

Evaluation Matrix

Interface Contract

GPT-Live-1 adds a useful architecture signal: keep interruptible listening and speaking in a low-latency duplex layer, while a backend reasoner owns tools and longer deliberation. The interface still needs explicit controls for interruption, tone, pacing, emotion, language, and response length. The announcement's linked official article was unavailable during review, so model-specific capability claims remain provisional. Source: X/@WorldLabsAI 2098099269551149398, reviewed 2026-09-11

Voice UI needs perceptual redundancy. Audio alone cannot communicate state, and animation alone cannot communicate meaning.

• Map every runtime state to text, controls, and optional motion/color; never use color or sound as the only cue.
• Let actual input/output energy drive the primary voice visualization. Timer-driven breathing may support idle state but cannot impersonate live audio.
• Keep transcript/tool history reviewable while marking partial versus final text.
• Barge-in clears local buffered audio immediately and cancels server work; stale text/audio/tool output cannot reappear afterward.
• Stop and mute are always reachable. Camera and search tools are off until explicitly enabled and visibly active.
• Show whether a room is connected but waiting for its worker, and whether the model is starting, listening, thinking, speaking, awaiting confirmation, or running a tool.
• A pinned frame is visibly distinct from the live feed and survives only for its declared task/session lifetime. Camera failure falls back to an explicit voice-only state.
• Show degraded audio mode, relevant latency, and late-task state. Never use a cheerful acknowledgement or haptic as proof that a tool completed.
• Treat the HF Space's “color belongs to the voice” and mono-machine-label pattern as a useful local direction, not a global design-system mandate. Compile the accepted visual direction into project tokens and Design System constraints.

Promotion Gate

Promote a route only when:

• exact source/package/model/provider identities are reproducible;
• all deterministic protocol/state tests pass;
• representative quality and p95 latency meet the declared product SLO;
• interruption, tool approval, auth, rate limiting, retention/deletion, and degraded states are proven;
• a confirmation actor cannot approve its own request, and deny/timeout/disconnect paths leave no action behind;
• session identity, media-source grants, per-user tool/vault boundaries, deferred-job durability, and late-result single delivery are proven;
• cost and capacity fit the expected session profile;
• the interface passes keyboard, mobile, silent, reduced-motion, error, and recovery review;
• a fallback to text or the incumbent voice route exists.

A source-authored demo, star count, qualitative “natural” claim, or one median latency is never sufficient.

Run Contract

This workflow follows Workflow Run Contract. Preserve raw traces and media under their authority policy, publish only allowed aggregate evidence, and write durable route changes back to the tool, design, project, and graph owners. A blocked provider, model license, test corpus, privacy review, or hardware requirement remains explicit rather than becoming a no-op approval.

---

Timeline

• 2026-09-11 | Added GPT-Live-1 as a provisional duplex-layer signal and retained the backend reasoner/tool boundary plus interruption and expression controls as the stable product contract. Source: X 2098099269551149398; local media review

• 2026-08-12 | Added Pipecat as the retained modular Python/provider/transport/multi-agent orchestration route from pinned commit 2a107a07; preserved the same component identity, consent, latency, interruption, authority, privacy, provider, cost, and failure proofs rather than treating its broad integration catalog as a production verdict. Source: X 2077743623781965895; pinned Pipecat repository evidence

• 2026-08-11 | Replayed the GPT-Realtime startup-ideas source and compiled it into a five-layer feasibility classifier, five opportunity lanes, and the current OpenAI route. Kept the services-as-software signal; rejected unsupported silent-mode, cost, duration, autonomy, and historical-impossibility claims; made regulated, monetary, legal, medical, insurance, housing, minor-safety, and product-safety actions explicitly human-owned. Source: X 2061129813750915508; .brain/artifacts/x/2061129813750915508/gpt-realtime-startup-ideas-review/source-audit.md

• 2026-08-11 | Replayed current VisionClaw and expanded the workflow for wearable/multimodal systems: separate perception and action loops, processor maps, current-versus-pinned visual reference, scoped room identity, non-self-approvable spoken confirmation, durable late work, context budgets, liveness/degraded states, and platform-specific proof. Source: .brain/artifacts/x/2037892504683716650/visionclaw-review/source-audit.md

• 2026-08-11 | Created from a full replay of HF Realtime Voice and the current Hugging Face backend/Space. Added route selection, frozen component identity, stage timing, interruption/quality/security/privacy/economics evaluation, sensory state design, and release-versus-main proof requirements. Source: X 2072825424800006350; capture manifest