Hugging Face Speech to Speech

A strong modular voice-agent reference and conditional project dependency: VAD → STT → LLM → TTS behind an OpenAI Realtime-compatible server, with explicit self-hosted and local component choices.

Watch the 104.7-second source demo on X. The reviewed local capture shows a real multi-turn voice session with visible transcripts, camera use, and a web-search tool result. It does not by itself prove latency percentiles, recognition quality, interruption behavior, availability, concurrency, or cost. Source: local X video artifact review, 2026-08-11

Routing Summary

Use Hugging Face Speech to Speech when a Python or device/backend project needs a replaceable VAD/STT/LLM/TTS cascade, local or self-hosted model choices, and an OpenAI Realtime-compatible WebSocket/WebRTC boundary. Follow Realtime Voice Agent Workflow rather than installing from a tweet.

Choose another route when the job shape differs:

No global install was made. A project pilot should use an isolated environment, exact package or commit, exact model revisions, and a generated lock/receipt.

What It Actually Is

The backend runs four swappable stages, each in a thread connected by queues:

Current supported choices include Parakeet and Whisper-family STT, remote OpenAI-compatible or local Transformers/MLX LLMs, and Qwen3-TTS, Kokoro, Pocket TTS, ChatTTS, or MMS TTS. The protocol surface supports core OpenAI Realtime events, streaming transcripts/audio, tool calls, response cancellation, WebSocket, and WebRTC. Source: huggingface/speech-to-speech current README/source at 0071d7d, 2026-08-11

Version Contract

The project currently has two materially different authorities:

Do not use current-main documentation to describe the published wheel. Choose a released package and its tag docs, or build a frozen source commit and treat the resulting wheel as a separate artifact. Source: current repository/tag/PyPI comparison, 2026-08-11

Claim Audit

The original signal usefully updates the prior that open-component realtime voice is practical, but its wording needs boundaries:

• “Fully open-source.” The backend code is Apache-2.0; current Parakeet is CC BY 4.0, Qwen3-TTS is Apache-2.0, and Gemma 4 is Apache-2.0. The current demo Space repository, however, declares no license. Public source is not automatic reuse permission.
• “Fully local.” Possible with explicit local STT, LLM, and TTS choices. The current default LLM is remote gpt-5.4-mini, so the default install is neither fully local nor a fully open inference stack.
• “Low latency / dramatically natural.” Plausible and visibly responsive in the saved demo, but neither the blog nor Space publishes a reproducible end-to-end p50/p95 matrix. The Space README's roughly 150–300 ms WebSocket figure is an undocumented assertion, not an acceptance receipt.
• “Production.” Upstream reports use behind thousands of Reachy Mini robots, and current CI is healthy. That does not substitute for the target project's traffic, network, accents/languages, devices, tools, privacy, cost, and error-budget tests.

Security And Privacy Boundary

The core server binds to 127.0.0.1 by default, which is a good local default. It provides no authentication or throttling of its own, and the optional remote-LLM proxy also lacks built-in access control. The hosted Space adds HF OAuth, an allocator, signed session tokens, time limits, and server-side token handling; those controls are not part of the base package. Source: current backend and Space source, 2026-08-11

A networked deployment needs:

• TLS/WSS, authenticated session creation, short-lived scoped tokens, origin and network policy, rate/concurrency limits, and abuse controls;
• exact provider/model endpoints and secret custody, with the LLM proxy disabled unless a gateway owns it;
• explicit consent and product indicators for microphone, recording, camera, transcript, search, and tool use;
• retention/deletion rules for raw audio, replay buffers, transcripts, prompts, tool results, traces, and provider logs;
• a disclosure of every external processor and whether a “local” mode actually prevents egress;
• prompt/tool boundaries so speech or visual input cannot silently authorize side effects.

Pilot And Acceptance Receipt

Follow Realtime Voice Agent Workflow. At minimum freeze:

transport + client + VAD + STT revision + LLM/provider revision + TTS/voice revision + hardware + region + network + prompt/tools + concurrency

Then measure:

• end-of-speech → final transcript;
• final transcript → first model token;
• first token → first audible audio;
• end-of-speech → first audible audio at p50/p95/p99;
• interruption detection → playback clear and server cancellation;
• WER or task accuracy across target accents, languages, noise, devices, and turn lengths;
• false starts/stops, dropped audio, echo, overlapping speech, empty transcripts, and tool-call correctness;
• session success/availability, concurrent-session degradation, provider failures/fallbacks, and cost per minute.

The source compiles and builds locally, both generated distributions pass strict metadata checks, and current/release GitHub checks pass. No local model session or benchmark was run during this review, so admission remains project-scoped and evidence-gated. Source: capture manifest, 2026-08-11

Interface Signal

The current Space's DESIGN.md contains a strong bounded pattern for voice products: the speaking/listening object owns saturated color; tiny transcript role echoes may reuse that state; actual audio levels drive the hero motion; machine identifiers use mono while human prose does not; and one state machine drives orb, transcript, controls, stop/recovery, and tool states. The literal orb and palette are not Kevin's new house style. The transferable rule is a coherent sensory-state mapping with silent, keyboard, 360 px, 44 px-target, reduced-motion, mute, and transcript fallbacks. Source: Space DESIGN.md@03a43b2, 2026-08-11; Design System

---

Timeline

• 2026-08-11 | Force-replayed the complete X source, demo video, blog, current Space, current backend, release/PyPI identity, model licenses, source tests, CI checks, package build, and UI design contract. Registered a guarded modular voice capability and routed it through a measurable workflow; no global install. Source: X 2072825424800006350; capture manifest