The Voice AI course dismissed speech-to-speech models in a line — little control, almost no African coverage, wrong tool today. This one earns that verdict: how these models actually work, how they're tuned, and a clear-eyed, step-by-step look at exactly what it would take to make one work for our languages and use cases.
The Voice AI course built a cascaded agent: separate STT, LLM, and TTS models chained together. An end-to-end (speech-to-speech) model collapses all three into one neural network — audio goes in, audio comes out, with no separate transcription step in the middle. One model that hears and speaks.
Chaining three models loses things between the links. A single model that works in audio can, in principle, be:
See the difference — toggle between the two architectures:
| Audio-native / full-duplex | Thinker-Talker (separated) | |
|---|---|---|
| Examples | Moshi, Spirit LM, Hertz-Dev | Qwen-Omni family, and kin |
| How | One model thinks directly in audio tokens | A text "Thinker" reasons; a "Talker" renders speech |
| Strength | Maximum naturalness, real duplex, lowest latency | Keeps text-LLM powers: tools, long context, retrieval |
| Weakness | Hardest to control or inspect | Slightly less "native", still heavy |
For anything regulated, the Thinker-Talker split is the important development: because a text model does the reasoning, you keep function calling, retrieval, and an inspectable text trace — most of what makes end-to-end un-governable is clawed back. If we ever go end-to-end, this is the family to start from.
The whole thing rests on one idea: turn continuous sound into discrete tokens, so a language model can treat speech exactly like it treats text. Get that, and everything else follows.
An LLM predicts tokens from a vocabulary. Audio is a continuous waveform — no tokens in sight. A neural audio codec (Mimi, EnCodec, SNAC, and others) solves this: it compresses audio into a small stream of discrete tokens, and reconstructs audio from them. The critical trick is a low frame rate — Mimi emits tokens at ~12.5 per second — slow enough that an LLM can read and write them in real time. Explore the trade-off:
Once audio is tokens, the model is a familiar transformer predicting the next token — except its vocabulary now mixes text tokens and audio tokens. Speech in becomes audio tokens; the model generates audio tokens back; a vocoder turns those into a waveform you hear. Many models also predict text alongside the audio — an "inner monologue" that keeps the speech coherent and gives you a transcript for free.
caller audio ─▶ [ codec encoder ] ─▶ audio tokens in
│
text + audio tokens interleaved▼ (one LLM)
[USR audio…] → model → [text: "balance is 500"][audio tokens…]
│
audio tokens out ─▶ [ vocoder ] ─▶ speech to callerTwo subtleties worth knowing. Codec tokens must carry both meaning (what was said) and acoustics (how it sounded) — balancing these is an active research problem ("the semantic-acoustic conflict"). And full-duplex models run two token streams at once — one for listening, one for speaking — which is what lets them talk and hear simultaneously, like a real conversation rather than walkie-talkie turns.
The Voice AI course's one-liner was right; here's the full reasoning, so the verdict is yours to check rather than take on faith. Three hard limits, and the conditions under which each lifts.
When one model does everything in audio, you lose the inspection points a cascaded stack gives you. You can't read the transcript and correct it, can't swap a weak component, can't easily inject a tool result or a retrieved fact, and guardrails are harder because there's no clean text to filter. For a lender, "the model said something wrong on a call and we can't see why" is a serious problem.
The Thinker-Talker split (Module 1) recovers much of this: the text Thinker gives you an inspectable trace, tool/function calling, and retrieval. The newest open models in this family added function calling — a real step toward governable end-to-end. Audio-native models give you the least control of all.
This is the decisive one for us. The best open end-to-end models support on the order of ~19 speech languages — overwhelmingly English, Chinese, and major European/Asian languages. African languages are essentially absent from speech output. And the gap is harder to close than for the cascaded stack, because of the data problem below.
To teach a cascaded stack Swahili you fine-tune an ASR model (transcribed audio) and a TTS model (audio + text) separately — both have real datasets. To teach an end-to-end model, you ideally need paired conversational speech — actual spoken dialogue in the target language — which barely exists for African languages. You're not fine-tuning one component; you're adapting a model that learned to converse in audio, in a language it never heard.
The strongest open models are large (tens of billions of parameters) and not yet comfortably self-hostable; the self-hostable ones (Moshi, ~7B) are less capable and less controllable. Hallucination and safety are harder to bound without a text checkpoint. All real, all surmountable with effort — but effort measured in quarters, not weeks.
End-to-end earns a serious look when: (a) conversational naturalness — real interruptions, emotion, instant response — becomes a competitive need our cascaded stack can't meet; and (b) open models add credible African speech coverage, or we've built the Swahili speech corpus ourselves. Note (b) is the same corpus the cascaded stack wants anyway — which is the thread Module 5 pulls.
Adapting a speech model to a new language or domain follows a well-established recipe — the same two-stage shape that works for text LLMs, plus a codec wrinkle. Knowing it makes Module 5's roadmap concrete.
# Stage 1 — CONTINUED PRE-TRAINING (CPT): learn the language's SOUND
# feed lots of UNLABELED target-language audio (cheaper to collect)
# the model adapts its audio representations to Swahili acoustics
# Stage 2 — SUPERVISED FINE-TUNING (SFT): learn the TASK
# feed PAIRED data: speech + text, ideally spoken dialogues
# upsample the low-resource data ~5x so each batch sees enough
# MIX IN original data to avoid catastrophic forgetting
# optional: LoRA/adapters instead of full fine-tune (cheaper)
# optional: preference tuning to polish naturalnessResearch on low-resource speech is consistent on this: continued pretraining on target-language audio is the single most effective step, and it needs only unlabeled audio, which is far easier to gather than transcribed or paired data. Supervised fine-tuning then teaches the actual task on the smaller paired set.
One wrinkle unique to speech models: the audio codec (Module 2) was trained on certain languages' sounds. If the target language has sounds it never saw, the codec itself may represent them poorly, capping quality no matter how well you tune the LLM. For Swahili (Bantu, relatively standard phonetics) existing codecs usually cope; for tonal or phonetically unusual languages, you may need to retrain or adapt the codec too — a significant extra step.
| You have… | Realistically unlocks |
|---|---|
| a few hours | Voice/style adaptation (a specific speaker) — not a new language |
| tens of hours | LoRA domain/accent adaptation; CPT bootstrapping begins to help |
| hundreds of hours | Viable language adaptation via CPT + SFT |
| thousands of hours | Strong, production-grade coverage of the language |
Suppose we decided to build an end-to-end Swahili voice agent anyway. Here is the honest, concrete path — not to talk you into it, but so the decision is made with the full cost in view. First, gauge where we'd stand on the thing that matters most — data:
End-to-end only clears the bar if we need naturalness/duplex/latency the cascaded stack can't give and can accept weaker control. For a regulated lender, that bar is high. If a cascaded Swahili agent is "good enough," stop here — it's the right answer today.
Start from the Thinker-Talker family (keeps function calling, retrieval, an inspectable text trace). Choose audio-native (Moshi) only if maximum naturalness and a ready serving stack outweigh governability.
Collect Swahili and code-switched speech in three tiers: lots of unlabeled audio for continued pretraining, paired speech-text for fine-tuning, and ideally real spoken dialogues in our domain. This is a multi-month, consented data programme — and the single biggest determinant of success.
Verify the model's audio tokenizer represents Swahili sounds well. Usually fine for Swahili; budget extra work if we extend to tonal languages whose sounds the codec never learned.
Continued pretraining on the unlabeled Swahili audio (learn the sound), then supervised fine-tuning on the paired/dialogue data (learn the task) — upsampling our data and mixing in the original to avoid forgetting. LoRA to keep it affordable.
Re-wire tool/function calling, retrieval grounding, and guardrails around the model — straightforward with Thinker-Talker, genuinely hard with audio-native. Add a human-escalation path. This is where end-to-end fights back hardest.
Test on real Swahili and code-switched calls — comprehension, latency, hallucination, safety. Then size the hardware: these models are heavier than the cascaded trio (a 7B+ model plus codec and vocoder), so plan GPU and concurrency accordingly.
This is a research-grade, multi-quarter undertaking needing a speech-ML team and a serious data programme — which is why the cascaded stack remains the right tool for us today. But notice step 3: the Swahili speech corpus you'd build for an end-to-end model is the same corpus that sharpens the cascaded stack's ASR and TTS. So the one expensive, slow, high-value investment pays off either way. Build the data now, ship on cascaded now, and keep end-to-end as a live option for when the models — or our naturalness needs — catch up.
From what end-to-end speech models are and the codec trick that makes them possible, through their honest limits for our context, the two-stage recipe for tuning them, and the seven concrete steps it would take to make one ours — you can now weigh speech-to-speech against the cascaded stack with clear eyes, and you know the one investment that serves both.