imara learning library ← all courses
Voice AI · companion course · 5 modules

End-to-end speech models, and what it would take to make one ours

The Voice AI course dismissed speech-to-speech models in a line — little control, almost no African coverage, wrong tool today. This one earns that verdict: how these models actually work, how they're tuned, and a clear-eyed, step-by-step look at exactly what it would take to make one work for our languages and use cases.

2
architecture families: audio-native & Thinker-Talker
~12Hz
audio-token rate that makes it feasible
~19
speech languages in the best open model — few African
7
steps in the roadmap to make one ours
Module 01the other architecture

What end-to-end speech models are

The Voice AI course built a cascaded agent: separate STT, LLM, and TTS models chained together. An end-to-end (speech-to-speech) model collapses all three into one neural network — audio goes in, audio comes out, with no separate transcription step in the middle. One model that hears and speaks.

Why they exist

Chaining three models loses things between the links. A single model that works in audio can, in principle, be:

See the difference — toggle between the two architectures:

Cascaded vs end-to-end
Cascaded: three swappable models, text you can inspect at every step. End-to-end: one model, audio-to-audio, nothing to inspect or swap in the middle.

The two families you'll meet

 Audio-native / full-duplexThinker-Talker (separated)
ExamplesMoshi, Spirit LM, Hertz-DevQwen-Omni family, and kin
HowOne model thinks directly in audio tokensA text "Thinker" reasons; a "Talker" renders speech
StrengthMaximum naturalness, real duplex, lowest latencyKeeps text-LLM powers: tools, long context, retrieval
WeaknessHardest to control or inspectSlightly less "native", still heavy
The one that matters for us

For anything regulated, the Thinker-Talker split is the important development: because a text model does the reasoning, you keep function calling, retrieval, and an inspectable text trace — most of what makes end-to-end un-governable is clawed back. If we ever go end-to-end, this is the family to start from.

Module 02under the hood

How they actually work

The whole thing rests on one idea: turn continuous sound into discrete tokens, so a language model can treat speech exactly like it treats text. Get that, and everything else follows.

The neural audio codec — the key enabler

An LLM predicts tokens from a vocabulary. Audio is a continuous waveform — no tokens in sight. A neural audio codec (Mimi, EnCodec, SNAC, and others) solves this: it compresses audio into a small stream of discrete tokens, and reconstructs audio from them. The critical trick is a low frame rate — Mimi emits tokens at ~12.5 per second — slow enough that an LLM can read and write them in real time. Explore the trade-off:

Audio tokenization · waveform → discrete tokens
Higher rate = better audio quality but more tokens for the LLM to handle (slower, pricier). Codec designers fight to keep the rate low while sounding good — that balance is what makes real-time end-to-end possible.

The model is an LLM over interleaved tokens

Once audio is tokens, the model is a familiar transformer predicting the next token — except its vocabulary now mixes text tokens and audio tokens. Speech in becomes audio tokens; the model generates audio tokens back; a vocoder turns those into a waveform you hear. Many models also predict text alongside the audio — an "inner monologue" that keeps the speech coherent and gives you a transcript for free.

the token stream, conceptually
caller audio ─▶ [ codec encoder ] ─▶ audio tokens in
                                         │
          text + audio tokens interleaved▼   (one LLM)
   [USR audio…] → model → [text: "balance is 500"][audio tokens…]
                                         │
              audio tokens out ─▶ [ vocoder ] ─▶ speech to caller

Semantic vs acoustic, and duplex

Two subtleties worth knowing. Codec tokens must carry both meaning (what was said) and acoustics (how it sounded) — balancing these is an active research problem ("the semantic-acoustic conflict"). And full-duplex models run two token streams at once — one for listening, one for speaking — which is what lets them talk and hear simultaneously, like a real conversation rather than walkie-talkie turns.

Module 03the honest verdict

The trade-offs — why it's not our tool today

The Voice AI course's one-liner was right; here's the full reasoning, so the verdict is yours to check rather than take on faith. Three hard limits, and the conditions under which each lifts.

1. Little control

When one model does everything in audio, you lose the inspection points a cascaded stack gives you. You can't read the transcript and correct it, can't swap a weak component, can't easily inject a tool result or a retrieved fact, and guardrails are harder because there's no clean text to filter. For a lender, "the model said something wrong on a call and we can't see why" is a serious problem.

The partial escape

The Thinker-Talker split (Module 1) recovers much of this: the text Thinker gives you an inspectable trace, tool/function calling, and retrieval. The newest open models in this family added function calling — a real step toward governable end-to-end. Audio-native models give you the least control of all.

2. Almost no African-language coverage

This is the decisive one for us. The best open end-to-end models support on the order of ~19 speech languages — overwhelmingly English, Chinese, and major European/Asian languages. African languages are essentially absent from speech output. And the gap is harder to close than for the cascaded stack, because of the data problem below.

3. The data requirement is brutal

To teach a cascaded stack Swahili you fine-tune an ASR model (transcribed audio) and a TTS model (audio + text) separately — both have real datasets. To teach an end-to-end model, you ideally need paired conversational speech — actual spoken dialogue in the target language — which barely exists for African languages. You're not fine-tuning one component; you're adapting a model that learned to converse in audio, in a language it never heard.

Plus: cost, hardware, safety

The strongest open models are large (tens of billions of parameters) and not yet comfortably self-hostable; the self-hostable ones (Moshi, ~7B) are less capable and less controllable. Hallucination and safety are harder to bound without a text checkpoint. All real, all surmountable with effort — but effort measured in quarters, not weeks.

When it becomes our tool

End-to-end earns a serious look when: (a) conversational naturalness — real interruptions, emotion, instant response — becomes a competitive need our cascaded stack can't meet; and (b) open models add credible African speech coverage, or we've built the Swahili speech corpus ourselves. Note (b) is the same corpus the cascaded stack wants anyway — which is the thread Module 5 pulls.

Module 04adapting one

How end-to-end models are tuned

Adapting a speech model to a new language or domain follows a well-established recipe — the same two-stage shape that works for text LLMs, plus a codec wrinkle. Knowing it makes Module 5's roadmap concrete.

The two-stage recipe

continued pretraining → supervised fine-tuning
# Stage 1 — CONTINUED PRE-TRAINING (CPT): learn the language's SOUND
#   feed lots of UNLABELED target-language audio (cheaper to collect)
#   the model adapts its audio representations to Swahili acoustics

# Stage 2 — SUPERVISED FINE-TUNING (SFT): learn the TASK
#   feed PAIRED data: speech + text, ideally spoken dialogues
#   upsample the low-resource data ~5x so each batch sees enough
#   MIX IN original data to avoid catastrophic forgetting

# optional: LoRA/adapters instead of full fine-tune (cheaper)
# optional: preference tuning to polish naturalness

Research on low-resource speech is consistent on this: continued pretraining on target-language audio is the single most effective step, and it needs only unlabeled audio, which is far easier to gather than transcribed or paired data. Supervised fine-tuning then teaches the actual task on the smaller paired set.

Three levers that stretch scarce data

The codec question

One wrinkle unique to speech models: the audio codec (Module 2) was trained on certain languages' sounds. If the target language has sounds it never saw, the codec itself may represent them poorly, capping quality no matter how well you tune the LLM. For Swahili (Bantu, relatively standard phonetics) existing codecs usually cope; for tonal or phonetically unusual languages, you may need to retrain or adapt the codec too — a significant extra step.

What each data tier unlocks

You have…Realistically unlocks
a few hoursVoice/style adaptation (a specific speaker) — not a new language
tens of hoursLoRA domain/accent adaptation; CPT bootstrapping begins to help
hundreds of hoursViable language adaptation via CPT + SFT
thousands of hoursStrong, production-grade coverage of the language
Module 05the roadmap

Step by step: what it would take for us

Suppose we decided to build an end-to-end Swahili voice agent anyway. Here is the honest, concrete path — not to talk you into it, but so the decision is made with the full cost in view. First, gauge where we'd stand on the thing that matters most — data:

Readiness · hours of paired Swahili conversational speech
Our own consented, transcribed call recordings are the best source — they match our accents, our code-switching, and our domain. Collecting them is the long pole, and it's a no-regret investment (see the end).

The seven steps

Decide why — honestly

End-to-end only clears the bar if we need naturalness/duplex/latency the cascaded stack can't give and can accept weaker control. For a regulated lender, that bar is high. If a cascaded Swahili agent is "good enough," stop here — it's the right answer today.

Pick an open base model

Start from the Thinker-Talker family (keeps function calling, retrieval, an inspectable text trace). Choose audio-native (Moshi) only if maximum naturalness and a ready serving stack outweigh governability.

Build the data corpus — the long pole

Collect Swahili and code-switched speech in three tiers: lots of unlabeled audio for continued pretraining, paired speech-text for fine-tuning, and ideally real spoken dialogues in our domain. This is a multi-month, consented data programme — and the single biggest determinant of success.

Check (and maybe adapt) the codec

Verify the model's audio tokenizer represents Swahili sounds well. Usually fine for Swahili; budget extra work if we extend to tonal languages whose sounds the codec never learned.

Adapt the model: CPT → SFT

Continued pretraining on the unlabeled Swahili audio (learn the sound), then supervised fine-tuning on the paired/dialogue data (learn the task) — upsampling our data and mixing in the original to avoid forgetting. LoRA to keep it affordable.

Restore control & safety

Re-wire tool/function calling, retrieval grounding, and guardrails around the model — straightforward with Thinker-Talker, genuinely hard with audio-native. Add a human-escalation path. This is where end-to-end fights back hardest.

Evaluate, then deploy on-prem

Test on real Swahili and code-switched calls — comprehension, latency, hallucination, safety. Then size the hardware: these models are heavier than the cascaded trio (a 7B+ model plus codec and vocoder), so plan GPU and concurrency accordingly.

The verdict, and the no-regret move

This is a research-grade, multi-quarter undertaking needing a speech-ML team and a serious data programme — which is why the cascaded stack remains the right tool for us today. But notice step 3: the Swahili speech corpus you'd build for an end-to-end model is the same corpus that sharpens the cascaded stack's ASR and TTS. So the one expensive, slow, high-value investment pays off either way. Build the data now, ship on cascaded now, and keep end-to-end as a live option for when the models — or our naturalness needs — catch up.

You've seen both roads

From what end-to-end speech models are and the codec trick that makes them possible, through their honest limits for our context, the two-stage recipe for tuning them, and the seven concrete steps it would take to make one ours — you can now weigh speech-to-speech against the cascaded stack with clear eyes, and you know the one investment that serves both.