How a machine holds a real-time spoken conversation — and how to make it work in Swahili and other African languages, run it on your own hardware, and solve the problems that make or break it: turn-taking, latency, interruptions, and code-switching. Grounded in the current open-source stack (Chatterbox, Whisper, MMS).
Text chat is forgiving: the user waits, nothing is real-time, mistakes can be edited. Voice is the opposite — it happens live, both parties expect human-paced turns, and a one-second delay feels broken. A voice agent is a tight real-time loop of specialised models, and the whole craft is making that loop feel natural.
The classic ("cascaded") voice agent chains four components, with a fifth — the orchestrator — conducting them:
caller speaks
|
v
[ VAD + turn detection ] # is the caller talking? have they finished?
|
v
[ STT / ASR ] # speech -> text
|
v
[ LLM ] # text -> a reply (+ tools, memory)
|
v
[ TTS ] # reply -> speech
|
v
caller hears the answer # target: under ~800 ms, end to end| Cascaded (STT → LLM → TTS) | Speech-to-speech (end-to-end) | |
|---|---|---|
| How | Separate models, chained | One model, audio in → audio out |
| Control | Full — swap any piece, see the text | Less — it's a black box |
| African languages | Tune each piece for Swahili etc. | Hard — few support them |
| On-prem | Flexible, mix open models | Limited open options |
For African languages on your own hardware, the cascaded approach wins decisively: you can pick a Swahili-tuned ASR, any LLM, and a Chatterbox voice independently, swap weak parts, and inspect the text at every step. End-to-end speech models are impressive but give little control and almost no African-language coverage — the wrong tool for this job today.
Gluing those models together is harder than it sounds, and it's where frameworks earn their place. The orchestrator streams audio and text between stages (never waiting for a whole stage to finish), manages conversation state, handles barge-in when the caller interrupts, calls tools (look up an account, log a payment), and connects to the phone network. Everything in Modules 3 and 6 lives here.
Every box in the pipeline has strong open-source options you can self-host. Knowing the real choices — and their sizes — is what lets you plan the hardware in Module 5.
The LLM is any model you can self-host — for on-prem, a quantised small model (7–13B at INT4) is the sweet spot. For TTS, the key family for this course is Chatterbox (Resemble AI, MIT-licensed, open weights):
| Model | Size | Languages | Best for |
|---|---|---|---|
| Chatterbox-Turbo | 350M | English | Lowest-latency English agents |
| Chatterbox | 500M | English | Expressive control |
| Chatterbox Multilingual V3 | 0.5B | 23+ incl. Swahili | African-language voices |
Chatterbox clones a voice from a ~10-second reference clip (zero-shot), does cross-language voice transfer, and is small enough to run almost anywhere. Other open TTS worth knowing: Kokoro and Piper (tiny, fast) and XTTS (expressive, heavier).
You don't build turn-taking from scratch — you use proven pieces: Silero VAD for speech detection, and a semantic turn detector (Pipecat's open Smart Turn, which covers 23 languages, or LiveKit's transformer turn detector). The two frameworks that assemble everything:
# conceptual Pipecat-style pipeline — swap in your own models
pipeline = [
transport.input(), # audio from the phone call
silero_vad, # speech vs silence
smart_turn, # has the caller finished?
faster_whisper_stt, # STT (Swahili-tuned)
llm, # your self-hosted LLM
chatterbox_tts, # TTS (Swahili voice)
transport.output(), # audio back to the caller
]The models are the easy part; making the conversation feel human is the hard part. Three problems dominate, and each has a known solution in the modern stack.
Humans expect a reply within roughly 300–500 ms. Past about 800 ms a voice agent feels sluggish; past ~1.2 s people start talking over it. That total is a budget split across every stage, and each stage has to stay small:
| Stage | Healthy budget |
|---|---|
| Endpointing / turn detection | ~80 ms |
| STT (streaming) | ~120 ms |
| LLM (to first token) | ~180 ms |
| TTS (to first audio) | ~70 ms |
| Total (p50) | ~450 ms |
How it's solved: stream everything. Don't wait for the caller to finish before starting STT; don't wait for the whole LLM reply before starting TTS — synthesise the first sentence while the rest generates (chunked TTS). Optimise time-to-first-token, not total generation. Co-locate the models (Module 5) so no stage waits on a network hop. Build the budget below and watch it cross the line:
The deceptively hard problem. Naive systems wait for a fixed silence (say 700 ms) and then respond — but humans pause mid-thought all the time ("I want to pay… my loan"). Wait too little and you cut them off; wait too long and the agent feels slow.
How it's solved: raw silence (VAD) is combined with a semantic turn detector — a small model that judges whether the thought is complete, not just whether there's silence. It can respond fast after a finished sentence and wait patiently through a mid-sentence pause. Drag the silence threshold and see the trade-off a fixed timer can't escape:
When the caller talks over the agent, the agent must stop instantly and listen — anything else feels robotic. How it's solved: when VAD fires on the caller's audio while the agent is speaking, the runtime cancels TTS, rolls back the interrupted reply, and processes the new input. The subtlety: not every sound is an interruption. People say "mm-hmm", "yeah", "sawa" while listening — backchannels, not interruptions. Modern handlers check the first few hundred milliseconds and reject roughly half of would-be interruptions as backchannels, so the agent isn't constantly stopping itself.
Teams hand-tune interruption and turn-taking heuristics for weeks. Don't — use the framework's turn detector and adaptive interruption handling, and only tune the thresholds. And never stack a fixed wait-timer on top of a semantic turn model: it just adds latency and overrides the smarter signal.
This is where generic voice AI falls over and where the real work is. Most models are trained English-first; your callers speak Swahili, often mixed with English, and sometimes Luganda, Yoruba, Hausa, or Amharic. Here's the honest state of each piece and how to close the gap.
The models are hungry for transcribed audio, and for most African languages that data barely exists. Swahili is mid-resource (over 100M speakers, some datasets); most others are genuinely low-resource. Many are also tonal or morphologically rich, which makes them harder still. Everything below is about squeezing good results from little data.
| Whisper | MMS (Meta) | |
|---|---|---|
| Coverage | ~99 languages; Swahili decent zero-shot | 1,000+ incl. many African |
| Fine-tune cost | ~15–20 hrs audio for a usable result | Tiny adapters — far less data |
| Low-resource winner | Good for mid-resource (Swahili) | Usually wins when data is scarce |
The practical rule from recent benchmarks: for Swahili, fine-tuned Whisper large-v3 is a strong, simple choice. For a truly low-resource language (a Kinyarwanda study is a good example), fine-tuning an MMS adapter — updating a tiny fraction of the model — beat a Whisper fine-tune using the same small dataset. Start with what fits your data volume, not the famous name.
Mozilla Common Voice, Google's FLEURS, and African-specific efforts like Kencorpus and Microsoft's Paza set (Swahili, Dholuo, Kalenjin, Kikuyu, Maasai, Somali) are real starting points. But the highest-value corpus is your own call recordings, consented and transcribed — they match your accents, your domain vocabulary, and your audio conditions exactly. Collecting and labelling that is the single best investment for African-language voice.
Your callers won't speak pure Swahili — they'll say "Nataka kulipa loan yangu lakini sina pesa till month-end." A monolingual model garbles every switched word. How it's solved: use a multilingual ASR that expects code-switching natively (modern streaming models advertise exactly this), and fine-tune on code-switched data so the model learns your real speech, not textbook Swahili. Toggle the model type below:
Good news first: Chatterbox Multilingual already supports Swahili out of the box. You can clone a warm, local-sounding voice from a ~10-second reference clip with no training at all — the fastest path to a Swahili agent voice:
from chatterbox.mtl_tts import ChatterboxMultilingualTTS
model = ChatterboxMultilingualTTS.from_pretrained(device="cuda")
wav = model.generate(
"Habari! Umefanikiwa kulipa mkopo wako. Asante sana.",
language_id="sw", # Swahili
audio_prompt_path="agent_voice_10s.wav" # clone this voice
)When you need more — a specific brand voice, a regional accent, or a language not in the supported 23 (Yoruba, Hausa, Zulu…) — you fine-tune with LoRA: a few hours of clean, transcribed audio in the target voice/language updates a small fraction of the weights, cheaply.
# 1. collect data: clean audio clips + a metadata.csv of
# text, audio_path, language_id (add a new language id if needed)
# 2. LoRA fine-tune the multilingual checkpoint on it
# (small, fast — updates ~a few % of parameters)
# 3. load base model + your LoRA adapter at inference
model.load_lora("adapters/swahili-coast-voice")
wav = model.generate(text, language_id="sw")Whether it's Whisper, an MMS adapter, or Chatterbox, the recipe is the same: start from a big multilingual model, then adapt it to your language and data with a small, cheap fine-tune. You are never training from scratch — you're nudging a capable model toward African speech.
The question you actually need answered: can this run on hardware you own, and how much do you need? The short answer is yes — a full pilot fits on a single GPU — and here's the reasoning so you can size it properly.
The whole feasibility question comes down to GPU memory. With modern optimised runtimes, the numbers are smaller than people expect:
| Stage | Model | VRAM |
|---|---|---|
| STT | Whisper large-v3 (faster-whisper) | ~3–4 GB (FP16) · ~2–3 GB (INT8) |
| LLM | 7–13B, quantised INT4 | ~5–8 GB |
| TTS | Chatterbox 0.5B | ~1–2 GB |
| VAD + turn | Silero + Smart Turn | negligible |
The LLM dominates memory, and quantisation (INT4/INT8) shrinks it dramatically for a small quality cost. A full small pipeline — Whisper + a quantised 7–8B LLM + Chatterbox — fits in roughly 10 GB, leaving headroom for several concurrent calls on a single 24 GB GPU.
A single RTX 4090 (24 GB) — or an L4/L40S in a server — runs the full small pipeline with room for a handful of concurrent calls. CPU-only is viable for STT and a small LLM but latency and concurrency suffer, so GPU is strongly recommended for anything beyond a demo. Then you scale by GPU. Size the hardware below for your expected peak concurrent calls:
Everything comes together as a system that answers and places real phone calls. The orchestrator is the conductor; around it sit telephony, the models, your business logic, and the operational scaffolding.
phone network (PSTN)
| SIP
v
[ telephony / media ] # Asterisk / FreeSWITCH, or LiveKit SIP
| audio stream
v
[ ORCHESTRATOR ] # Pipecat / LiveKit Agents
| VAD + turn detection # barge-in, backchannels
| STT -> LLM -> TTS # streaming, co-located (Module 5)
| tools --------------> [ your systems: LMS, CRM, payments ]
| state + session
|
v
monitoring + logging # per-turn latency, transcripts, outcomesStand up a single-GPU pilot (Module 5) with LiveKit Agents (telephony included) or Pipecat, wire in faster-whisper, a quantised LLM, and a Swahili Chatterbox voice, and connect one phone number. Run a small outbound campaign to real customers, measure latency and comprehension, collect the recordings — and use those recordings as the fine-tuning data that makes version two markedly better. The system improves itself once it's live.
From the real-time pipeline and its models, through the hard problems of turns, delays, interruptions and code-switching and how each is solved, to tuning for Swahili and other African languages, fitting it all on a single on-prem GPU, and wiring it to the phone network as an orchestrator — you have the end-to-end picture of a voice AI system built for your callers, your languages, and your hardware.