Technical July 23, 2026

The Stack Behind Clara: Twilio, Groq, and the Race for Sub-1s Latency

When someone calls your business and gets an AI, they're already annoyed. Every second of silence makes it worse. Cross 2 seconds and they hang up. Cross 3 and they think the line is dead.

So the entire stack — phone, transcription, LLM, text-to-speech — has to happen in under 1 second. Not "around 1 second." Not "usually under 1 second." Under 1 second, consistently, on every turn of the conversation.

Here's what I'm actually running, why I picked each piece, and where it still breaks.

The Pipeline

Clara's call flow is six steps, and every millisecond in that chain matters:

  1. Twilio receives the inbound call and streams audio via WebSocket
  2. Whisper (Groq) transcribes the caller's audio in real-time
  3. Groq llama-3.3-70b generates the response text
  4. Kokoro TTS converts the response to audio
  5. Twilio streams the audio back to the caller
  6. Repeat for each turn

The total budget is ~800ms. That leaves room for network jitter without crossing the 1s threshold the caller notices.

Twilio: The Phone Part

Twilio handles the telephony. Inbound call comes in, Twilio opens a WebSocket to my server, and bidirectional audio flows as base64-encoded mulaw chunks at 8kHz.

I picked Twilio because the WebSocket streaming API is well-documented and the latency from their PSTN bridge to my server is predictable — usually 40-60ms. I'm in Texas, my server is in my house, and Twilio's edge relay adds minimal hop time.

The other option was Vonage (formerly Nexmo). Cheaper per minute, but their real-time audio streaming felt bolted on. Twilio's Media Streams are clearly designed for this use case.

Cost: $1.15/month per phone number, plus ~$0.003/minute for inbound calls. For a small business getting 30-50 calls a day, the phone bill is maybe $5/month. That's not the expensive part.

Whisper on Groq: The Transcription Part

This is where Groq changed everything. Groq runs LPU (Language Processing Unit) chips that are stupidly fast at inference. Whisper-large-v3 on Groq transcribes a 5-second utterance in about 60-80ms. The same model on a standard GPU takes 300-500ms.

I run Whisper-large-v3 because accuracy matters more than model size. A smaller model would be faster but would mistranscribe names, numbers, and anything with background noise. A medical office call where "I need a refill on lisinopril" becomes "I need a refill on vitamins" is a liability, not a convenience.

Streaming transcription is the key. I don't wait for the caller to finish speaking. As audio chunks arrive over the WebSocket, I forward them to Groq and start getting partial transcripts. When the VAD (Voice Activity Detection) detects end-of-speech, I have the final transcript within 80ms.

Cost: Groq's free tier handles development. Production usage is fractions of a cent per transcription. Honestly the cheapest part of the stack.

Groq Llama-3.3-70b: The Brain

The LLM generates the actual response. I use llama-3.3-70b on Groq for the same reason I use Whisper there — speed. Groq's LPU inference on a 70B model gives me first-token latency around 150ms and throughput that completes a typical 2-3 sentence response in another 200-300ms.

I tried smaller models. Llama-3.1-8b is faster (~80ms to first token) but the quality drop is noticeable. It gives shorter, less natural responses. It misses nuance in questions. For a receptionist, that matters — the caller is judging the conversation quality whether they know it's AI or not.

I also tried running everything locally on my RTX 3090. Whisper worked fine. The 70B LLM did not. Even quantized to 4-bit, it took 400-600ms to first token. Add TTS and I was at 1.5-2s total — too slow. Local inference is great for development and testing. For production with real callers, Groq's dedicated hardware is the only way I've found to hit sub-1s.

Kokoro TTS: The Voice Part

I run Kokoro locally on the RTX 3090. It's a lightweight, high-quality TTS model that generates audio in about 80-120ms for a typical sentence. Running it locally means no network round-trip for the TTS step — the text comes off the LLM and goes straight to the GPU.

The alternative was cloud TTS (ElevenLabs, Deepgram, Azure). All three have better voice quality. All three add 100-300ms of network latency. ElevenLabs was the best-sounding but the slowest — 250-400ms per response. For a phone call where the caller is waiting, that's the difference between "responsive" and "annoying delay."

Kokoro's voice is good enough. Not ElevenLabs-good, but clear, natural intonation, no robotic artifacts. On a phone call with typical background noise and 8kHz audio, the difference between Kokoro and a premium cloud TTS is barely perceptible.

The Budget: Where the 800ms Goes

Here's the typical breakdown for a single turn:

That's ~700ms on a good call. Under 800ms most of the time. Under 1s consistently.

The LLM is the biggest chunk. If I could cut that in half, the whole pipeline drops to 500ms — which feels instant on a phone call. But I haven't found a 70B model that runs faster without quality loss, and a smaller model makes the conversation worse.

Where It Still Breaks

This isn't a solved problem. It's a "works most of the time" problem.

Long utterances. If the caller talks for 15+ seconds, Whisper transcription time scales linearly. A 15-second utterance takes ~200ms to transcribe. The total pipeline creeps to 1.2-1.5s. Not terrible, but the caller notices the pause.

LLM response length. If the LLM generates a 5-sentence response, Kokoro needs 300ms+ to synthesize all of it. I cap the response at 3 sentences in the system prompt, but the model doesn't always listen.

Groq rate limits. During peak hours, Groq occasionally throttles. First-token latency jumps from 150ms to 400ms. That pushes the total to 1.3s. I have a fallback to a local smaller model, but the quality drop is obvious.

Background noise. A caller in a car with the window down gives Whisper a harder time. Transcription takes longer, accuracy drops, and the LLM sometimes responds to the wrong question.

What I'd Do Differently

If I were starting today, I'd look hard at Deepgram's Nova-2 for transcription. It's purpose-built for real-time phone audio and might shave 30-40ms off the Whisper step. I'd also prototype with Groq's newer models — if they have a 70B-equivalent with 100ms first-token latency, that alone would drop the total to 600ms.

What I wouldn't change: Twilio for telephony, Kokoro for local TTS, and the overall pipeline structure. The architecture is sound. It's the individual components that will get faster over time.

The honest summary: I hit sub-1s latency about 85% of the time. The other 15% is 1-1.5s, which feels like a slight pause but doesn't kill the conversation. I'm not claiming 500ms or "instant" — I'm claiming "fast enough that callers don't hang up." That's the bar. That's always been the bar.

← Back to blog