Voice Agent Latency Explained: Where the Delay Comes From and What Feels Natural

What latency does a voice agent actually need?

The target comes from how humans take turns. In a cross-cultural study of ten languages, Stivers, Enfield and colleagues found that response gaps cluster around a modal offset of zero to 200 milliseconds in every language sampled, from traditional indigenous communities to major world languages (PNAS, 2009). The range of cultural variation is small: the modes sit between 0 and +200 ms, with an overall mode of 0 ms. People start replies that fast because they predict the end of a turn and plan their response while the other speaker is still talking.

Voice AI cannot do that yet, so it has to catch up. The working rule most practitioners cite is under 800 ms time-to-first-audio at the median and under 1,500 ms at the 95th percentile (Twig). Soniox treats 800 ms as the point where perceived latency starts to feel slow (Soniox). Gaps under 500 ms read as natural rhythm. Gaps from 800 to 1,500 ms feel awkward or robotic. Past 1,500 ms, callers interrupt, repeat themselves, or ask whether anyone is there.

This shapes two decisions. First, the budget is first-audio, not full-response, so streaming the first spoken word out the door matters more than finishing the answer fast. Second, averages lie. A p50 of 700 ms that spikes to 2.5 seconds on one call in twenty will still make the agent feel unreliable. Design to the p95.

The six places the delay comes from

A voice turn passes through six stages before the caller hears a reply. Each has a floor set by physics and a ceiling set by configuration. Published budgets from ElevenLabs, Soniox, Twig and others cluster around these ranges (ElevenLabs, Soniox):

The two biggest lines in that list are almost always turn detection and the LLM's time-to-first-token. STT and TTS have become the small items in a modern stack. Network only dominates when a US-East server is talking to a user in Sydney.

Why time to first audio is the number that matters

Time to first audio (TTFA), sometimes called mouth-to-ear or first-byte audio, is the interval between the end of the user's turn and the first playable audio sample on the user's device. It is the only latency metric the caller's ear can detect. Time to first token (TTFT) at the LLM, STT finalization time, TTS first-byte time: those are internal stops on the way. If the pipeline streams, the user hears the opening of the sentence while the model is still writing the rest of it.

Soniox puts this bluntly: perceived latency turns on how long until the first word is ready to speak, not on how long the full answer takes to generate (Soniox). This is why a slow model that streams can feel faster than a fast model that waits. It is also why vendor TTS numbers are meaningful only when the rest of the stack streams too. Deepgram quotes sub-200 ms time-to-first-byte for Aura-2 (Deepgram), and Cartesia quotes first audio in under 90 ms for Sonic (Cartesia). Both are model-side numbers, not end-to-end.

Across the RankGrids field, catalogued end-to-end latency ranges from 90 ms at the TTS-component end (Deepgram Aura-2) to 1,200 ms for an expressive native model (Hume AI EVI 3). Neither number tells you what callers will feel on your deployment. Only the glass-to-glass p95 in your region does.

Turn detection is the longest hidden pause

If an agent feels sluggish, suspect the endpointer before the model. A naive stack sets a fixed silence timer, often 800 ms, after voice activity drops. That means the agent pays almost a full second on every reply before the LLM even sees the transcript (LiveKit). Shorten the timeout to 300 ms and the agent cuts people off mid-thought. There is no silence threshold that satisfies both failure modes at once.

The fix is semantic turn detection: a small classifier reads the streaming transcript and predicts turn completion from the content, not from silence. LiveKit's EOT transformer and similar approaches can trigger before the trailing silence begins because "I would like to book a flight to" is obviously incomplete while "Thanks, goodbye" is obviously done. Deepgram Flux builds this directly into the STT layer, which is why the RankGrids listing calls out native end-of-turn detection as the main reason to pick it despite its lack of TTS.

Native speech-to-speech models handle this differently. Instead of running a separate VAD and endpointer, they treat the conversation as a continuous audio stream and emit a response when the model decides it is the agent's turn. OpenAI gpt-realtime-2.1 and Google Gemini 2.0 Flash Live both do this, which is one reason they post lower end-to-end numbers than cascade stacks with the same models underneath.

Why your AI voice agent is slow to respond: the LLM and TTS math

Once the turn ends, the LLM has to produce its first token. On a conversational prompt with a system message, a few tool definitions and some dialog history, cached prefix inference typically returns the first token in a few hundred milliseconds on the frontier models. Long prompts, cold caches and reasoning modes push that number up. Latent Space measures around 500 ms time-to-first-byte for US clients on OpenAI's Realtime API and treats 800 ms voice-to-voice as the practical target for conversational apps (Latent Space). Reasoning modes trade latency for quality: fine for a research agent, not for a phone call.

TTS is the stage that has moved fastest. Streaming synthesis models now start emitting audio chunks before the LLM finishes its sentence. The pattern is sentence-boundary buffering: hold tokens until a period or comma, synthesize that clause while the next one is still being generated, then concatenate. ElevenLabs recommends this stage overlap, and separately suggests tuning the player buffer from a 500 ms default toward 50 to 150 ms on stable connections (ElevenLabs). Cartesia Sonic drives the TTS first-byte number down further, quoting under 90 ms first audio (Cartesia).

The practical takeaway: the LLM's first token and the TTS's first audio chunk should overlap. If your stack waits for the full LLM response before calling TTS, you have left several hundred milliseconds on the table.

Transport and buffering: WebRTC, WebSocket and SIP each set a different floor

Audio has to get to the server and back, and the choice of transport sets a floor the model cannot undo. Famulor's measurements put WebRTC first-packet latency at 60 to 120 ms, with jitter buffering, echo cancellation and packet loss recovery built in. SIP over PSTN sits at 250 to 400 ms first-packet, and each carrier hop adds 20 to 50 ms before audio reaches the AI stack, so a multi-hop path can accumulate roughly 300 ms of pure transport delay (Famulor). WebSocket is simpler to deploy but adds a server hop and does not include a jitter buffer, so any loss resilience has to be built into the client.

This is why SIP support on a voice API is not just a compliance tickbox. It changes the latency envelope. Among the RankGrids-ranked APIs, OpenAI gpt-realtime-2.1, Microsoft Azure AI Voice Live API, Amazon Nova 2.5 Sonic and Retell AI list native SIP trunking; ElevenLabs ElevenAgents, Google Gemini 3.8 Live and Hume AI EVI 3 do not. Add a managed telephony integration on top of a non-SIP API and the transport floor rises before your model does anything.

Player buffering on the client is the last stage and the one most people never measure. A default WebRTC jitter buffer can sit at 100 to 200 ms, and a browser audio element can add another 150 ms if you feed it chunks faster than it drains them. Reducing the buffer makes the agent snappier but less resilient to packet loss, so this is a tuning exercise against the actual network, not a value to copy from a blog.

What does time to first audio look like in practice?

A worked example makes the number concrete. Take a managed cascade agent on a US-East deployment answering a US caller over SIP. Realistic p50 numbers: 30 ms SIP network, 200 ms endpointing with a semantic turn detector, 60 ms STT finalization, 280 ms LLM time-to-first-token on a cached prompt, 120 ms TTS first-byte, 80 ms player buffer. That sums to about 770 ms glass-to-glass, right at the edge of the natural-feel budget.

Now add two realistic failures. The LLM has a cold cache because the session just started: +400 ms on first-token. The caller has 150 ms of extra network latency because they are on a mobile carrier with a bad path to your region: +150 ms. Suddenly the first turn feels like 1.3 seconds, and the agent gets interrupted. Most of that recovery comes from co-locating servers with users, warming caches across session setup, and shortening the endpointer, not from a faster model.

The RankGrids 2026 ranking of 22 speech-to-speech APIs catalogs end-to-end latency figures for US-East against a conversational benchmark. The range is wide: 90 ms at the TTS-component end (Deepgram Aura-2), 150 ms for Gemini 2.0 Flash Live, 320 ms for OpenAI gpt-realtime-2.1, 500 to 650 ms for most managed cascades, and over 1,000 ms for Hume AI EVI 3, AssemblyAI Voice Agent API, Amazon Nova 2.5 Sonic and Microsoft Azure Cognitive Services Speech. Those numbers come from vendor materials, so treat them as best-case ceilings, not production guarantees.

The levers that actually shorten response time

In rough order of impact, these are the changes that move the glass-to-glass number the most on a working agent.

The order matters. A team that has already done the first three usually sees little further improvement from a model swap. A team that has not will see the swap do nothing, because the endpointer is eating 800 ms regardless.

• Replace a fixed silence timeout with a semantic end-of-turn model, or pick an API with one built in (Deepgram Flux, native speech-to-speech).
• Overlap streaming across stages: STT partials into LLM prompt, LLM tokens into TTS at sentence boundaries, TTS chunks into the player.
• Co-locate your region with your users. Round-trip for a US-to-EU call over SIP can be 150 ms of pure physics.
• Match the transport to the use case. WebRTC for browser copilots, SIP for PSTN, and avoid WebSocket when you cannot afford to build your own jitter buffer.
• Trim the player buffer from 500 ms toward 100 ms on stable links, and accept the trade on lossy ones.
• Keep prompts short and cache aggressively. First-token on a cold 20K-token system message is a lot slower than on a warmed cache.
• Pick a native speech-to-speech model for latency-critical workloads, and a cascade only when you need BYO components or specific voices.