How to Choose a Speech-to-Speech API for a Voice Agent

Pin the latency budget before you compare anything else

Human turn-taking in conversation happens at a median gap of about 200 milliseconds, and the feel degrades steeply past 800 milliseconds. Twig's summary of the research lays out the bands: 0 to 500 ms feels natural, 500 to 800 ms reads as deliberate, 800 to 1500 ms as awkward or robotic, and above 1500 ms as broken. The target Twig recommends for a production voice agent is sub-800 ms first-audio latency at p50 and sub-1500 ms at p95, and every component (endpoint detection, final transcription, LLM inference, any tool call, TTS first chunk, network jitter) has to share that budget.

Vendor-reported numbers look better than the numbers users feel. OpenAI's gpt-realtime-2.1 posts a 320 ms p95 for its model stage. Google Gemini 2.0 Flash Live reports 150 ms end-to-end. Cartesia Sonic 3.6 advertises sub-90 ms time-to-first-audio at the TTS layer. These are component latencies, not what the caller experiences. End-of-turn detection usually adds hundreds of milliseconds on top, which is why Deepgram's Flux model pushes turn detection inside the STT model and, per Telnyx's integration report, cuts agent response time by 200 to 600 ms versus a VAD-plus-STT pipeline. If latency is your top priority, budget for the detector before you pick a provider.

Native speech-to-speech or a cascade of STT, LLM and TTS

A native speech-to-speech model takes audio in and returns audio out in one pass. OpenAI's Realtime models, Google Gemini Live, Hume EVI 3, xAI Grok Voice 3 and Amazon Nova 2.5 Sonic work this way. A cascade runs STT, then an LLM, then TTS as separate services. ElevenLabs ElevenAgents, Azure Voice Live, Deepgram's Voice Agent API and Retell AI are cascades.

Native models usually win on latency and on prosody that reflects what the user said (tone, laughter, pauses) because the audio never gets flattened to text. Cascades win on flexibility: you can swap the LLM for a cheaper or smarter one, you can route STT through your own compliance stack, and you can pick a TTS voice from a library of thousands instead of a handful of presets. Retell, ElevenLabs and Azure all support BYO LLM; Deepgram Flux deliberately ships only the STT side because its authors argue end-of-turn detection belongs in the speech model and nowhere else.

The trade-off that is easy to miss: cascades let you redact or audit text at every stage. Native speech-to-speech models give you audio in and audio out, with less to hook compliance tooling into. For regulated workloads (healthcare, finance), teams often choose Azure's cascade or a custom pipeline over a native model for exactly this reason.

If it has to answer the phone, you need SIP

Native SIP support is the single biggest axis separating these APIs for telephony. OpenAI added native SIP to the Realtime API in August 2025: a carrier points its trunk at sip.api.openai.com, a webhook fires on each incoming call, and the model answers without a FreeSWITCH media bridge. Azure Voice Live routes SIP through Azure Communication Services, with Direct Routing for your existing PSTN carrier or an ACS-provided number. Those are the hyperscaler choices with real native SIP today.

Gemini Live, ElevenLabs, Hume and Cartesia do not expose a SIP endpoint. You bridge to them with Twilio, Telnyx or a partner like LiveKit, which works fine but adds a hop and some latency. Retell AI takes a different path: it bundles SIP trunking inside the platform, and using your own SIP trunk drops the $0.015-per-minute telephony charge to zero. That matters at volume. Amazon Nova 2.5 Sonic gets SIP via Amazon Bedrock and Amazon Connect, which is the AWS answer to the same problem.

If you are building a contact center agent, treat native SIP as a hard filter. Bridged telephony works for prototyping and for low-volume outbound, but every carrier hop you remove is latency and failure modes you do not have to debug later.

How interruptions and turn detection actually behave

The feature labeled interruptionHandling on a spec sheet is rarely the same thing across providers. OpenAI's Realtime API over WebRTC gets this almost for free: the media stack on the browser handles echo cancellation and barge-in, and OpenAI recommends WebRTC over WebSocket for anything client-side exactly because you would otherwise reimplement jitter buffering and VAD by hand. The practical effect is that a user can start talking over the agent and the agent will stop within a few audio frames.

Google Gemini Live supports three response-scheduling policies for interruption: SILENT, WHEN_IDLE and INTERRUPT, each tunable per tool and per turn. When a user barges in on a blocking tool call, the server sends a tool_call_cancellation message and the model regenerates against the new input. That is more control than OpenAI offers, but it is also more plumbing you own.

Deepgram Flux is the odd one out: it builds end-of-turn detection into the STT model rather than relying on an external VAD. Telnyx's integration report puts the savings at 200 to 600 ms of agent response time versus an STT-plus-VAD pipeline, a bigger swing than switching LLMs. If you are building your own cascade, Flux at $0.0077 per minute of STT is doing more than transcription.

Hume EVI, Alibaba Qwen-Audio-3.1-Realtime and Inworld add prosodic interruption: they modulate the voice based on detected caller emotion. That is a differentiator for companion apps and game characters, not for contact centers.

Tool calling that does not stall the voice

Every modern realtime API supports function calling, but how a tool call interacts with the voice stream is where they differ. Early 2024-era implementations stopped generating audio while a tool ran, leaving dead air of a second or more. OpenAI fixed this in gpt-realtime with asynchronous function calling: long-running calls no longer disrupt the session, and the model keeps conversing while it waits for results.

Google Gemini Live goes further with explicit behaviors. A tool can be marked NON_BLOCKING for background lookups whose result can be delivered later, and function responses can be scheduled with SILENT, WHEN_IDLE or INTERRUPT. That gives a developer fine-grained control over whether a database query makes the agent go quiet, or whether the agent fills the gap with a 'let me check that' until the result arrives.

At the cascade layer, Retell, ElevenLabs, Cartesia Sonic and Azure Voice Live all support tool calls through their orchestration layer, but the behavior during the call is a function of the LLM you plug in. If the underlying LLM does not stream partial output while a function runs, no amount of transport cleverness recovers the pause. Deepgram Aura-2 and standalone TTS products do not do tool calling at all; they are the voice component, not the brain.

Voices, languages, and whether you can bring your own

Voice library size is the clearest axis here, but it is misleading on its own. OpenAI gpt-realtime-2.1 ships 11 preset voices and no cloning; Gemini 3.8 Live ships 30 preset voices across 97 languages and no cloning; Azure Voice Live has 600 neural voices across 150 locales plus Custom Neural Voice for cloning that requires a review process. For an enterprise agent that just needs to sound competent in one language, 11 preset voices is enough.

Where cloning matters is companion apps, game characters, and any product where the voice is a brand asset. ElevenLabs Eleven Realtime v1 Max ships 17,000-plus voices and cloning with 90-plus languages. Hume EVI 3 offers a 30-second cloning workflow and 30 expressive styles tuned for empathy. Cartesia Sonic 3.6 offers instant cloning across 44 languages. Inworld ships Unity and Unreal SDKs with 200-plus-language TTS and character-driven voices.

Language coverage splits cleanly: Gemini 3.8 Live at 97 languages and ElevenLabs Realtime v1 Max at 90-plus lead on breadth; OpenAI at 57 and Azure Voice Live at 50 are solid on the top global languages; Deepgram Aura-2 and CosyVoice 2 are narrow (six languages or fewer) and are not the right pick for a global product. Alibaba Qwen-Audio-3.1-Realtime is the right pick when Chinese and East Asian markets dominate.

What a conversation minute actually costs

Headline pricing on these APIs ranges roughly 10x, from $0.011 per minute for Deepgram Aura-2 TTS alone to $0.10 per minute for gpt-realtime-2.1. The real per-minute number for a working agent is almost always higher than what the vendor advertises because the headline bundles voice but excludes the LLM or telephony. GetMacha's breakdown of a realistic Retell support call (GPT 5.4 mini, US telephony, knowledge base) lands at $0.114 per minute all-in.

The table below shows published voice-side pricing for the APIs most commonly shortlisted. Telephony, LLM fees for BYO cascades and premium support tiers sit on top.

OpenAI Realtime vs Gemini Live: which should I use

This is the choice most builders actually face, and the honest answer is that they optimize for different things. OpenAI gpt-realtime-2.1 is the steadier starting point: native SIP, a 99.9 percent uptime target, a 128K context, interruption handling that just works over WebRTC with no code, and asynchronous tool calls that keep the conversation alive during slow lookups. Google Gemini 3.8 Live is dramatically cheaper (around a quarter the per-minute cost), covers 97 languages against OpenAI's 57, and supports explicit tool-scheduling behaviors (SILENT, WHEN_IDLE, INTERRUPT) that give fine-grained control over barge-in.

Forasoft's side-by-side and Just Think AI's comparison agree on the shape: OpenAI gets a first production voice feature shipped with the least code, while Gemini Live wins on cost once the UX is settled. If long call histories matter, Gemini 2.0 Flash Live carries a 1M-token context (Gemini 3.8 Live itself sits at 131K). The current limitation is telephony: Gemini has no native SIP, so a phone product routes through Twilio or similar.

A reasonable default: prototype on OpenAI Realtime to get interruptions and tool calls working, then port to Gemini Live if cost math forces it or if you need languages OpenAI does not cover. Builders shipping a browser-only product can start on either; builders shipping a phone product should start on OpenAI or on a provider with bundled SIP like Retell.

The decision, by use case

Contact center replacing an IVR: OpenAI gpt-realtime-2.1 or Azure Voice Live. Both have native SIP (OpenAI direct, Azure via ACS), both carry a 99.9 percent uptime target, and both handle tool calling without going mute. Azure wins if compliance pulls you to the Microsoft stack; OpenAI wins on latency and on simpler onboarding.

High-volume outbound dialer: Retell AI or a cascade built on Deepgram Flux plus an LLM plus Cartesia Sonic or Deepgram Aura-2. Retell's bundled SIP and second-based billing are hard to beat out of the box; the custom cascade wins on margin once you are past 100,000 minutes per month.

Companion app or wellness product: Hume EVI 3 for empathic prosody, or ElevenLabs Eleven Realtime v1 Max for voice identity across languages. The 1,200 ms latency on EVI 3 is survivable here because the product is not a phone line.

Global multilingual product on a budget: Gemini 3.8 Live at $0.023 a minute and 97 languages, or Alibaba Qwen-Audio-3.1-Realtime where Chinese and East Asian markets dominate.

Game character or interactive media: Inworld AI for Unity and Unreal SDKs, or ElevenLabs ElevenAgents for voice library breadth and BYO LLM flexibility.