Speech-to-Speech Models vs an STT, LLM and TTS Pipeline: The Real Trade-offs

The architectural difference in one minute

A cascaded pipeline runs three models in series. Streaming STT transcribes what the caller said, a language model generates a response in text, and a TTS model renders that text as audio. A native speech-to-speech model does all of that inside one network that takes audio tokens in and emits audio tokens out, with reasoning happening between hear and speak, so there is no intermediate text by default Soniox.

The split matters for two reasons. First, the cascade's text is both its weakness and its strength: handoffs add latency, but every stage produces an inspectable transcript you can log, redact, or rewrite. Second, a native model learns to react to how something was said, not just what the words were, because the recognizer in a cascade permanently discards tone, hesitation, and emotion before the language model ever sees anything Soniox.

Some products blur the line. Hume's EVI 3 is a native speech-to-speech model that will route conversations through a custom language model you host, generating the initial response from its own weights before handing control to your LLM Hume AI. The industry has settled on calling the pure chained design a cascade or pipeline, and the single-model design native speech-to-speech or S2S, and we will use those labels throughout.

Where native speech-to-speech models actually win

Two things. Latency, and the paralanguage that transcripts throw away. On latency, Forasoft's 2026 production guide for OpenAI's Realtime API puts a well-tuned cascade at roughly 1.0 to 1.5 seconds voice-to-voice across three model hops, and a single native model at around 800ms, with the native architecture reducing the latency floor by approximately 200 to 700ms by eliminating STT finalization delays and TTS startup overhead Forasoft. The RankGrids data mirrors this: gpt-realtime-2.1 lists 320ms end-to-end against 500 to 1,000ms for cascade products like Azure Voice Live and AssemblyAI's Voice Agent API.

The paralanguage point is harder to see on a spec sheet but changes how conversations feel. Transcription discards how something was said, so a cascade cannot react to frustration, laughter, or a trailing-off sentence unless you add a separate affect model Soniox. A native model hears the tone and can match energy, which is why Hume's EVI 3 and Grok Voice 3.0 are pitched at companion apps and expressive characters rather than hold-music replacement.

Interruption handling tells a similar story. OpenAI reports its native speech-to-speech stack cut mid-conversation interruptions by almost 80 percent versus previous turn-based systems, because one model reasons over incoming and outgoing audio at the same time rather than waiting for a VAD signal to decide a turn has ended Context Studios. A cascade can get close with aggressive endpointing and tools like Deepgram Flux that fold end-of-turn detection into STT, but full duplex is still easier when hearing and speaking share weights.

Where the cascade still wins: control, text logs, and swapping models

If you need the agent to say an exact sentence word for word, the cascade is the honest answer. The text between the LLM and the TTS stage is where policy lives: compliance filters, PII redaction, scripted disclosures, and brand-safe rewrites all operate on text before anything is spoken Soniox. Native models can be prompted to say something, but there is no observable layer between reasoning and audio where you can enforce it.

Compliance teams care for a related reason. A HIPAA or SOC 2 review expects audit controls that log every access and show component-level traces. A cascade produces text at every boundary, which yields both debugging logs and the audit trail regulators ask for Context Studios. Native models give you audio in and audio out, and reconstructing what the model understood or said requires a separate transcription pass that is itself a cost line and an extra place where sensitive content lives.

Control also means component swaps. A cascade lets you pin a specific STT at a given word error rate, run the LLM on your own GPUs, and change TTS without retraining the rest. Coval's 2026 benchmark puts AssemblyAI Universal 3.6 Pro at 2.2 percent WER, Deepgram Nova 3 at 5.2 percent, and ElevenLabs Scribe v2 Realtime at 4.3 percent, and if accuracy matters for your use case, the cascade lets you buy the best STT separately rather than accepting whatever is bundled Coval. Native speech-to-speech models bundle all three layers from one vendor, so you take the STT, reasoning, and voice quality the vendor shipped.

Can you bring your own LLM to a speech-to-speech API?

Sometimes. The hyperscaler native models say no. OpenAI's gpt-realtime-2.1 and Google's Gemini 3.8 Live are closed stacks where the reasoning model is baked into the audio weights, and the RankGrids data lists BYO components as None for both OpenAI gpt-realtime-2.1 and Gemini 3.8 Live.

Hume takes a different line. EVI 3 generates the initial response from its own weights and then lets your custom LLM take over the remainder of each turn as soon as it can Hume AI. ElevenLabs is similar at the agent layer: ElevenAgents and Eleven Realtime v1 Max are LLM-agnostic, letting you plug in GPT, Claude, Gemini, or a fine-tuned open model through their orchestration ElevenLabs.

Cascades give you the most freedom here, almost by definition. Managed cascade products such as Azure AI Voice Live, Deepgram Voice Agent API, and AssemblyAI Voice Agent API all expose BYO LLM, and self-built cascades over primitives like Deepgram Flux and Cartesia Sonic 3.6 let you pick every component. The short answer: if BYO LLM is non-negotiable, you are choosing between a cascade and the small group of native models (Hume, ElevenLabs) that support external LLMs explicitly.

What counts as a realtime AI voice model in 2026

The phrase is overloaded. In practice it covers three distinct product shapes. First, native speech-to-speech models that handle hearing, reasoning, and speaking inside one model. Second, managed cascade platforms that chain the three stages for you and expose a single WebSocket or SIP endpoint. Third, component-level realtime APIs (STT-only, TTS-only, or endpointing-only) that you glue into your own stack. Inworld's 2026 roundup identifies these categories and notes that pure native models currently lock you to one vendor for all three layers Inworld.

The RankGrids Best Speech-to-Speech AI APIs 2026 ranking scores 22 products across these three shapes on latency, STT accuracy, telephony integration, language coverage, and context window. The headline split: five of the top six architectures are either native speech-to-speech or managed cascades, but Deepgram Aura-2 appears high as a pure-TTS component because teams keep assembling their own cascades for the control it gives them.

A quick side-by-side of the most-asked-about models.

Is there a single best speech-to-speech model in 2026?

No, and treat any article that gives you one as a vendor pitch. There is a best model for enterprise phone agents, a different best for companion apps, and a third best for global consumer products on a tight budget.

For a native speech-to-speech agent in a US contact center, OpenAI gpt-realtime-2.1 is the current default. It scores highest in the RankGrids ranking at 9.2, with 320ms p95 latency, a 128K context window, native SIP trunking at a 99.9 percent SLA, and reliable tool calling. The costs are real: $0.10 per minute is premium, and there is no voice cloning, so you cannot ship a bespoke brand voice.

For consumer and companion apps that need emotional range or a huge voice library, Hume EVI 3 at $0.06/min with 30 expressive styles, or ElevenLabs Eleven Realtime v1 Max at $0.08/min with 17,000-plus voices across 90-plus languages, are better fits than the OpenAI stack. EVI 3 pays for the expressiveness in latency, at 1,200ms per the RankGrids data, which is slow for telephony but acceptable for in-app conversations.

For global reach on the cheapest native price, Google Gemini 3.8 Live covers 97 languages at $0.023/min, more than four times cheaper than OpenAI, with a free tier for prototyping. It gives up SIP trunking and voice cloning. For AWS shops, Amazon Nova 2.5 Sonic at $0.015/min with SIP via Bedrock is the equivalent trade, but at 1,080ms latency.

How to pick the architecture before you pick a vendor

Start with four hard questions. Does an auditor or lawyer need to read every word your agent said? If yes, a cascade or a native-plus-logging setup wins, because you will spend less time reconstructing what happened Context Studios. Does the agent need to say exact scripted sentences for compliance or brand reasons? Again, cascade, because text-stage rewrites and claim checks are real.

Do emotional nuance and interruption handling matter more than the above? If the product is a wellness companion, a game character, or a sales rep who has to read tone, the native model usually wins on feel, and the roughly 80 percent reduction in overlap errors OpenAI reports for its native stack is representative Context Studios. Finally, is your LLM already fine-tuned on your data, or do you plan to swap LLMs as pricing changes? If yes, you need a cascade or one of the native-plus-custom-LLM products (Hume, ElevenLabs, Inworld).

The cost picture is not what most intros claim. Below roughly 10,000 minutes a month, the differences are rounding error and the right answer is whichever platform has the shortest integration time. Above that, token-based speech-to-speech pricing can grow non-linearly with conversation length, and observed costs have been reported at up to four times the theoretical minimum, so a cascade with per-minute STT and TTS plus per-token LLM becomes easier to forecast Context Studios.