Gemini Live API vs OpenAI Realtime API: How Sessions, Events and Audio Differ

Why do the two APIs look similar but not swap cleanly?

Both APIs give you a persistent duplex connection with speech in, speech out, text transcripts, tool calls and barge-in. That is the surface. OpenAI documents three connection types for its Realtime API: WebRTC, WebSocket and SIP, with WebRTC handling media buffering and truncation server-side (OpenAI Realtime conversations). Google's Gemini Live API is WebSocket only, a bidirectional stream against `BidiGenerateContent` (Gemini Live API get-started).

The implication for porting work is that if your existing OpenAI integration leans on WebRTC for media handling, moving to Gemini means picking up jitter buffering, playback-position tracking and interruption bookkeeping yourself. A third-party comparison from Micdrop recommends OpenAI Realtime for tool-heavy agents because of its MCP support and single 60-minute session, and Gemini Live when per-minute cost dominates at roughly one-quarter the price (Micdrop). RankGrids ranks both at the top of the Best Speech To Speech AI APIs 2026 list, with OpenAI gpt-realtime-2.1 at rank 1 and Google Gemini 3.8 Live at rank 5, with integration and reliability scores favoring OpenAI even though Gemini 3.8 Live lists at 0.023 dollars per minute against OpenAI's 0.10 dollars per minute.

Connection and session setup: the first message is not the same shape

OpenAI's WebSocket endpoint is `wss://api.openai.com/v1/realtime`. You can open it, start streaming audio and the model will respond using the session defaults: `pcm16` input and output at 24 kHz, the configured default voice, server VAD on, and a 60-minute cap (OpenAI Realtime conversations). To change anything, send a `session.update` event with the fields you want.

Gemini's endpoint is `wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent?key=YOUR_API_KEY`, or the Constrained variant with an ephemeral `access_token` (Gemini Live API get-started). The first message is not optional: it must be a `BidiGenerateContentSetup` envelope with `model`, `responseModalities` (for example `["AUDIO"]`), optional `systemInstruction`, `speechConfig`, `realtimeInputConfig` for VAD and `thinkingConfig` for reasoning depth. Everything after that is wrapped in `realtimeInput`, `clientContent`, or `toolResponse` messages. There is no flat `session.update`; the setup message is single-shot.

Event names: there is no one-to-one mapping

Neither side published a Rosetta Stone. The pragmatic mapping developers have reverse-engineered looks like this, with caveats that it is lossy in places because the semantics do not line up (Zylos adapter research).

Audio formats do not line up, and the fix is resampling

The Gemini Live API's input is raw 16-bit PCM at 16 kHz, little-endian, with MIME type `audio/pcm;rate=16000`; output is 24 kHz PCM (Gemini Live API get-started). The API will accept other input sample rates and resample server-side, but if you are building for latency you want to send 16 kHz directly.

OpenAI's Realtime API defaults to 24 kHz PCM16 mono in both directions (OpenAI Realtime conversations). An OpenAI capture pipeline that is already emitting 24 kHz has to be downsampled to 16 kHz before it hits Gemini. Naive 3:2 decimation introduces aliasing; the correct approach is band-limited resampling with a low-pass filter at the target Nyquist frequency before decimation (Zylos adapter research). If you are going the other way, 48 kHz microphone capture to Gemini, staged conversion (48 to 24 to 16) is safer than a single filter because each stage introduces less passband ripple.

Output does not need conversion: both APIs emit 24 kHz PCM, which lines up with most browser and SIP playback pipelines.

Turn detection is server-declarative on Gemini and client-coordinated on OpenAI

OpenAI's VAD ships in two flavors. `server_vad` is energy-based with `threshold`, `prefix_padding_ms` and `silence_duration_ms`, plus the newer `create_response` and `interrupt_response` knobs. `semantic_vad` is model-judged with an `eagerness` parameter (`low`, `high`, or `auto`, where `auto` maps to a medium setting) that tunes how readily the model cuts a user off (OpenAI Realtime VAD). When a user interrupts, the client is responsible for telling the server exactly how much audio was played by sending `conversation.item.truncate` with `item_id`, `content_index` and `audio_end_ms` in milliseconds (OpenAI Realtime conversations).

Gemini's automatic VAD exposes `startOfSpeechSensitivity` and `endOfSpeechSensitivity` (`LOW`, `MEDIUM`, `HIGH`) plus `prefixPaddingMs` and `silenceDurationMs` (Gemini Live capabilities). You can also turn it off by setting `automaticActivityDetection.disabled` and manually emit `activityStart` and `activityEnd`. Interruption is server-declarative: the server unilaterally sets an `interrupted` flag and the client just drains its playback queue. There is no mechanism to tell Gemini how many milliseconds of its last turn the user actually heard. Adapters that want parity with OpenAI must track local playback position independently.

Tool calling: synchronous wait on OpenAI, non-blocking by default on Gemini

OpenAI's tool-call cycle is a three-step dance: the model emits a `function_call` with a `call_id`, your client runs the function, you post a `function_call_output` conversation item with the same `call_id`, then you send `response.create` to resume (OpenAI Realtime client events). The model is silent during the tool round-trip.

Gemini Live is different: on `gemini-3.8-live`, function calling defaults to `NON_BLOCKING` so the model keeps talking while the tool runs in the background and patches results in later (Gemini Live capabilities). The upside is a more natural rhythm on long-running tools. The downside is that an adapter assuming OpenAI's block-until-tool-returns semantics will either stall waiting for a turn that was never going to block, or mishandle audio that keeps streaming past a still-pending tool call. You can force `BLOCKING` mode per call if you need OpenAI-shaped behavior. The extended-thinking variant of Gemini 3.8 Live only supports `NON_BLOCKING`.

OpenAI additionally supports remote MCP servers directly inside a Realtime session so the tool call runs server-side and bypasses your infrastructure altogether (Micdrop). Gemini does not have the equivalent today; you still ship the tool loop to your own worker.

Session limits and resumption are different problems

OpenAI caps a single Realtime session at 60 minutes (OpenAI Realtime conversations). For most inbound support calls that is more than enough, but for long campaign calls or AI companion sessions you need to stitch multiple sessions together on your side.

Gemini's limits are tighter per connection but flexible across them. Audio-only sessions top out at 15 minutes (Gemini Live capabilities), and individual WebSocket connections last about 10 minutes before the server cycles them (Micdrop). Session resumption returns a handle valid for two hours that lets you reopen the session on a new connection without losing conversation state. Context window is 131K tokens on the current Gemini 3.8 Live model and 128K tokens on OpenAI's gpt-realtime-2.1 per the specs on the OpenAI gpt-realtime-2.1 page and Google Gemini 3.8 Live page.

If your product has calls that regularly exceed 10 minutes, the Gemini side needs explicit resumption logic and the OpenAI side does not. If calls exceed 60 minutes, both need it.

Wrapping one interface over both: what teams actually ship

There are three paths. The first is to build a thin adapter that normalizes one API to the surface of the other. Most teams pick the OpenAI surface as the target because it has more public documentation and more existing client code. The adapter translates `session.update` to a `BidiGenerateContentSetup` at connect time, synthesizes `speech_started` and `speech_stopped` from Gemini's VAD or runs a local VAD as fallback, buffers transcription deltas to look like OpenAI's framing, resamples audio and translates tool calls (Zylos adapter research).

The second is to pick up Pipecat. Pipecat's processor model wraps both OpenAI Realtime and Gemini Live as speech-to-speech services behind a common Frame-based pipeline, alongside STT-LLM-TTS cascades (Forasoft Pipecat vs LiveKit). Swapping providers is roughly a one-line service change because the frames entering and leaving the processor are the same shape. Pipecat transports are also agnostic: the same pipeline runs on Daily, raw WebSocket, Twilio or LiveKit's audio-only mode.

The third is LiveKit Agents with its provider plugins (LiveKit OpenAI plugin). LiveKit handles the media transport (WebRTC to end users, PSTN via SIP providers) and the agent framework handles the provider glue. You configure the provider as a plugin and the rest of the agent code does not change when you flip it.

Worth noting: Inworld AI Realtime API ships explicit OpenAI Realtime protocol compatibility, so clients that already speak OpenAI's event shape can hit Inworld without modification. That is the exception; most providers invent their own event names and leave the glue to you or to Pipecat and LiveKit.

What actually breaks when you flip one API for the other

The failure modes cluster in a few places. Audio arriving at the wrong sample rate sounds harsh and sometimes degrades speech recognition on the Gemini side because the resampler has to clean up aliasing. Interruptions produce clipped words at the start of the agent's next turn when the adapter is not tracking local playback position. Tool calls stall when you assume OpenAI semantics on Gemini, or produce jarring overlapping audio when you do not expect Gemini's model to keep talking during a background tool. Sessions drop silently at 10 minutes on Gemini when you forget resumption. And thinking budget has no OpenAI analog: Gemini's `thinkingConfig` adds latency to time-to-first-audio and some model versions leak thinking content into responses even with `thinking_budget=0`, so adapters defensively filter (Zylos adapter research).

The reasonable rule: do not try to make the two APIs identical. Make the parts of your app that do not care about provider truly not care, and quarantine the parts that do.