Gemini Live API Dev

Gemini Live API Development Skill

Overview

The Live API enables low-latency, real-time voice and video interactions with Gemini over WebSockets. It processes continuous streams of audio, video, or text to deliver immediate, human-like spoken responses and background reasoning.

Key capabilities:

• Bidirectional audio streaming — real-time mic-to-speaker conversations
• Background reasoning (extended thinking) — multi-step background reasoning with spoken conversational fillers
• Live streaming transcription — real-time speech-to-text with interim and finalized streams
• Video streaming — send camera/screen frames alongside audio
• Text input/output — send and receive text within a live session
• Audio transcriptions — get text transcripts of both input and output audio
• Voice Activity Detection (VAD) — automatic server VAD, client-side Hybrid VAD, and manual Push-to-Talk
• Asynchronous function calling — non-blocking tool execution while audio continues streaming
• Full-session client content — inject and update conversation turns mid-stream
• Session management — context compression, session resumption, GoAway signals
• Ephemeral tokens — secure client-side authentication

[!NOTE] The Live API connects directly via WebSockets. For WebRTC support or simplified integration, use a partner integration.

Models

Current Models (Use These)

• gemini-3.8-live — Default option for most low-latency voice agent experiences and real-time dialogue without reasoning delays. Supports interleaved reasoning, asynchronous function calling by default (behavior: NON_BLOCKING), and full-session client content updates.
• gemini-3.8-live-extended-thinking — High-reasoning audio-to-audio model recommended when higher background reasoning is required during live interactions. Processes background reasoning and async tool calls (behavior: NON_BLOCKING required) while streaming continuous spoken conversational fillers; lifecycle managed via interaction_status (IN_PROGRESS vs IDLE).
• gemini-3.5-transcribe-live — Real-time streaming speech-to-text with interim hypotheses, finalized transcripts, smart formatting, and Hybrid VAD.
• gemini-3.5-live-translate-preview — Real-time speech-to-speech streaming translation across 70+ languages.

[!WARNING] *Legacy Models (`gemini-3.1-flash-live-preview`, `gemini-2.5-flash-native-audio-, gemini-live-2.5-flash-preview, gemini-2.0-flash-live-001)**: Read [references/migration.md](references/migration.md) for breaking protocol changes (behavior: "NON_BLOCKING", thinking_level, interaction_status, send_client_content`).

SDKs

• Python: google-genai >= 2.3.0 — pip install -U google-genai
• JavaScript/TypeScript: @google/genai >= 2.3.0 — npm install @google/genai

[!WARNING] Legacy SDKs google-generativeai (Python) and @google/generative-ai (JS) are deprecated. Never use them.

Partner Integrations

To streamline real-time audio/video app development, use a third-party integration supporting the Gemini Live API over WebRTC or WebSockets:

• LiveKit — Use the Gemini Live API with LiveKit Agents.
• Pipecat by Daily — Create a real-time AI chatbot using Gemini Live and Pipecat.
• Fishjam by Software Mansion — Create live video and audio streaming applications with Fishjam.
• Vision Agents by Stream — Build real-time voice and video AI applications with Vision Agents.
• Voximplant — Connect inbound and outbound calls to Live API with Voximplant.
• Firebase AI SDK — Get started with the Gemini Live API using Firebase AI Logic.

Audio Formats

• Input: Raw PCM, little-endian, 16-bit, mono. 16kHz native (will resample others). MIME type: audio/pcm;rate=16000
• Output: Raw PCM, little-endian, 16-bit, mono. 24kHz sample rate.

[!IMPORTANT] Use send_realtime_input / sendRealtimeInput for all real-time streaming user input (audio, video, and text). On Gemini 3.8 models, send_client_content / sendClientContent is supported across the full session lifecycle with explicit roles (user or model) to inject conversation context (turn_complete=true unconditionally interrupts active generation).

[!WARNING] Do not use media in sendRealtimeInput. Use the specific keys: audio for audio data, video for images/video frames, and text for text input.

Quick Start

Authentication

Python

from google import genai

client = genai.Client(api_key="YOUR_API_KEY")

JavaScript

import { GoogleGenAI } from '@google/genai';

const ai = new GoogleGenAI({ apiKey: 'YOUR_API_KEY' });

Connecting to the Live API

Python

from google.genai import types

config = types.LiveConnectConfig(
    response_modalities=[types.Modality.AUDIO],
    system_instruction=types.Content(
        parts=[types.Part(text="You are a helpful assistant.")]
    )
)

async with client.aio.live.connect(model="gemini-3.8-live", config=config) as session:
    pass  # Session is active

JavaScript

const session = await ai.live.connect({
  model: 'gemini-3.8-live',
  config: {
    responseModalities: ['audio'],
    systemInstruction: { parts: [{ text: 'You are a helpful assistant.' }] }
  },
  callbacks: {
    onopen: () => console.log('Connected'),
    onmessage: (response) => console.log('Message:', response),
    onerror: (error) => console.error('Error:', error),
    onclose: () => console.log('Closed')
  }
});

Sending Text

Python

await session.send_realtime_input(text="Hello, how are you?")

JavaScript

session.sendRealtimeInput({ text: 'Hello, how are you?' });

Sending Audio

Python

await session.send_realtime_input(
    audio=types.Blob(data=chunk, mime_type="audio/pcm;rate=16000")
)

JavaScript

session.sendRealtimeInput({
  audio: { data: chunk.toString('base64'), mimeType: 'audio/pcm;rate=16000' }
});

Sending Video

Python

# frame: raw JPEG-encoded bytes
await session.send_realtime_input(
    video=types.Blob(data=frame, mime_type="image/jpeg")
)

JavaScript

session.sendRealtimeInput({
  video: { data: frame.toString('base64'), mimeType: 'image/jpeg' }
});

Receiving Audio and Text

[!IMPORTANT] A single server event can contain multiple content parts simultaneously (e.g., audio chunks and transcript). Always process all parts in each event to avoid missing content.

Python

async for response in session.receive():
    content = response.server_content
    if content:
        # Audio — process ALL parts in each event
        if content.model_turn:
            for part in content.model_turn.parts:
                if part.inline_data:
                    audio_data = part.inline_data.data
        # Transcription
        if content.input_transcription:
            print(f"User: {content.input_transcription.text}")
        if content.output_transcription:
            print(f"Gemini: {content.output_transcription.text}")
        # Interruption
        if content.interrupted is True:
            pass  # Stop playback, clear audio queue

JavaScript

// Inside the onmessage callback
const content = response.serverContent;
if (content?.modelTurn?.parts) {
  for (const part of content.modelTurn.parts) {
    if (part.inlineData) {
      const audioData = part.inlineData.data; // Base64 encoded
    }
  }
}
if (content?.inputTranscription) console.log('User:', content.inputTranscription.text);