tell me about the multimodal live api
- User
tell me about the multimodal live api
- Assistant
The Multimodal Live API is a service provided by Google that enables real-time, bidirectional interactions with the Gemini model through voice and video. This API allows developers to create applications where users can engage in natural, human-like conversations with the model, utilizing text, audio, and video inputs, and receiving text and audio outputs[1].
Key Features:
-
Multimodal Interaction: Supports text, audio, and video inputs and outputs, facilitating rich, interactive experiences.
-
Low-Latency Communication: Ensures fast responses, making it suitable for real-time applications.
-
Session Memory: Remembers interactions within a session, allowing the model to recall previously heard or seen information.
-
Function Calling and Code Execution: Integrates with external services and data sources, enabling the model to perform actions beyond its initial training.
Getting Started:
To begin using the Multimodal Live API, you can access the Web Console Demo project, which provides a tutorial on interacting with Gemini using voice and camera inputs[1].
Integration Details:
The API operates over WebSockets, establishing a session between the client and the Gemini server. This session allows for the exchange of text, audio, and video messages. The initial message after connection sets the session configuration, including the model, generation parameters, system instructions, and tools[2].
Limitations:
-
Client Authentication: The API is designed for server-to-server communication and is not recommended for direct client use. Client input should be routed through an intermediate application server for secure authentication.
-
Session Duration: Sessions are limited to up to 15 minutes for audio or up to 2 minutes of audio and video. Exceeding these limits will terminate the connection.
-
Voice Activity Detection (VAD): The model automatically performs VAD on continuous audio input streams, and its parameters are not configurable.
-
Rate Limits: The API enforces rate limits, including a maximum of 3 concurrent sessions per API key and 4 million tokens per minute.
For detailed information on using the Multimodal Live API, including code examples and configuration options, refer to the official documentation[1].
-