how to use the latest gpt-4o mini TTS in JS
- User
how to use the latest gpt-4o mini TTS in JS
- Assistant
To integrate OpenAI's GPT-4o Mini Text-to-Speech (TTS) capabilities into your JavaScript application, you can utilize the Azure OpenAI Service's Realtime API. This API enables real-time, low-latency conversational interactions, making it suitable for applications like voice assistants and live translators.
Prerequisites:
- Azure Subscription: Ensure you have an active Azure subscription.
- Azure OpenAI Resource: Create an Azure OpenAI resource in a supported region.
- Node.js Environment: Set up a Node.js environment with the latest LTS version.
Steps to Implement GPT-4o Mini TTS in JavaScript:
-
Deploy the GPT-4o Mini Realtime Model:
- Navigate to the Azure AI Foundry portal.
- Select the "Real-time audio playground" from the Playgrounds section.
- Click on "+ Create new deployment" and choose "From base models."
- Search for and select the
gpt-4o-mini-realtime-previewmodel. - Deploy the model to your selected resource.
-
Set Up Your JavaScript Project:
-
Create a new project directory and initialize it:
mkdir gpt4o-mini-tts-js && cd gpt4o-mini-tts-js npm init -y -
Set the project to use ECMAScript modules:
npm pkg set type=module -
Install the required packages:
npm install https://github.com/Azure-Samples/aoai-realtime-audio-sdk/releases/download/js/v0.5.2/rt-client-0.5.2.tgz @azure/identity
-
-
Implement the TTS Functionality:
-
Create a file named
text-to-speech.jsand add the following code:import { DefaultAzureCredential } from "@azure/identity"; import { LowLevelRTClient } from "rt-client"; import dotenv from "dotenv"; dotenv.config(); async function textToSpeech() { const endpoint = process.env["AZURE_OPENAI_ENDPOINT"]; const deployment = "gpt-4o-mini-realtime-preview"; if (!endpoint) { throw new Error("AZURE_OPENAI_ENDPOINT environment variable is not set."); } const client = new LowLevelRTClient( new URL(endpoint), new DefaultAzureCredential(), { deployment: deployment } ); try { await client.send({ type: "response.create", response: { modalities: ["audio", "text"], instructions: "Please assist the user." } }); for await (const message of client.messages()) { switch (message.type) { case "response.done": break; case "error": console.error(message.error); break; case "response.audio_transcript.delta": console.log(`Received text delta: ${message.delta}`); break; case "response.audio.delta": const buffer = Buffer.from(message.delta, "base64"); console.log(`Received ${buffer.length} bytes of audio data.`); // Here, you can handle the audio buffer, e.g., save it to a file or stream it. break; } if (message.type === "response.done" || message.type === "error") { break; } } } finally { client.close(); } } textToSpeech().catch(console.error);
-
-
Configure Environment Variables:
-
Create a
.envfile in your project directory with the following content:AZURE_OPENAI_ENDPOINT=<your azure="" endpoint="" openai=""> -
Replace
<your azure="" endpoint="" openai="">with the endpoint URL from your Azure OpenAI resource.
-
-
Run the Application:
-
Ensure you're authenticated with Azure CLI:
az login -
Execute the script:
node text-to-speech.js -
The script will process text input and output audio data, which you can handle as needed in your application.
-
Additional Considerations:
- Voice Customization: GPT-4o Mini TTS allows for voice customization through prompts. You can specify voice characteristics like tone, accent, and emotion by modifying the
instructionsfield in theresponse.createrequest. For example:
instructions: "Speak in a cheerful tone with a slight British accent."- Streaming Audio: The Realtime API supports streaming audio output, enabling low-latency responses suitable for interactive applications.
By following these steps, you can effectively integrate GPT-4o Mini TTS into your JavaScript application, enhancing user interactions with natural and dynamic speech capabilities.