Skip to content
TheChatArchiveThe Chat Archive
gpt-4o-search-preview-high-loc· OpenAIHugging Face LMArena Search Arena 24K

how to use the latest gpt-4o mini TTS in JS

  1. User

    how to use the latest gpt-4o mini TTS in JS

  2. Assistant

    To integrate OpenAI's GPT-4o Mini Text-to-Speech (TTS) capabilities into your JavaScript application, you can utilize the Azure OpenAI Service's Realtime API. This API enables real-time, low-latency conversational interactions, making it suitable for applications like voice assistants and live translators.

    Prerequisites:

    • Azure Subscription: Ensure you have an active Azure subscription.
    • Azure OpenAI Resource: Create an Azure OpenAI resource in a supported region.
    • Node.js Environment: Set up a Node.js environment with the latest LTS version.

    Steps to Implement GPT-4o Mini TTS in JavaScript:

    1. Deploy the GPT-4o Mini Realtime Model:

      • Navigate to the Azure AI Foundry portal.
      • Select the "Real-time audio playground" from the Playgrounds section.
      • Click on "+ Create new deployment" and choose "From base models."
      • Search for and select the gpt-4o-mini-realtime-preview model.
      • Deploy the model to your selected resource.
    2. Set Up Your JavaScript Project:

      • Create a new project directory and initialize it:

        mkdir gpt4o-mini-tts-js && cd gpt4o-mini-tts-js
        npm init -y
        
      • Set the project to use ECMAScript modules:

        npm pkg set type=module
        
      • Install the required packages:

        npm install https://github.com/Azure-Samples/aoai-realtime-audio-sdk/releases/download/js/v0.5.2/rt-client-0.5.2.tgz @azure/identity
        
    3. Implement the TTS Functionality:

      • Create a file named text-to-speech.js and add the following code:

        import { DefaultAzureCredential } from "@azure/identity";
        import { LowLevelRTClient } from "rt-client";
        import dotenv from "dotenv";
        dotenv.config();
        
        async function textToSpeech() {
          const endpoint = process.env["AZURE_OPENAI_ENDPOINT"];
          const deployment = "gpt-4o-mini-realtime-preview";
        
          if (!endpoint) {
            throw new Error("AZURE_OPENAI_ENDPOINT environment variable is not set.");
          }
        
          const client = new LowLevelRTClient(
            new URL(endpoint),
            new DefaultAzureCredential(),
            { deployment: deployment }
          );
        
          try {
            await client.send({
              type: "response.create",
              response: {
                modalities: ["audio", "text"],
                instructions: "Please assist the user."
              }
            });
        
            for await (const message of client.messages()) {
              switch (message.type) {
                case "response.done":
                  break;
                case "error":
                  console.error(message.error);
                  break;
                case "response.audio_transcript.delta":
                  console.log(`Received text delta: ${message.delta}`);
                  break;
                case "response.audio.delta":
                  const buffer = Buffer.from(message.delta, "base64");
                  console.log(`Received ${buffer.length} bytes of audio data.`);
                  // Here, you can handle the audio buffer, e.g., save it to a file or stream it.
                  break;
              }
              if (message.type === "response.done" || message.type === "error") {
                break;
              }
            }
          } finally {
            client.close();
          }
        }
        
        textToSpeech().catch(console.error);
        
    4. Configure Environment Variables:

      • Create a .env file in your project directory with the following content:

        AZURE_OPENAI_ENDPOINT=<your azure="" endpoint="" openai="">
        
      • Replace <your azure="" endpoint="" openai=""> with the endpoint URL from your Azure OpenAI resource.

    5. Run the Application:

      • Ensure you're authenticated with Azure CLI:

        az login
        
      • Execute the script:

        node text-to-speech.js
        
      • The script will process text input and output audio data, which you can handle as needed in your application.

    Additional Considerations:

    • Voice Customization: GPT-4o Mini TTS allows for voice customization through prompts. You can specify voice characteristics like tone, accent, and emotion by modifying the instructions field in the response.create request. For example:
      instructions: "Speak in a cheerful tone with a slight British accent."
    
    • Streaming Audio: The Realtime API supports streaming audio output, enabling low-latency responses suitable for interactive applications.

    By following these steps, you can effectively integrate GPT-4o Mini TTS into your JavaScript application, enhancing user interactions with natural and dynamic speech capabilities.