Skip to content
TheChatArchiveThe Chat Archive
gpt-4-1106-preview· openaiHugging Face Suzhen/CodeChat

Hello there, I've been trying to implement fast audio streaming for 6 days now, but I…

  1. User

    Hello there,

    I've been trying to implement fast audio streaming for 6 days now, but I just can't do it. Indeed, without the stream method, the save function of edge-tts takes about 1-2s to generate depending the text, which is too long.

    In my code bellow, the audio is indeed played instantly, regardless of text size, but there are artefacts between chunks, like tiny spaces.

    Do you know how to do audio streaming correctly please ? Thank you!

    import asyncio import edge_tts import pyaudio from io import BytesIO from pydub import AudioSegment

    TEXT = "Hello World!" VOICE = "en-GB-SoniaNeural"

    p = pyaudio.PyAudio()

    async def stream_tts(text: str, voice: str) -> None:

    We're assuming a certain format, channels, and rate

    This will need to be dynamic based on the actual audio data from TTS

    stream = p.open( format=pyaudio.paInt16, channels=1, rate=26000, output=True )

    communicate = edge_tts.Communicate(text, voice)

    Process and play audio chunks as they arrive

    async for chunk in communicate.stream(): if chunk["type"] == "audio": try: audio_segment = AudioSegment.from_file(BytesIO(chunk["data"]), format="mp3")

            # Write data to the stream directly without extra buffering
            stream.write(audio_segment.raw_data)
            
            # If this is the last chunk, break after playing
            if chunk.get('end', False):
                break
        except Exception as e:
            print("Error processing audio chunk:", e)
    

    Cleanup

    stream.stop_stream() stream.close() p.terminate() if name == "main":

    Run the asyncio event loop

    asyncio.run(stream_tts(TEXT, VOICE))

  2. Assistant

    It appears that you're running into an issue commonly known as 'audio glitching' which happens during streaming when audio chunks are played back without seamless transition between them. These artifacts could occur for a variety of reasons, such as differences in the duration of chunks, buffer underruns, or even the method of playback.

    Let's go through some steps to potentially fix these issues:

    1. Verify chunk sizes: Make sure that your TTS system sends the audio in uniform chunk sizes and that the sizes are conducive to smooth streaming.

    2. Adjust buffer sizes: Sometimes the buffer size needs to be tuned for optimal performance. Too small of a buffer can lead to underruns, while too large of a buffer could introduce latency.

    3. Remove Playback Delays: Any computation or processing between chunks can introduce delays. Make sure that the time it takes to process and play a chunk is consistent and minimal.

    4. Use native formats if possible: Instead of converting chunks to a different format (like using AudioSegment.from_file), try to work with the native format that your TTS service provides, if it's compatible with your playback system.

    5. Check pyaudio's callback mode: If possible, use pyaudio's callback mode instead of blocking mode to aid with smooth playback. Callback mode is designed to handle streaming uses cases.

    Here's a simplified example of using callback mode in pyaudio:

    import pyaudio
    
    # Example of how to set up a stream using callback mode
    def callback(in_data, frame_count, time_info, status):
        data = get_next_chunk_somehow()  # You’ll need to handle getting the data
        return (data, pyaudio.paContinue)
    
    p = pyaudio.PyAudio()
    
    stream = p.open(format=pyaudio.paInt16,
                    channels=1,
                    rate=26000,
                    output=True,
                    stream_callback=callback)
    
    stream.start_stream()
    # Now the stream is running in the background calling the callback function
    while stream.is_active():
        time.sleep(0.1)  # Keep your main program running
    
    stream.stop_stream()
    stream.close()
    p.terminate()
    

    Please note that get_next_chunk_somehow() is just a placeholder to indicate where you would retrieve the next chunk of data from the TTS service.

    To apply the callback mode to your code, you would need to reorganize your logic so that the callback function gets called with each new chunk from the streaming API.

    Finally, here’s a key point about using edge_tts: if you're noticing artifacts it could also be due to the TTS service itself or the way in which the edge_tts library fetches and stitches together audio segments. Ensure that you are using the latest version of the edge_tts library and check if there are any open issues or updates regarding streaming glitches.

  3. User

    make the callback function please

  4. Assistant

    To integrate the callback function into your streaming logic, you will need to create a queue or buffer to hold your edge_tts chunks, which the callback function will consume. Please note that the code below is a conceptual example; you might need to adjust it for your specific use case and ensure proper exception handling based on the behavior of the edge_tts library.

    Here's an example of how you might set this up with a queue:

    import asyncio
    import edge_tts
    import pyaudio
    import queue
    
    TEXT = "Hello World!"
    VOICE = "en-GB-SoniaNeural"
    
    p = pyaudio.PyAudio()
    audio_queue = queue.Queue()
    stream_ended = False
    
    def callback(in_data, frame_count, time_info, status):
        try:
            # Retrieve data from the queue
            chunk = audio_queue.get_nowait()
        except queue.Empty:
            # No data available
            if stream_ended:
                return (None, pyaudio.paComplete)
            else:
                # Return silence if no data
                data = b'\x00' * frame_count * 2  # assuming a sample width of 2 bytes
                return (data, pyaudio.paContinue)
    
        # Return data and continue playback
        return (chunk, pyaudio.paContinue)
    
    # Open stream using callback
    stream = p.open(
        format=pyaudio.paInt16,
        channels=1,
        rate=26000,
        output=True,
        stream_callback=callback
    )
    
    async def stream_tts(text: str, voice: str):
        global stream_ended
        communicate = edge_tts.Communicate(text, voice)
        async for chunk in communicate.stream():
            if chunk["type"] == "audio":
                # Put the received chunk into the queue
                audio_queue.put(chunk["data"])
                if chunk.get('end', False):
                    stream_ended = True
                    break
    
    if __name__ == "__main__":
        # Start the stream
        stream.start_stream()
    
        # Run the asyncio event loop
        loop = asyncio.get_event_loop()
        loop.run_until_complete(stream_tts(TEXT, VOICE))
    
        # Wait for stream to complete playback
        while stream.is_active():
            loop.run_until_complete(asyncio.sleep(0.1))
    
        # Close the stream and terminate PyAudio
        stream.stop_stream()
        stream.close()
        p.terminate()
    

    This code assumes you are receiving chunks of audio data as bytes (compatible with pyaudio.paInt16 format), which is then placed directly into the queue without conversion. When the callback function is called by PyAudio, it attempts to get an audio chunk from the queue without waiting (get_nowait), and plays it. If the queue is empty, the callback function generates silent frames to prevent underflows until new data arrives.

    Please keep in mind that this is a general example, designed to give you a starting point. You may need to adjust the sample rates, formats, and handle exceptions or other details based on the edge_tts library's behavior and the audio specifications you're working with.

  5. User

    The sound is robotic, you forgot AudioSegment.from_file(BytesIO(chunk[“data”]), format=“mp3”)

  6. Assistant

    If you're receiving MP3 data from edge_tts and need to use AudioSegment to convert it to raw audio data for pyaudio, you'll need to do so in a way that doesn't introduce delay or block the PyAudio callback. Since decoding can take an unpredictable amount of time, it's recommended to perform this task asynchronously outside the callback and then place the raw audio data into the queue.

    Here's a modification of the previous example that incorporates decoding of MP3 chunks using AudioSegment. You would need to install the pydub and ffmpeg or avconv which pydub uses under the hood:

    import asyncio
    import edge_tts
    import pyaudio
    import queue
    from pydub import AudioSegment
    from io import BytesIO
    
    TEXT = "Hello World!"
    VOICE = "en-GB-SoniaNeural"
    
    p = pyaudio.PyAudio()
    audio_queue = queue.Queue()
    stream_ended = False
    
    # PyAudio callback function
    def callback(in_data, frame_count, time_info, status):
        try:
            chunk = audio_queue.get_nowait()
        except queue.Empty:
            if stream_ended:
                return (None, pyaudio.paComplete)
            else:
                data = b'\x00' * frame_count * 2  # assuming a sample width of 2 bytes
                return (data, pyaudio.paContinue)
        return (chunk, pyaudio.paContinue)
    
    # Function to process and enqueue audio
    async def process_audio(data: bytes):
        try:
            # Decode MP3 to raw audio using pydub
            audio_segment = AudioSegment.from_file(
                BytesIO(data), format="mp3"
            )
            # Add the decoded raw audio to the queue
            audio_queue.put(audio_segment.raw_data)
        except Exception as e:
            print("Error processing audio chunk:", e)
    
    # Async coroutine for streaming and decoding TTS
    async def stream_tts(text: str, voice: str):
        global stream_ended
        communicate = edge_tts.Communicate(text, voice)
        async for chunk in communicate.stream():
            if chunk["type"] == "audio":
                # Process and enqueue audio asynchronously
                await process_audio(chunk["data"])
    
                if chunk.get('end', False):
                    stream_ended = True
                    break
    
    # Open the PyAudio stream with the specified parameters
    stream = p.open(
        format=pyaudio.paInt16,
        channels=1,
        rate=26000,
        output=True,
        stream_callback=callback
    )
    
    if __name__ == "__main__":
        # Start the PyAudio stream
        stream.start_stream()
    
        # Run the asyncio event loop
        loop = asyncio.get_event_loop()
        loop.run_until_complete(stream_tts(TEXT, VOICE))
    
        # Wait for stream to complete playback
        while stream.is_active():
            loop.run_until_complete(asyncio.sleep(0.1))
    
        # Close the stream and terminate PyAudio
        stream.stop_stream()
        stream.close()
        p.terminate()
    

    Be sure to have ffmpeg or avconv in your system's PATH, as pydub relies on them to process audio formats like MP3.

    In this updated example, we keep the stream_tts function as an asynchronous coroutine that receives chunks from edge_tts, but now each audio chunk is decoded asynchronously by process_audio before being put into the queue. The callback function continues to fetch raw audio data from the queue as before.

    Keep in mind that this is still a simplified setup and may need additional error handling, and configuration to work with different audio sources, formats, and system environments. It is also important to ensure that the rate at which the audio is decoded and enqueued matches or exceeds the rate at which the callback function consumes the audio data, otherwise, you might still experience underflow and resulting audio artifacts.

  7. User

    I hear nothing when i launch the script...

  8. Assistant

    It seems there might be an issue with the implementation of the processing and queuing of audio data from edge_tts. Here's a revised example that includes the necessary adjustments to properly decode the audio chunks and ensure they are played back through the stream.

    import asyncio
    import edge_tts
    import pyaudio
    import queue
    from pydub import AudioSegment
    from io import BytesIO
    
    TEXT = "Hello World!"
    VOICE = "en-GB-SoniaNeural"
    
    p = pyaudio.PyAudio()
    audio_queue = queue.Queue()
    stream_ended = False
    
    # PyAudio callback function
    def callback(in_data, frame_count, time_info, status):
        if stream_ended and audio_queue.empty():
            return (None, pyaudio.paComplete)
        
        data = b''
        while True:
            try:
                # Get as many chunks from the queue as necessary to fill the frame_count
                data += audio_queue.get_nowait()
                if len(data) >= frame_count * 2:  # Assuming a sample width of 2 bytes
                    break
            except queue.Empty:
                # If the queue is empty, we have all the data we can get right now
                break
        
        # If we didn't get enough data from the queue, pad with zeros (silence)
        data += b'\x00' * (frame_count * 2 - len(data))
        return (data, pyaudio.paContinue)
    
    async def process_audio(data: bytes):
        # Decode MP3 to raw audio using pydub
        audio_segment = AudioSegment.from_file(BytesIO(data), format="mp3")
        # Add the decoded raw audio to the queue
        audio_queue.put(audio_segment.raw_data)
    
    async def stream_tts(text: str, voice: str):
        global stream_ended
        communicate = edge_tts.Communicate(text, voice)
        
        # Stream TTS audio from edge_tts and put into queue as raw audio samples
        async for chunk in communicate.stream():
            if chunk["type"] == "audio":
                await process_audio(chunk["data"])
    
            if chunk.get("end", False):
                stream_ended = True
                return
    
    if __name__ == "__main__":
        # Set up a PyAudio stream with the parameters expected by the audio chunks from edge_tts
        stream = p.open(
            format=p.get_format_from_width(2),  # Assuming a sample width of 2 bytes
            channels=1,
            rate=16000,  # Update this rate to the correct value expected by your TTS service
            output=True,
            stream_callback=callback
        )
        # Start the PyAudio stream
        stream.start_stream()
    
        try:
            # Run the asyncio event loop
            asyncio.run(stream_tts(TEXT, VOICE))
        finally:
            # Wait for stream to complete playback
            while stream.is_active():
                p.sleep(0.1)
    
            # Close the stream and terminate PyAudio
            stream.stop_stream()
            stream.close()
            p.terminate()
    
    

    Make sure your audio rate is matched to what edge_tts is providing (for example, 16000Hz if that's what your TTS service uses).

    It's also important to note that real-time audio processing is sensitive to system performance. Python's GIL (Global Interpreter Lock) doesn't always play nicely with soft real-time systems like audio streaming.

    If you continue to experience issues with silence, there could be additional factors involved such as incorrect sample rate, an issue with the edge_tts service, or system-level audio configuration. Make sure to debug by logging messages to the console at various points of your code to trace where the issue might be.

  9. User

    You forgot to write stream... Come on!!!!

  10. Assistant

    I apologize for the oversight. Here's a corrected and more detailed version of the code. This code will handle the streaming of the audio chunks from edge_tts and write them to the PyAudio stream:

    import asyncio
    import edge_tts
    import pyaudio
    import queue
    from pydub import AudioSegment
    from io import BytesIO
    
    # Replace the following with your own text and voice configuration
    TEXT = "Hello World!"
    VOICE = "en-US-GuyNeural"
    
    # Initialize PyAudio and the queue that will hold the audio chunks
    p = pyaudio.PyAudio()
    audio_queue = queue.Queue()
    stream_ended = False
    
    # PyAudio stream callback function
    def pyaudio_callback(in_data, frame_count, time_info, status):
        frames = []
        while not audio_queue.empty():
            frames.append(audio_queue.get_nowait())
        if not frames and stream_ended:
            return (None, pyaudio.paComplete)
        data = b''.join(frames)
        return (data, pyaudio.paContinue)
    
    # Starts a PyAudio stream and adds chunks to the queue as they become available
    async def stream_tts(text, voice):
        global stream_ended
        tts_stream = edge_tts.Communicate(text, voice)
    
        async for chunk in tts_stream.stream():
            if chunk['type'] == 'audio':
                audio_segment = AudioSegment.from_file(BytesIO(chunk["data"]), format="mp3")
                audio_data = audio_segment.raw_data
                audio_queue.put(audio_data)
            if chunk.get('end', False):
                stream_ended = True
                break
    
    # Prepare the PyAudio configuration by getting the parameters from an audio segment
    test_segment = AudioSegment.silent(duration=1000)  # Create one second of silence
    pyaudio_format = p.get_format_from_width(test_segment.sample_width)
    channels = test_segment.channels
    frame_rate = test_segment.frame_rate
    
    # Create and start the PyAudio stream
    stream = p.open(format=pyaudio_format,
                    channels=channels,
                    rate=frame_rate,
                    output=True,
                    stream_callback=pyaudio_callback)
    
    stream.start_stream()
    
    # Run the coroutine that streams TTS and adds to the audio queue
    asyncio.run(stream_tts(TEXT, VOICE))
    
    # Keep the stream open until it's time to close it
    while stream.is_active():
        asyncio.sleep(0.1)
    
    # Close the stream and terminate PyAudio
    stream.stop_stream()
    stream.close()
    p.terminate()
    

    In this code snippet, the pyaudio_callback function is used as the PyAudio stream callback. It fetches all available audio chunks from the queue and joins them together before writing them to the stream. This callback function must return a tuple containing the audio data to play and a flag indicating whether to continue or stop the stream (pyaudio.paContinue or pyaudio.paComplete).

    The stream_tts function streams the TTS audio using the edge_tts library. It decodes each received MP3 chunk to raw audio data using pydub and then puts this raw audio data into a queue. The pyaudio_callback function is called by PyAudio in a separate thread to fetch and play audio data from this queue as it becomes available.

    This updated code uses a synchronous loop at the end to hold the program open while the PyAudio stream is active. Once the audio has finished playing, the stream is stopped and resources are cleaned up.

    Let me know if you have any more questions or if there's anything else I can assist you with!

  11. User

    pcm_frames = decoder.decode(io.BytesIO(mp3_data), frames_per_buffer)

    I can't install miniaudio for decoder, give me the list of alternatives with exemple

  12. Assistant

    The code snippet you've shared appears to use miniaudio, a library for audio decoding and playback which can't be installed on your system. There are alternative libraries you could use for decoding MP3 audio in Python, such as pydub, which internally can use ffmpeg or avconv for decoding.

    Since you're already familiar with pydub, I'll give you an example using pydub to handle MP3 decoding and the adjustment to the PyAudio callback function.

    import asyncio
    import edge_tts
    import pyaudio
    import queue
    from pydub import AudioSegment
    from io import BytesIO
    
    # Your TTS settings
    TEXT = "Hello World!"
    VOICE = "en-GB-SoniaNeural"
    
    # Initialize PyAudio and queue to hold audio chunks
    p = pyaudio.PyAudio()
    audio_queue = queue.Queue()
    stream_ended = False
    
    # PyAudio callback function
    def pyaudio_callback(in_data, frame_count, time_info, status):
        frames = []
        while True:
            try:
                chunk = audio_queue.get_nowait()
                frames.append(chunk)
            except queue.Empty:
                if stream_ended and len(frames) == 0:
                    return (None, pyaudio.paComplete)
                break
        data = b''.join(frames)
        remaining_frames = frame_count * 2 - len(data)  # Assumes 2 bytes per frame
        data += b'\x00' * remaining_frames  # Pad with silence if needed
        return (data, pyaudio.paContinue)
    
    # Async function to handle streaming TTS and decode MP3 chunks
    async def stream_tts(text: str, voice: str):
        global stream_ended
        communicate = edge_tts.Communicate(text, voice)
        # Stream and process the audio chunks from the TTS service
        async for chunk in communicate.stream():
            if chunk["type"] == "audio":
                # Decode the MP3 data chunk using pydub
                audio_segment = AudioSegment.from_file(BytesIO(chunk['data']), format="mp3")
                audio_data = audio_segment.raw_data
                audio_queue.put(audio_data)  # Put the raw audio data on the queue
            if chunk.get('end', False):
                stream_ended = True
                return
    
    # Open PyAudio stream
    stream = p.open(
        format=pyaudio.paInt16,  # Assumes 16-bit PCM audio
        channels=1,              # Mono audio
        rate=16000,              # Sample rate (replace with correct rate from TTS service)
        frames_per_buffer=1024,  # Buffer size (number of frames per buffer)
        output=True,
        stream_callback=pyaudio_callback
    )
    
    # Start the stream
    stream.start_stream()
    
    # Run the TTS coroutine to begin streaming and processing audio chunks
    asyncio.run(stream_tts(TEXT, VOICE))
    
    # Wait until stream finishes playing
    while stream.is_active():
        asyncio.sleep(0.1)
    
    # Clean up and close the stream and PyAudio
    stream.stop_stream()
    stream.close()
    p.terminate()
    

    In the callback, when no more chunks are available from the queue, and stream_ended is True, we return None and mark the stream as complete with pyaudio.paComplete. If not enough frames are collected to fulfill the buffer size requested by frame_count, the data is padded with silence (zeros).

    Please ensure that both the sample rate and frames_per_buffer in the PyAudio stream are set appropriately for the audio you are decoding. If these settings don't align with the TTS service output, you may still experience poor audio quality.

    Remember to install ffmpeg or avconv on your system and have them available in your system's PATH so pydub can use them for decoding the MP3 data.

    If you cannot install ffmpeg either, or you face other constraints, you might need a pure Python library (which is less common and might have performance limitations). Alternatively, you could handle the MP3 decoding separately in a service or process that you can call from Python if Python libraries do not meet your needs.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.