Skip to content
TheChatArchiveThe Chat Archive
gpt-4-1106-preview· openaiHugging Face Suzhen/CodeChat

Hello there, I've been trying to implement fast audio streaming for 6 days now, but I…

  1. User

    Hello there,

    I've been trying to implement fast audio streaming for 6 days now, but I just can't do it. Indeed, without the stream method, the save function of edge-tts takes about 1-2s to generate depending the text, which is too long.

    In my code bellow, the audio is indeed played instantly, regardless of text size, but there are artefacts between chunks, like tiny spaces.

    Do you know how to do audio streaming correctly please ? Thank you!

    import asyncio import edge_tts import pyaudio from io import BytesIO from pydub import AudioSegment

    TEXT = "Hello World!" VOICE = "en-GB-SoniaNeural"

    p = pyaudio.PyAudio()

    async def stream_tts(text: str, voice: str) -> None:

    We're assuming a certain format, channels, and rate # This will need to be dynamic based on the actual audio data from TTS stream = p.open( format=pyaudio.paInt16, channels=1, rate=26000, output=True ) communicate = edge_tts.Communicate(text, voice) # Process and play audio chunks as they arrive async for chunk in communicate.stream(): if chunk["type"] == "audio": try: audio_segment = AudioSegment.from_file(BytesIO(chunk["data"]), format="mp3") # Write data to the stream directly without extra buffering stream.write(audio_segment.raw_data) # If this is the last chunk, break after playing if chunk.get('end', False): break except Exception as e: print("Error processing audio chunk:", e) # Cleanup stream.stop_stream() stream.close() p.terminate()

    if name == "main":

    Run the asyncio event loop

    asyncio.run(stream_tts(TEXT, VOICE))

  2. Assistant

    Your code already takes the right approach by streaming the audio and playing it as it becomes available, which is ideal for reducing initial latency. However, if you're encountering artifacts or "tiny spaces" between chunks, it generally implies that there's a discontinuity in the audio stream.

    While streaming, you'll have to ensure that the transition from one audio chunk to the next is seamless. With your current code, there could be a few places to check:

    1. Chunk Boundaries: Ensure each chunk ends and starts at appropriate boundaries where the audio waveform is continuous. This is more of an issue with the TTS engine and how it streams the data, but you have limited control here unless you buffer and process to ensure continuity.

    2. Buffer Underflows: Your audio playback could be running out of data to play if you're not writing the next chunk fast enough, or if the system is too busy to process the next chunk immediately. This will cause gaps in playback.

    3. Thread Scheduling: Python's GIL (Global Interpreter Lock) can sometimes cause issues if your processing is too heavy on one thread. Consider offloading the TTS streaming and the audio playback code onto separate threads.

    4. Streaming Format: Make sure that the data you receive is formatted correctly and matches what the audio playback expects, including sample rates, bit depth, and channels.

    5. Network Latency: Since TTS often requires a network request, any latency or variability in network speed could also cause artifacts. You can't do much about this except ensure a stable and fast internet connection.

    Considering all the above possibilities, you might want to adjust your code sample as follows:

    • Pre-buffer the first few chunks to ensure a smooth start.
    • Investigate the possibility of crossfade or smooth transitions between chunks.
    • Monitor your system resources to ensure they are not causing the playback to delay.

    Given that your code uses from_file() which is targeted at file playback, you may also be unintentionally introducing a delay there as the system initializes for each file playback. One way to improve the streaming quality is to remove any intermediate steps (such as creation of BytesIO objects or using AudioSegment.from_file for each chunk) that are not necessary. Open the PyAudio stream once and write the raw MP3 bytes directly from the chunk to PyAudio (if PyAudio can handle the encoded MP3, which typically it can't, as it expects PCM data).

    It appears you're using pydub to convert from MP3 to raw audio data that PyAudio can play, which could be introducing gaps. Pydub may introduce a tiny amount of silence at the start of each MP3 file it processes, so doing this per-chunk would result in those artifacts. If possible, decode the MP3 to PCM before streaming and see if this eliminates the artifacts. You can do this with a library capable of decoding MP3 on-the-fly, like miniaudio.

    Here's an example of how you might decode MP3 directly to PCM without saving to a temporary file, which should improve performance:

    import edge_tts
    import asyncio
    import pyaudio
    import miniaudio
    import io
    
    async def stream_tts(text: str, voice: str):
        p = pyaudio.PyAudio()
        
        # Placeholder for the sample rate, should be determined from the TTS service
        sample_rate = 24000  
        
        # Placeholder for the frame count
        frames_per_buffer = 1024  
        
        # Open audio stream
        stream = p.open(format=p.get_format_from_width(2),  #assuming 16-bit PCM
                        channels=1,
                        rate=sample_rate,
                        output=True,
                        frames_per_buffer=frames_per_buffer)
        
        communicate = edge_tts.Communicate(text, voice)
        
        decoder = miniaudio.Decoder()  # Assumed to use an MP3 decoder here
    
        async for chunk in communicate.stream():
            if chunk["type"] == "audio":
                
                # Decode MP3 data directly
                mp3_data = chunk["data"]
                pcm_frames = decoder.decode(io.BytesIO(mp3_data), frames_per_buffer)
                
                # Now pcm_frames is a byte array containing raw PCM frame data
                stream.write(pcm_frames)
                
                if chunk.get('end', False):
                    break
        
        # Cleanup
        stream.stop_stream()
        stream.close()
        p.terminate()
    
    if __name__ == "__main__":
        asyncio.run(stream_tts(TEXT, VOICE))
    

    Please install miniaudio if you haven't already:

    pip install miniaudio
    

    This code hasn't been tested with edge_tts.Communicate and assumes miniaudio can be correctly used like this (check the miniaudio documentation for specifics on working with a stream of MP3 data).

    Finally, since you're dealing with real-time streaming, a lot will depend on ensuring that the processing pipeline stays as lightweight and fast as possible. Measure execution times and identify any bottlenecks if issues persist.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.