Hello there, I've been trying to implement fast audio streaming for 6 days now, but I…
- User
Hello there,
I've been trying to implement fast audio streaming for 6 days now, but I just can't do it. Indeed, without the stream method, the save function of edge-tts takes about 1-2s to generate depending the text, which is too long.
In my code bellow, the audio is indeed played instantly, regardless of text size, but there are artefacts between chunks, like tiny spaces.
Do you know how to do audio streaming correctly please ? Thank you!
import asyncio import edge_tts import pyaudio from io import BytesIO from pydub import AudioSegment
TEXT = "Hello World!" VOICE = "en-GB-SoniaNeural"
p = pyaudio.PyAudio()
async def stream_tts(text: str, voice: str) -> None:
We're assuming a certain format, channels, and rate # This will need to be dynamic based on the actual audio data from TTS stream = p.open( format=pyaudio.paInt16, channels=1, rate=26000, output=True ) communicate = edge_tts.Communicate(text, voice) # Process and play audio chunks as they arrive async for chunk in communicate.stream(): if chunk["type"] == "audio": try: audio_segment = AudioSegment.from_file(BytesIO(chunk["data"]), format="mp3") # Write data to the stream directly without extra buffering stream.write(audio_segment.raw_data) # If this is the last chunk, break after playing if chunk.get('end', False): break except Exception as e: print("Error processing audio chunk:", e) # Cleanup stream.stop_stream() stream.close() p.terminate()
if name == "main":
Run the asyncio event loop
asyncio.run(stream_tts(TEXT, VOICE))
- Assistant
Your code already takes the right approach by streaming the audio and playing it as it becomes available, which is ideal for reducing initial latency. However, if you're encountering artifacts or "tiny spaces" between chunks, it generally implies that there's a discontinuity in the audio stream.
While streaming, you'll have to ensure that the transition from one audio chunk to the next is seamless. With your current code, there could be a few places to check:
-
Chunk Boundaries: Ensure each chunk ends and starts at appropriate boundaries where the audio waveform is continuous. This is more of an issue with the TTS engine and how it streams the data, but you have limited control here unless you buffer and process to ensure continuity.
-
Buffer Underflows: Your audio playback could be running out of data to play if you're not writing the next chunk fast enough, or if the system is too busy to process the next chunk immediately. This will cause gaps in playback.
-
Thread Scheduling: Python's GIL (Global Interpreter Lock) can sometimes cause issues if your processing is too heavy on one thread. Consider offloading the TTS streaming and the audio playback code onto separate threads.
-
Streaming Format: Make sure that the data you receive is formatted correctly and matches what the audio playback expects, including sample rates, bit depth, and channels.
-
Network Latency: Since TTS often requires a network request, any latency or variability in network speed could also cause artifacts. You can't do much about this except ensure a stable and fast internet connection.
Considering all the above possibilities, you might want to adjust your code sample as follows:
- Pre-buffer the first few chunks to ensure a smooth start.
- Investigate the possibility of crossfade or smooth transitions between chunks.
- Monitor your system resources to ensure they are not causing the playback to delay.
Given that your code uses
from_file()which is targeted at file playback, you may also be unintentionally introducing a delay there as the system initializes for each file playback. One way to improve the streaming quality is to remove any intermediate steps (such as creation ofBytesIOobjects or usingAudioSegment.from_filefor each chunk) that are not necessary. Open the PyAudio stream once and write the raw MP3 bytes directly from the chunk to PyAudio (if PyAudio can handle the encoded MP3, which typically it can't, as it expects PCM data).It appears you're using pydub to convert from MP3 to raw audio data that PyAudio can play, which could be introducing gaps. Pydub may introduce a tiny amount of silence at the start of each MP3 file it processes, so doing this per-chunk would result in those artifacts. If possible, decode the MP3 to PCM before streaming and see if this eliminates the artifacts. You can do this with a library capable of decoding MP3 on-the-fly, like
miniaudio.Here's an example of how you might decode MP3 directly to PCM without saving to a temporary file, which should improve performance:
import edge_tts import asyncio import pyaudio import miniaudio import io async def stream_tts(text: str, voice: str): p = pyaudio.PyAudio() # Placeholder for the sample rate, should be determined from the TTS service sample_rate = 24000 # Placeholder for the frame count frames_per_buffer = 1024 # Open audio stream stream = p.open(format=p.get_format_from_width(2), #assuming 16-bit PCM channels=1, rate=sample_rate, output=True, frames_per_buffer=frames_per_buffer) communicate = edge_tts.Communicate(text, voice) decoder = miniaudio.Decoder() # Assumed to use an MP3 decoder here async for chunk in communicate.stream(): if chunk["type"] == "audio": # Decode MP3 data directly mp3_data = chunk["data"] pcm_frames = decoder.decode(io.BytesIO(mp3_data), frames_per_buffer) # Now pcm_frames is a byte array containing raw PCM frame data stream.write(pcm_frames) if chunk.get('end', False): break # Cleanup stream.stop_stream() stream.close() p.terminate() if __name__ == "__main__": asyncio.run(stream_tts(TEXT, VOICE))Please install
miniaudioif you haven't already:pip install miniaudioThis code hasn't been tested with
edge_tts.Communicateand assumesminiaudiocan be correctly used like this (check theminiaudiodocumentation for specifics on working with a stream of MP3 data).Finally, since you're dealing with real-time streaming, a lot will depend on ensuring that the processing pipeline stays as lightweight and fast as possible. Measure execution times and identify any bottlenecks if issues persist.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.