Hello there, I've been trying to implement fast audio streaming for 6 days now, but I…
- User
Hello there,
I've been trying to implement fast audio streaming for 6 days now, but I just can't do it. Indeed, without the stream method, the save function of edge-tts takes about 1-2s to generate depending the text, which is too long.
In my code bellow, the audio is indeed played instantly, regardless of text size, but there are artefacts between chunks, like tiny spaces.
Do you know how to do audio streaming correctly please ? Thank you!
import asyncio import edge_tts import pyaudio from io import BytesIO from pydub import AudioSegment
TEXT = "Hello World!" VOICE = "en-GB-SoniaNeural"
p = pyaudio.PyAudio()
async def stream_tts(text: str, voice: str) -> None:
We're assuming a certain format, channels, and rate
This will need to be dynamic based on the actual audio data from TTS
stream = p.open( format=pyaudio.paInt16, channels=1, rate=26000, output=True )
communicate = edge_tts.Communicate(text, voice)
Process and play audio chunks as they arrive
async for chunk in communicate.stream(): if chunk["type"] == "audio": try: audio_segment = AudioSegment.from_file(BytesIO(chunk["data"]), format="mp3")
# Write data to the stream directly without extra buffering stream.write(audio_segment.raw_data) # If this is the last chunk, break after playing if chunk.get('end', False): break except Exception as e: print("Error processing audio chunk:", e)Cleanup
stream.stop_stream() stream.close() p.terminate() if name == "main":
Run the asyncio event loop
asyncio.run(stream_tts(TEXT, VOICE))
- Assistant
It appears that you're running into an issue commonly known as 'audio glitching' which happens during streaming when audio chunks are played back without seamless transition between them. These artifacts could occur for a variety of reasons, such as differences in the duration of chunks, buffer underruns, or even the method of playback.
Let's go through some steps to potentially fix these issues:
-
Verify chunk sizes: Make sure that your TTS system sends the audio in uniform chunk sizes and that the sizes are conducive to smooth streaming.
-
Adjust buffer sizes: Sometimes the buffer size needs to be tuned for optimal performance. Too small of a buffer can lead to underruns, while too large of a buffer could introduce latency.
-
Remove Playback Delays: Any computation or processing between chunks can introduce delays. Make sure that the time it takes to process and play a chunk is consistent and minimal.
-
Use native formats if possible: Instead of converting chunks to a different format (like using
AudioSegment.from_file), try to work with the native format that your TTS service provides, if it's compatible with your playback system. -
Check pyaudio's callback mode: If possible, use pyaudio's callback mode instead of blocking mode to aid with smooth playback. Callback mode is designed to handle streaming uses cases.
Here's a simplified example of using callback mode in pyaudio:
import pyaudio # Example of how to set up a stream using callback mode def callback(in_data, frame_count, time_info, status): data = get_next_chunk_somehow() # You’ll need to handle getting the data return (data, pyaudio.paContinue) p = pyaudio.PyAudio() stream = p.open(format=pyaudio.paInt16, channels=1, rate=26000, output=True, stream_callback=callback) stream.start_stream() # Now the stream is running in the background calling the callback function while stream.is_active(): time.sleep(0.1) # Keep your main program running stream.stop_stream() stream.close() p.terminate()Please note that
get_next_chunk_somehow()is just a placeholder to indicate where you would retrieve the next chunk of data from the TTS service.To apply the callback mode to your code, you would need to reorganize your logic so that the callback function gets called with each new chunk from the streaming API.
Finally, here’s a key point about using
edge_tts: if you're noticing artifacts it could also be due to the TTS service itself or the way in which theedge_ttslibrary fetches and stitches together audio segments. Ensure that you are using the latest version of theedge_ttslibrary and check if there are any open issues or updates regarding streaming glitches. -
- User
make the callback function please
- Assistant
To integrate the callback function into your streaming logic, you will need to create a queue or buffer to hold your
edge_ttschunks, which the callback function will consume. Please note that the code below is a conceptual example; you might need to adjust it for your specific use case and ensure proper exception handling based on the behavior of theedge_ttslibrary.Here's an example of how you might set this up with a queue:
import asyncio import edge_tts import pyaudio import queue TEXT = "Hello World!" VOICE = "en-GB-SoniaNeural" p = pyaudio.PyAudio() audio_queue = queue.Queue() stream_ended = False def callback(in_data, frame_count, time_info, status): try: # Retrieve data from the queue chunk = audio_queue.get_nowait() except queue.Empty: # No data available if stream_ended: return (None, pyaudio.paComplete) else: # Return silence if no data data = b'\x00' * frame_count * 2 # assuming a sample width of 2 bytes return (data, pyaudio.paContinue) # Return data and continue playback return (chunk, pyaudio.paContinue) # Open stream using callback stream = p.open( format=pyaudio.paInt16, channels=1, rate=26000, output=True, stream_callback=callback ) async def stream_tts(text: str, voice: str): global stream_ended communicate = edge_tts.Communicate(text, voice) async for chunk in communicate.stream(): if chunk["type"] == "audio": # Put the received chunk into the queue audio_queue.put(chunk["data"]) if chunk.get('end', False): stream_ended = True break if __name__ == "__main__": # Start the stream stream.start_stream() # Run the asyncio event loop loop = asyncio.get_event_loop() loop.run_until_complete(stream_tts(TEXT, VOICE)) # Wait for stream to complete playback while stream.is_active(): loop.run_until_complete(asyncio.sleep(0.1)) # Close the stream and terminate PyAudio stream.stop_stream() stream.close() p.terminate()This code assumes you are receiving chunks of audio data as bytes (compatible with
pyaudio.paInt16format), which is then placed directly into the queue without conversion. When thecallbackfunction is called by PyAudio, it attempts to get an audio chunk from the queue without waiting (get_nowait), and plays it. If the queue is empty, the callback function generates silent frames to prevent underflows until new data arrives.Please keep in mind that this is a general example, designed to give you a starting point. You may need to adjust the sample rates, formats, and handle exceptions or other details based on the
edge_ttslibrary's behavior and the audio specifications you're working with. - User
The sound is robotic, you forgot AudioSegment.from_file(BytesIO(chunk[“data”]), format=“mp3”)
- Assistant
If you're receiving MP3 data from
edge_ttsand need to useAudioSegmentto convert it to raw audio data forpyaudio, you'll need to do so in a way that doesn't introduce delay or block the PyAudio callback. Since decoding can take an unpredictable amount of time, it's recommended to perform this task asynchronously outside the callback and then place the raw audio data into the queue.Here's a modification of the previous example that incorporates decoding of MP3 chunks using
AudioSegment. You would need to install thepydubandffmpegoravconvwhichpydubuses under the hood:import asyncio import edge_tts import pyaudio import queue from pydub import AudioSegment from io import BytesIO TEXT = "Hello World!" VOICE = "en-GB-SoniaNeural" p = pyaudio.PyAudio() audio_queue = queue.Queue() stream_ended = False # PyAudio callback function def callback(in_data, frame_count, time_info, status): try: chunk = audio_queue.get_nowait() except queue.Empty: if stream_ended: return (None, pyaudio.paComplete) else: data = b'\x00' * frame_count * 2 # assuming a sample width of 2 bytes return (data, pyaudio.paContinue) return (chunk, pyaudio.paContinue) # Function to process and enqueue audio async def process_audio(data: bytes): try: # Decode MP3 to raw audio using pydub audio_segment = AudioSegment.from_file( BytesIO(data), format="mp3" ) # Add the decoded raw audio to the queue audio_queue.put(audio_segment.raw_data) except Exception as e: print("Error processing audio chunk:", e) # Async coroutine for streaming and decoding TTS async def stream_tts(text: str, voice: str): global stream_ended communicate = edge_tts.Communicate(text, voice) async for chunk in communicate.stream(): if chunk["type"] == "audio": # Process and enqueue audio asynchronously await process_audio(chunk["data"]) if chunk.get('end', False): stream_ended = True break # Open the PyAudio stream with the specified parameters stream = p.open( format=pyaudio.paInt16, channels=1, rate=26000, output=True, stream_callback=callback ) if __name__ == "__main__": # Start the PyAudio stream stream.start_stream() # Run the asyncio event loop loop = asyncio.get_event_loop() loop.run_until_complete(stream_tts(TEXT, VOICE)) # Wait for stream to complete playback while stream.is_active(): loop.run_until_complete(asyncio.sleep(0.1)) # Close the stream and terminate PyAudio stream.stop_stream() stream.close() p.terminate()Be sure to have
ffmpegoravconvin your system's PATH, aspydubrelies on them to process audio formats like MP3.In this updated example, we keep the
stream_ttsfunction as an asynchronous coroutine that receives chunks fromedge_tts, but now each audio chunk is decoded asynchronously byprocess_audiobefore being put into the queue. The callback function continues to fetch raw audio data from the queue as before.Keep in mind that this is still a simplified setup and may need additional error handling, and configuration to work with different audio sources, formats, and system environments. It is also important to ensure that the rate at which the audio is decoded and enqueued matches or exceeds the rate at which the callback function consumes the audio data, otherwise, you might still experience underflow and resulting audio artifacts.
- User
I hear nothing when i launch the script...
- Assistant
It seems there might be an issue with the implementation of the processing and queuing of audio data from
edge_tts. Here's a revised example that includes the necessary adjustments to properly decode the audio chunks and ensure they are played back through the stream.import asyncio import edge_tts import pyaudio import queue from pydub import AudioSegment from io import BytesIO TEXT = "Hello World!" VOICE = "en-GB-SoniaNeural" p = pyaudio.PyAudio() audio_queue = queue.Queue() stream_ended = False # PyAudio callback function def callback(in_data, frame_count, time_info, status): if stream_ended and audio_queue.empty(): return (None, pyaudio.paComplete) data = b'' while True: try: # Get as many chunks from the queue as necessary to fill the frame_count data += audio_queue.get_nowait() if len(data) >= frame_count * 2: # Assuming a sample width of 2 bytes break except queue.Empty: # If the queue is empty, we have all the data we can get right now break # If we didn't get enough data from the queue, pad with zeros (silence) data += b'\x00' * (frame_count * 2 - len(data)) return (data, pyaudio.paContinue) async def process_audio(data: bytes): # Decode MP3 to raw audio using pydub audio_segment = AudioSegment.from_file(BytesIO(data), format="mp3") # Add the decoded raw audio to the queue audio_queue.put(audio_segment.raw_data) async def stream_tts(text: str, voice: str): global stream_ended communicate = edge_tts.Communicate(text, voice) # Stream TTS audio from edge_tts and put into queue as raw audio samples async for chunk in communicate.stream(): if chunk["type"] == "audio": await process_audio(chunk["data"]) if chunk.get("end", False): stream_ended = True return if __name__ == "__main__": # Set up a PyAudio stream with the parameters expected by the audio chunks from edge_tts stream = p.open( format=p.get_format_from_width(2), # Assuming a sample width of 2 bytes channels=1, rate=16000, # Update this rate to the correct value expected by your TTS service output=True, stream_callback=callback ) # Start the PyAudio stream stream.start_stream() try: # Run the asyncio event loop asyncio.run(stream_tts(TEXT, VOICE)) finally: # Wait for stream to complete playback while stream.is_active(): p.sleep(0.1) # Close the stream and terminate PyAudio stream.stop_stream() stream.close() p.terminate()Make sure your audio rate is matched to what
edge_ttsis providing (for example, 16000Hz if that's what your TTS service uses).It's also important to note that real-time audio processing is sensitive to system performance. Python's GIL (Global Interpreter Lock) doesn't always play nicely with soft real-time systems like audio streaming.
If you continue to experience issues with silence, there could be additional factors involved such as incorrect sample rate, an issue with the
edge_ttsservice, or system-level audio configuration. Make sure to debug by logging messages to the console at various points of your code to trace where the issue might be. - User
You forgot to write stream... Come on!!!!
- Assistant
I apologize for the oversight. Here's a corrected and more detailed version of the code. This code will handle the streaming of the audio chunks from
edge_ttsand write them to the PyAudio stream:import asyncio import edge_tts import pyaudio import queue from pydub import AudioSegment from io import BytesIO # Replace the following with your own text and voice configuration TEXT = "Hello World!" VOICE = "en-US-GuyNeural" # Initialize PyAudio and the queue that will hold the audio chunks p = pyaudio.PyAudio() audio_queue = queue.Queue() stream_ended = False # PyAudio stream callback function def pyaudio_callback(in_data, frame_count, time_info, status): frames = [] while not audio_queue.empty(): frames.append(audio_queue.get_nowait()) if not frames and stream_ended: return (None, pyaudio.paComplete) data = b''.join(frames) return (data, pyaudio.paContinue) # Starts a PyAudio stream and adds chunks to the queue as they become available async def stream_tts(text, voice): global stream_ended tts_stream = edge_tts.Communicate(text, voice) async for chunk in tts_stream.stream(): if chunk['type'] == 'audio': audio_segment = AudioSegment.from_file(BytesIO(chunk["data"]), format="mp3") audio_data = audio_segment.raw_data audio_queue.put(audio_data) if chunk.get('end', False): stream_ended = True break # Prepare the PyAudio configuration by getting the parameters from an audio segment test_segment = AudioSegment.silent(duration=1000) # Create one second of silence pyaudio_format = p.get_format_from_width(test_segment.sample_width) channels = test_segment.channels frame_rate = test_segment.frame_rate # Create and start the PyAudio stream stream = p.open(format=pyaudio_format, channels=channels, rate=frame_rate, output=True, stream_callback=pyaudio_callback) stream.start_stream() # Run the coroutine that streams TTS and adds to the audio queue asyncio.run(stream_tts(TEXT, VOICE)) # Keep the stream open until it's time to close it while stream.is_active(): asyncio.sleep(0.1) # Close the stream and terminate PyAudio stream.stop_stream() stream.close() p.terminate()In this code snippet, the
pyaudio_callbackfunction is used as the PyAudio stream callback. It fetches all available audio chunks from the queue and joins them together before writing them to the stream. This callback function must return a tuple containing the audio data to play and a flag indicating whether to continue or stop the stream (pyaudio.paContinueorpyaudio.paComplete).The
stream_ttsfunction streams the TTS audio using theedge_ttslibrary. It decodes each received MP3 chunk to raw audio data usingpyduband then puts this raw audio data into a queue. Thepyaudio_callbackfunction is called by PyAudio in a separate thread to fetch and play audio data from this queue as it becomes available.This updated code uses a synchronous loop at the end to hold the program open while the PyAudio stream is active. Once the audio has finished playing, the stream is stopped and resources are cleaned up.
Let me know if you have any more questions or if there's anything else I can assist you with!
- User
pcm_frames = decoder.decode(io.BytesIO(mp3_data), frames_per_buffer)
I can't install miniaudio for decoder, give me the list of alternatives with exemple
- Assistant
The code snippet you've shared appears to use
miniaudio, a library for audio decoding and playback which can't be installed on your system. There are alternative libraries you could use for decoding MP3 audio in Python, such aspydub, which internally can useffmpegoravconvfor decoding.Since you're already familiar with
pydub, I'll give you an example usingpydubto handle MP3 decoding and the adjustment to the PyAudio callback function.import asyncio import edge_tts import pyaudio import queue from pydub import AudioSegment from io import BytesIO # Your TTS settings TEXT = "Hello World!" VOICE = "en-GB-SoniaNeural" # Initialize PyAudio and queue to hold audio chunks p = pyaudio.PyAudio() audio_queue = queue.Queue() stream_ended = False # PyAudio callback function def pyaudio_callback(in_data, frame_count, time_info, status): frames = [] while True: try: chunk = audio_queue.get_nowait() frames.append(chunk) except queue.Empty: if stream_ended and len(frames) == 0: return (None, pyaudio.paComplete) break data = b''.join(frames) remaining_frames = frame_count * 2 - len(data) # Assumes 2 bytes per frame data += b'\x00' * remaining_frames # Pad with silence if needed return (data, pyaudio.paContinue) # Async function to handle streaming TTS and decode MP3 chunks async def stream_tts(text: str, voice: str): global stream_ended communicate = edge_tts.Communicate(text, voice) # Stream and process the audio chunks from the TTS service async for chunk in communicate.stream(): if chunk["type"] == "audio": # Decode the MP3 data chunk using pydub audio_segment = AudioSegment.from_file(BytesIO(chunk['data']), format="mp3") audio_data = audio_segment.raw_data audio_queue.put(audio_data) # Put the raw audio data on the queue if chunk.get('end', False): stream_ended = True return # Open PyAudio stream stream = p.open( format=pyaudio.paInt16, # Assumes 16-bit PCM audio channels=1, # Mono audio rate=16000, # Sample rate (replace with correct rate from TTS service) frames_per_buffer=1024, # Buffer size (number of frames per buffer) output=True, stream_callback=pyaudio_callback ) # Start the stream stream.start_stream() # Run the TTS coroutine to begin streaming and processing audio chunks asyncio.run(stream_tts(TEXT, VOICE)) # Wait until stream finishes playing while stream.is_active(): asyncio.sleep(0.1) # Clean up and close the stream and PyAudio stream.stop_stream() stream.close() p.terminate()In the callback, when no more chunks are available from the queue, and
stream_endedisTrue, we returnNoneand mark the stream as complete withpyaudio.paComplete. If not enough frames are collected to fulfill the buffer size requested byframe_count, the data is padded with silence (zeros).Please ensure that both the
sample rateandframes_per_bufferin the PyAudio stream are set appropriately for the audio you are decoding. If these settings don't align with the TTS service output, you may still experience poor audio quality.Remember to install
ffmpegoravconvon your system and have them available in your system's PATH sopydubcan use them for decoding the MP3 data.If you cannot install
ffmpegeither, or you face other constraints, you might need a pure Python library (which is less common and might have performance limitations). Alternatively, you could handle the MP3 decoding separately in a service or process that you can call from Python if Python libraries do not meet your needs.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.