export async function* streamToAsyncIterable(readableStream) { const reader =…
- User
export async function* streamToAsyncIterable(readableStream) { const reader = readableStream.getReader(); const decoder = new TextDecoder();
while (true) { const { value, done } = await reader.read(); if (done) break; yield decoder.decode(value); } }
Assistant
Your code is a great start and fundamentally correct for converting a
ReadableStreamofUint8Arraychunks into anAsyncIterableof strings.However, there's a common subtle issue when using
TextDecoderwith streaming data: handling multi-byte characters that might be split acrossvaluechunks.If, for example, a 3-byte UTF-8 character's bytes are split such that the first chunk contains the first two bytes and the next chunk contains the third byte,
decoder.decode(value)on the first chunk might produce an incorrect or replacement character, and the seconddecode()won't know about the preceding part.To correctly handle this, you need to tell
TextDecoderthat it's operating instreammode. Additionally, it's good practice to ensure thereader's lock is released and any final buffered characters are flushed.Here's the improved version:
/** * Converts a ReadableStream into an AsyncIterable of strings. * * This function is robust against multi-byte characters split across stream chunks * by using TextDecoder's `stream: true` option. It also ensures the reader's lock * is released when done or if an error occurs. * * @param {ReadableStream<Uint8Array>} readableStream The readable stream to convert. * @returns {AsyncIterable<string>} An async iterable that yields decoded string chunks. */ export async function* streamToAsyncIterable(readableStream) { const reader = readableStream.getReader(); // Use `stream: true` to tell the decoder to buffer incomplete multi-byte characters // and process them correctly with subsequent chunks. const decoder = new TextDecoder('utf-8', { stream: true }); try { while (true) { const { value, done } = await reader.read(); if (done) { // If the stream is done, we need to flush any remaining buffered characters // from the decoder. This can happen if the very last byte(s) completed a // multi-byte character. const finalChunk = decoder.decode(undefined, { stream: false }); // Pass undefined and stream:false to get the final result if (finalChunk.length > 0) { yield finalChunk; } break; // Exit the loop } // Decode the current chunk. The decoder will internally buffer if a character // is incomplete and combine with the next chunk. yield decoder.decode(value, { stream: true }); } } finally { // Ensure the reader's lock is released, regardless of how the generator exits // (e.g., normally, via break, return, or throw). reader.releaseLock(); } }Key Improvements:
new TextDecoder('utf-8', { stream: true }): This is the most crucial change. It tells theTextDecoderto expect more data and to correctly handle multi-byte characters that might be split acrossUint8Arraychunks.- Final
decoder.decode(undefined, { stream: false }): After the loop finishes (i.e.,doneis true), there might still be buffered bytes inside theTextDecoderif the very last character was split and its final bytes were part of thedonesignal (thoughreader.read()usually ensures whole chunks, it's safer to explicitly flush). Callingdecoder.decode()without arguments (or withundefinedas the first argument andstream: false) flushes any remaining buffered data. try...finallyBlock: This ensures thatreader.releaseLock()is called whether the loop completes naturally, is exited early (e.g., if the consumer of the async iterable callsbreakorreturn), or if an error occurs during reading. This prevents resource leaks.
This revised version is more robust for general-purpose
ReadableStreamto string conversion.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.