import { Writable } from "stream"; import fs from "fs"; class FileWriteStream extends…
- User
import { Writable } from "stream"; import fs from "fs";
class FileWriteStream extends Writable { constructor({ highWaterMark, fileName }) { super({ highWaterMark });
this.fileName = fileName; this.fd = null; this.chunks = []; this.chunksSize = 0; this.writesCount = 0; this.errorHappened = false;}
// This will run after the constructor, and it will put off calling all the other // methods until we call the callback function _construct(callback) { fs.open(this.fileName, "w", (err, fd) => { if (err) { // so if we call the callback with an argument, it means that we have an error // and we should not proceed callback(err); } else { this.fd = fd; // no argument means it was successful callback(); } }); }
_write(chunk, encoding, callback) { this.chunks.push(chunk); this.chunksSize += chunk.length;
if (this.chunksSize > this.writableHighWaterMark) { fs.write(this.fd, Buffer.concat(this.chunks), (err) => { if (err) { return callback(new Error("you make wrong")); } this.chunks = []; this.chunksSize = 0; ++this.writesCount; callback(); }); } else { // when we're done, we should call the callback function callback(); }}
// My main question: if i comment _destroy method out, I would get: /* Number of writes: 0 hi --- STREAM ERROR EVENT CAUGHT --- EBADF: bad file descriptor, close / // And if I uncomment it out, I would get: /--- STREAM ERROR EVENT CAUGHT --- you make wrong */
// _destroy(error, callback) { // console.log("Number of writes:", this.writesCount); // if (this.fd) { // fs.close(this.fd, (err) => { // callback(err || error); // }); // } else { // callback(error); // } // } } const stream = new FileWriteStream({ highWaterMark: 20, fileName: "test.txt", });
stream.on("error", (error) => { console.log("--- STREAM ERROR EVENT CAUGHT ---", error.message); });
stream.write("hello world"); // 11 bytes, under highWaterMark
setTimeout(() => { fs.closeSync(stream.fd); }, 100);
setTimeout(() => { stream.write("this write will fail"); }, 200);
Assistant
You've hit upon a subtle but important aspect of Node.js stream error handling, especially when interacting with underlying system resources like file descriptors.
The core issue is that you are externally closing the file descriptor (
stream.fd) usingfs.closeSync(stream.fd)without the stream instance being aware of it. This puts the stream into an inconsistent state.Let's break down why you get different errors:
Understanding the
WritableStream Lifecycle and Error Handling_construct: Used for asynchronous initialization (like opening a file). If it callscallback(err), the stream immediately emits an 'error' event and_destroyis called._write: Handles incoming chunks. It must callcallback()when it's done processing a chunk (orcallback(err)if an error occurred during processing). Ifcallback(err)is called, the stream emits an 'error' event and triggers_destroy._destroy: This method is called by the stream internally when it needs to shut down. This happens in several scenarios:stream.end()is called, and all data is flushed.stream.destroy()is called explicitly.- An internal error occurs within the stream (e.g.,
_writecallscallback(err)). - It receives an
errorargument that represents the error that caused the destruction. - Its responsibility is to clean up resources (like closing file descriptors). It must call
callback(err)to signal that destruction is complete (or if new errors occurred during cleanup).
Scenario 1:
_destroycommented out// _destroy(error, callback) { ... } // COMMENTED OUTstream.write("hello world"):_writeis called.chunksgets "hello world",chunksSizeis 11.11 > 20is false.callback()is immediately called. The data ("hello world") is still buffered within the stream'sthis.chunksarray, not yet written to the file.this.fdis open.writesCountis 0.
setTimeout(() => { fs.closeSync(stream.fd); }, 100);:- At ~100ms,
fs.closeSyncis called. This forcefully closes the underlying file descriptor at the operating system level. - The
FileWriteStreaminstance itself does not know that itsthis.fdis now invalid.
- At ~100ms,
setTimeout(() => { stream.write("this write will fail"); }, 200);:_writeis called again.this.chunksnow contains "hello world" + "this write will fail".chunksSizebecomes 11 + 20 = 31.31 > 20is true.fs.write(this.fd, Buffer.concat(this.chunks), (err) => { ... });is called.- Problem:
this.fdis an invalid file descriptor (closed at 100ms). Whenfs.writeattempts to write to an invalid file descriptor, the operating system returns anEBADF(Bad File Descriptor) error. - The
fs.writecallback receives thisEBADFerror. - Your
if (err)block in_writeis entered. You then callreturn callback(new Error("you make wrong"));. - Why
EBADFand not "you make wrong"? When_destroyis missing, the stream's default behavior for an unrecoverable system-level I/O error likeEBADFoccurring during an internalfs.writecall tends to take precedence. The stream might immediately emit this fundamental system error and terminate. Your explicitcallback(new Error("you make wrong"))is still called, but the stream's core might prioritize theEBADFerror from the low-levelfs.writeoperation itself, leading toEBADFbeing the error emitted. Theconsole.log("Number of writes: 0")appears because your_destroymethod (which contains that log) is NOT called. The outputNumber of writes: 0in your question for this scenario is thus contradictory if_destroyis truly commented out. Assuming your output reflects what you see, it would mean_destroywas somehow involved or you have other logging. However, based on theEBADFerror, the core reason is writing to a closed FD.
Scenario 2:
_destroyuncommented_destroy(error, callback) { console.log("Number of writes:", this.writesCount); if (this.fd) { fs.close(this.fd, (err) => { callback(err || error); }); } else { callback(error); } }- Steps 1 & 2 (
stream.write("hello world")andfs.closeSync(stream.fd)) are the same as above. setTimeout(() => { stream.write("this write will fail"); }, 200);:_writeis called.chunksSizebecomes 31.fs.writeis called with the invalidthis.fd. It receivesEBADFaserr.- Your
if (err)block in_writeis entered. - You explicitly call
return callback(new Error("you make wrong"));. - Crucially: Because you explicitly called
_write'scallbackwithnew Error("you make wrong"), the Node.jsWritablestream machinery now understands that your implementation has signalled this specific error. - The stream then proceeds to:
- Emit an
'error'event withnew Error("you make wrong"). - Call
_destroy(new Error("you make wrong"), destroyCallback).
- Emit an
- Inside
_destroy:console.log("Number of writes:", this.writesCount);is executed (outputtingNumber of writes: 0because no successfulfs.writeoccurred).if (this.fd)is true.fs.close(this.fd, (err) => { ... });is called on the already invalidthis.fd. Thisfs.closemight itself fail withEBADFor succeed gracefully depending on the OS.- The
_destroy'scallbackis called witherr || error. Sinceerrorisnew Error("you make wrong")(the error passed to_destroyby the stream), andfs.closemight not produce a new, different error (or itsEBADFis less significant than the explicitly passed error),new Error("you make wrong")is what gets passed as the final destruction error.
- Result: The stream's
'error'event listener catchesnew Error("you make wrong").
Summary of Differences
- No
_destroy: The stream's default internal error handling for a fatal underlying system error (EBADFfromfs.write) takes over, often emitting the raw system error. Your custom error from_writemight be ignored or superseded. - With
_destroy: Your_writemethod explicitly signals an error ("you make wrong") to the stream's callback. This error is then properly passed to and handled by your_destroymethod, which ensures that specific error is the one ultimately propagated and emitted by the stream. Theconsole.loginside_destroyconfirms that_destroyis being called in this scenario.
The Correct Way to Handle File Descriptors in Custom Writable Streams
You should never manually call
fs.closeSync(stream.fd). The stream's_destroymethod is designed to be the single point of truth for closing its resources.To demonstrate, let's modify your code to work correctly and avoid this external interference:
import { Writable } from "stream"; import fs from "fs"; class FileWriteStream extends Writable { constructor({ highWaterMark, fileName }) { super({ highWaterMark }); this.fileName = fileName; this.fd = null; this.chunks = []; this.chunksSize = 0; this.writesCount = 0; // this.errorHappened = false; // Not used, can remove } _construct(callback) { fs.open(this.fileName, "w", (err, fd) => { if (err) { callback(err); } else { this.fd = fd; callback(); } }); } _write(chunk, encoding, callback) { this.chunks.push(chunk); this.chunksSize += chunk.length; if (this.chunksSize >= this.writableHighWaterMark) { // Use >= for full chunks too fs.write(this.fd, Buffer.concat(this.chunks), (err) => { if (err) { // It's good to pass the actual system error here, or wrap it return callback(err); // Pass the original error } this.chunks = []; this.chunksSize = 0; ++this.writesCount; callback(); }); } else { callback(); } } _destroy(error, callback) { console.log("Number of writes:", this.writesCount); // Only attempt to close if fd was actually opened and is not null if (this.fd) { fs.close(this.fd, (err) => { // Pass the original error if no new error from close, otherwise pass new error callback(err || error); }); } else { // If fd was never opened, just pass the original error callback(error); } } // Add _final to ensure remaining chunks are written when stream ends _final(callback) { if (this.chunksSize > 0) { fs.write(this.fd, Buffer.concat(this.chunks), (err) => { if (err) { return callback(err); } this.chunks = []; this.chunksSize = 0; ++this.writesCount; callback(); }); } else { callback(); } } } const stream = new FileWriteStream({ highWaterMark: 20, fileName: "test.txt", }); stream.on("error", (error) => { console.error("--- STREAM ERROR EVENT CAUGHT ---", error.message); }); // Write less than HWM, data is buffered stream.write("hello world"); // 11 bytes // Write more than HWM, triggers a flush stream.write("this is a test string for flushing"); // 35 bytes + 11 buffered = 46 bytes, > 20 HWM, so first write happens here stream.end("final bit of data"); // Will call _final after all previous writes are handled // If you want to simulate an error, you'd do it differently: // e.g., stream.destroy(new Error("Manual destruction")); // Do NOT do this, it breaks the stream's internal state management // setTimeout(() => { // fs.closeSync(stream.fd); // }, 100); // This write would now work as expected if you don't externally close the FD // setTimeout(() => { // stream.write("this write will now work"); // }, 200); // You can verify the file content after a short delay setTimeout(() => { fs.readFile('test.txt', 'utf8', (err, data) => { if (err) { console.error("Error reading file:", err); return; } console.log("File content:", data); }); }, 500);In the corrected code:
- I've added a
_finalmethod, which is crucial forWritablestreams to ensure any remaining buffered data is written whenstream.end()is called. - I've removed the external
fs.closeSync(stream.fd), which was the root cause of your problem. - The
_writemethod passes the actualerrfromfs.writeif an error occurs. - The
_destroymethod handles closing the file gracefully.
Now, if you were to introduce an actual error (e.g., trying to write to a non-existent path in
_construct, or causing anfs.writeto fail for legitimate reasons), the stream's error handling and_destroywould work as expected, consistently reporting the error.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.