I have compression and decompression function for ZSTD. Somehow after first compressing…
- User
I have compression and decompression function for ZSTD. Somehow after first compressing 7200 bytes of input and then decompressing them returned decompression vector is a bit more than 1MB in size. Both in/out buffers have 1MB size.
void ZipMiniStream::ZipMiniCompressZstd( std::function<std::pair<uint64_t, bool>(uint8_t * dst, uint64_t size)> stream_in, std::function<void(uint8_t const * dst, uint64_t size)> stream_out, uint64_t compression_level) { ZSTD_CStream * ctx_ = ZSTD_createCStream(); ASSERT(ctx_);
ZSTD_CHECK(ZSTD_initCStream(ctx_, compression_level)); uint64_t readed_size = 0, ret = 0; bool finished = false; ZSTD_inBuffer in{.src = s_inbuf.data(), .size = s_inbuf.size(), .pos = 0}; ZSTD_outBuffer out{.dst = s_outbuf.data(), .size = s_outbuf.size(), .pos = 0}; while (true) { if (s_inbuf.size() > in.size) { std::tie(readed_size, finished) = stream_in(s_inbuf.data() + in.size, s_inbuf.size() - in.size); in.size += readed_size; ASSERT(readed_size > 0 || finished); } ZSTD_CHECK((ret = ZSTD_compressStream2( ctx_, &out, &in, finished ? ZSTD_e_end : ZSTD_e_continue))); if (in.pos > 0) { std::memmove(s_inbuf.data(), s_inbuf.data() + in.pos, in.size - in.pos); in.size -= in.pos; in.pos = 0; } if (out.pos > 0) { stream_out(s_outbuf.data(), out.pos); out.pos = 0; } if (finished && ret == 0) { ASSERT(in.size == 0); break; } } ZSTD_CHECK(ZSTD_freeCStream(ctx_));}
void ZipMiniStream::ZipMiniDeCompressZstd( std::function<std::pair<uint64_t, bool>(uint8_t * dst, uint64_t size)> stream_in, std::function<void(uint8_t const * dst, uint64_t size)> stream_out) {
ZSTD_DStream * ctx_ = ZSTD_createDStream(); ASSERT(ctx_); ZSTD_CHECK(ZSTD_initDStream(ctx_)); uint64_t readed_size = 0, ret = 0; bool finished = false; ZSTD_inBuffer in{.src = s_inbuf.data(), .size = 0, .pos = 0}; ZSTD_outBuffer out{.dst = s_outbuf.data(), .size = s_outbuf.size(), .pos = 0}; while (true) { if (s_inbuf.size() > in.size) { std::tie(readed_size, finished) = stream_in(s_inbuf.data() + in.size, s_inbuf.size() - in.size); in.size += readed_size; ASSERT(readed_size > 0 || finished); } DUMP(in.size); DUMP(in.pos); DUMP(out.size); DUMP(out.pos); ZSTD_CHECK((ret = ZSTD_decompressStream(ctx_, &out, &in))); DUMP(ret); DUMP(in.size); DUMP(in.pos); DUMP(out.size); DUMP(out.pos); if (in.pos > 0) { std::memmove(s_inbuf.data(), s_inbuf.data() + in.pos, in.size - in.pos); in.size -= in.pos; in.pos = 0; } if (out.pos > 0) { stream_out(s_outbuf.data(), out.pos); out.pos = 0; } if (finished && ret == 0) { ASSERT(in.size == 0); break; } } ZSTD_CHECK(ZSTD_freeDStream(ctx_));}
- Assistant
The issue you're encountering—where compressing 7200 bytes of input and then decompressing results in a decompressed vector that's unexpectedly large (~1MB)—is likely due to how the input buffer sizes are being managed in both your compression and decompression functions. Specifically, the problem arises from incorrectly setting the
ZSTD_inBuffer.sizeto the buffer's capacity rather than the actual number of bytes read.Detailed Explanation
1. Understanding
ZSTD_inBufferandZSTD_outBufferZstandard (ZSTD) uses
ZSTD_inBufferandZSTD_outBufferstructures to manage input and output data for compression and decompression streams. These structures have the following fields:src/dst: Pointers to the input/output data.size: The number of bytes available insrcordstrespectively.pos: The current position in the buffer that the engine has processed or filled up to.
It's crucial that the
sizefield accurately reflects the actual amount of data present in the buffer, not just the buffer's total capacity.2. Issues in the Compression Function (
ZipMiniCompressZstd)In your compression function, you initialize the input buffer as follows:
ZSTD_inBuffer in{.src = s_inbuf.data(), .size = s_inbuf.size(), .pos = 0};Here,
s_inbuf.size()likely represents the capacity of the buffer (e.g., 1MB), not the actual number of bytes read (7200 bytes). By setting.sizeto the buffer's capacity, the compressor assumes that the entire buffer is filled with valid data, which isn't the case. This leads to undefined behavior where the compressor might process uninitialized or irrelevant data beyond the 7200 bytes you intended, resulting in incorrect compression and subsequently an inflated decompressed size.3. Issues in the Decompression Function (
ZipMiniDeCompressZstd)Similarly, in your decompression function, you initialize the input buffer with:
ZSTD_inBuffer in{.src = s_inbuf.data(), .size = 0, .pos = 0};While
in.sizestarts at 0, ensure that each time you read new data into the buffer,in.sizeaccurately reflects the number of bytes actually read, not the buffer's total capacity. Mismanagement here can also cause the decompressor to read beyond the valid data, leading to the unexpected large output.4. Recommended Fix
To resolve this issue, you need to ensure that the
.sizefield ofZSTD_inBufferaccurately represents the actual number of bytes read into the buffer, not the buffer's total capacity. Here's how you can adjust both functions:Adjusting the Compression Function
void ZipMiniStream::ZipMiniCompressZstd( std::function<std::pair<uint64_t, bool>(uint8_t * dst, uint64_t size)> stream_in, std::function<void(uint8_t const * dst, uint64_t size)> stream_out, uint64_t compression_level) { ZSTD_CStream *ctx_ = ZSTD_createCStream(); ASSERT(ctx_); ZSTD_CHECK(ZSTD_initCStream(ctx_, compression_level)); uint64_t read_size = 0, ret = 0; bool finished = false; // Initialize in.size to 0, as no data has been read yet ZSTD_inBuffer in{.src = s_inbuf.data(), .size = 0, .pos = 0}; ZSTD_outBuffer out{.dst = s_outbuf.data(), .size = s_outbuf.size(), .pos = 0}; while (true) { if (in.size < s_inbuf.size()) { // Check if there is space to read more data std::tie(read_size, finished) = stream_in(s_inbuf.data() + in.size, s_inbuf.size() - in.size); in.size += read_size; ASSERT(read_size > 0 || finished); } ret = ZSTD_compressStream2( ctx_, &out, &in, finished ? ZSTD_e_end : ZSTD_e_continue); ZSTD_CHECK(ret); if (in.pos > 0) { std::memmove(s_inbuf.data(), s_inbuf.data() + in.pos, in.size - in.pos); in.size -= in.pos; in.pos = 0; } if (out.pos > 0) { stream_out(s_outbuf.data(), out.pos); out.pos = 0; } if (finished && ret == 0) { ASSERT(in.size == 0); break; } } ZSTD_check(ZSTD_freeCStream(ctx_)); }Adjusting the Decompression Function
void ZipMiniStream::ZipMiniDeCompressZstd( std::function<std::pair<uint64_t, bool>(uint8_t * dst, uint64_t size)> stream_in, std::function<void(uint8_t const * dst, uint64_t size)> stream_out) { ZSTD_DStream *ctx_ = ZSTD_createDStream(); ASSERT(ctx_); ZSTD_CHECK(ZSTD_initDStream(ctx_)); uint64_t read_size = 0, ret = 0; bool finished = false; // Initialize in.size to 0, as no data has been read yet ZSTD_inBuffer in{.src = s_inbuf.data(), .size = 0, .pos = 0}; ZSTD_outBuffer out{.dst = s_outbuf.data(), .size = s_outbuf.size(), .pos = 0}; while (true) { if (in.size < s_inbuf.size()) { // Check if there is space to read more data std::tie(read_size, finished) = stream_in(s_inbuf.data() + in.size, s_inbuf.size() - in.size); in.size += read_size; ASSERT(read_size > 0 || finished); } ret = ZSTD_decompressStream(ctx_, &out, &in); ZSTD_CHECK(ret); if (in.pos > 0) { std::memmove(s_inbuf.data(), s_inbuf.data() + in.pos, in.size - in.pos); in.size -= in.pos; in.pos = 0; } if (out.pos > 0) { stream_out(s_outbuf.data(), out.pos); out.pos = 0; } if (finished && ret == 0) { ASSERT(in.size == 0); break; } } ZSTD_CHECK(ZSTD_freeDStream(ctx_)); }Key Changes:
-
Initialize
in.sizeto 0: Start with no data read, allowing the first read to accurately set the number of bytes available. -
Update
in.sizeBased on Actual Data Read: After each read operation, incrementin.sizeby the number of bytes actually read (read_size), not the buffer's total capacity. -
Check Available Space Before Reading More Data: Ensure there's space in the buffer before attempting to read more data by comparing
in.sizewiths_inbuf.size(). -
Handle
in.posCorrectly: After processing, move any remaining data to the beginning of the buffer and reset positions appropriately.
5. Additional Recommendations
-
Validate Buffer Reads: Always ensure that the number of bytes read does not exceed the buffer's capacity to prevent buffer overflows.
-
Error Handling: Ensure that all ZSTD function calls are checked for errors, and handle them gracefully.
-
Buffer Sizes: Re-evaluate whether a 1MB buffer is necessary for both input and output. Depending on your use case, smaller buffers might be more efficient and reduce memory usage.
-
Testing: After making these changes, thoroughly test with various input sizes to ensure that compression and decompression behave as expected.
By ensuring that
ZSTD_inBuffer.sizeaccurately reflects the actual number of bytes of data rather than the buffer's capacity, you should see correct compression and decompression behavior, eliminating the issue of unexpectedly large decompressed data.
References:
Note: Always ensure that buffers are managed correctly when working with low-level APIs like ZSTD to prevent data corruption and unexpected behavior.
Summary: The compressed data likely includes unintended bytes because the input buffer’s size is set to the entire buffer capacity (1 MB) instead of the actual data size (7200 bytes). To fix this, ensure that
ZSTD_inBuffer.sizeis set to the number of bytes read (7200), not the buffer’s total size, in both compression and decompression functions.