Skip to content
TheChatArchiveThe Chat Archive
o1-mini-2024-09-12· OpenAIAi2 WildChat-4.8M (AI2)

I have compression and decompression function for ZSTD. Somehow after first compressing…

  1. User

    I have compression and decompression function for ZSTD. Somehow after first compressing 7200 bytes of input and then decompressing them returned decompression vector is a bit more than 1MB in size. Both in/out buffers have 1MB size.

    void ZipMiniStream::ZipMiniCompressZstd( std::function<std::pair<uint64_t, bool>(uint8_t * dst, uint64_t size)> stream_in, std::function<void(uint8_t const * dst, uint64_t size)> stream_out, uint64_t compression_level) { ZSTD_CStream * ctx_ = ZSTD_createCStream(); ASSERT(ctx_);

    ZSTD_CHECK(ZSTD_initCStream(ctx_, compression_level));
    
    uint64_t readed_size = 0, ret = 0;
    bool finished = false;
    
    ZSTD_inBuffer in{.src = s_inbuf.data(), .size = s_inbuf.size(), .pos = 0};
    ZSTD_outBuffer out{.dst = s_outbuf.data(), .size = s_outbuf.size(), .pos = 0};
    
    while (true) {
        if (s_inbuf.size() > in.size) {
            std::tie(readed_size, finished) =
                stream_in(s_inbuf.data() + in.size, s_inbuf.size() - in.size);
            in.size += readed_size;
            ASSERT(readed_size > 0 || finished);
        }
        
        ZSTD_CHECK((ret = ZSTD_compressStream2(
            ctx_, &out, &in, finished ? ZSTD_e_end : ZSTD_e_continue)));
        
        if (in.pos > 0) {
            std::memmove(s_inbuf.data(), s_inbuf.data() + in.pos, in.size - in.pos);
            in.size -= in.pos;
            in.pos = 0;
        }
        if (out.pos > 0) {
            stream_out(s_outbuf.data(), out.pos);
            out.pos = 0;
        }
        if (finished && ret == 0) {
            ASSERT(in.size == 0);
            break;
        }
    }
    
    ZSTD_CHECK(ZSTD_freeCStream(ctx_));
    

    }

    void ZipMiniStream::ZipMiniDeCompressZstd( std::function<std::pair<uint64_t, bool>(uint8_t * dst, uint64_t size)> stream_in, std::function<void(uint8_t const * dst, uint64_t size)> stream_out) {

    ZSTD_DStream * ctx_ = ZSTD_createDStream();
    ASSERT(ctx_);
    
    ZSTD_CHECK(ZSTD_initDStream(ctx_));
    
    uint64_t readed_size = 0, ret = 0;
    bool finished = false;
    
    ZSTD_inBuffer in{.src = s_inbuf.data(), .size = 0, .pos = 0};
    ZSTD_outBuffer out{.dst = s_outbuf.data(), .size = s_outbuf.size(), .pos = 0};
    
    while (true) {
        if (s_inbuf.size() > in.size) {
            std::tie(readed_size, finished) =
                stream_in(s_inbuf.data() + in.size, s_inbuf.size() - in.size);
            in.size += readed_size;
            ASSERT(readed_size > 0 || finished);
        }
        
        DUMP(in.size); DUMP(in.pos); DUMP(out.size); DUMP(out.pos);
        ZSTD_CHECK((ret = ZSTD_decompressStream(ctx_, &out, &in)));
        DUMP(ret); DUMP(in.size); DUMP(in.pos); DUMP(out.size); DUMP(out.pos);
        
        if (in.pos > 0) {
            std::memmove(s_inbuf.data(), s_inbuf.data() + in.pos, in.size - in.pos);
            in.size -= in.pos;
            in.pos = 0;
        }
        if (out.pos > 0) {
            stream_out(s_outbuf.data(), out.pos);
            out.pos = 0;
        }
        if (finished && ret == 0) {
            ASSERT(in.size == 0);
            break;
        }
    }
    
    ZSTD_CHECK(ZSTD_freeDStream(ctx_));
    

    }

  2. Assistant

    The issue you're encountering—where compressing 7200 bytes of input and then decompressing results in a decompressed vector that's unexpectedly large (~1MB)—is likely due to how the input buffer sizes are being managed in both your compression and decompression functions. Specifically, the problem arises from incorrectly setting the ZSTD_inBuffer.size to the buffer's capacity rather than the actual number of bytes read.

    Detailed Explanation

    1. Understanding ZSTD_inBuffer and ZSTD_outBuffer

    Zstandard (ZSTD) uses ZSTD_inBuffer and ZSTD_outBuffer structures to manage input and output data for compression and decompression streams. These structures have the following fields:

    • src / dst: Pointers to the input/output data.
    • size: The number of bytes available in src or dst respectively.
    • pos: The current position in the buffer that the engine has processed or filled up to.

    It's crucial that the size field accurately reflects the actual amount of data present in the buffer, not just the buffer's total capacity.

    2. Issues in the Compression Function (ZipMiniCompressZstd)

    In your compression function, you initialize the input buffer as follows:

    ZSTD_inBuffer in{.src = s_inbuf.data(), .size = s_inbuf.size(), .pos = 0};
    

    Here, s_inbuf.size() likely represents the capacity of the buffer (e.g., 1MB), not the actual number of bytes read (7200 bytes). By setting .size to the buffer's capacity, the compressor assumes that the entire buffer is filled with valid data, which isn't the case. This leads to undefined behavior where the compressor might process uninitialized or irrelevant data beyond the 7200 bytes you intended, resulting in incorrect compression and subsequently an inflated decompressed size.

    3. Issues in the Decompression Function (ZipMiniDeCompressZstd)

    Similarly, in your decompression function, you initialize the input buffer with:

    ZSTD_inBuffer in{.src = s_inbuf.data(), .size = 0, .pos = 0};
    

    While in.size starts at 0, ensure that each time you read new data into the buffer, in.size accurately reflects the number of bytes actually read, not the buffer's total capacity. Mismanagement here can also cause the decompressor to read beyond the valid data, leading to the unexpected large output.

    4. Recommended Fix

    To resolve this issue, you need to ensure that the .size field of ZSTD_inBuffer accurately represents the actual number of bytes read into the buffer, not the buffer's total capacity. Here's how you can adjust both functions:

    Adjusting the Compression Function

    void ZipMiniStream::ZipMiniCompressZstd(
            std::function<std::pair<uint64_t, bool>(uint8_t * dst, uint64_t size)> stream_in,
            std::function<void(uint8_t const * dst, uint64_t size)> stream_out,
            uint64_t compression_level) {
        ZSTD_CStream *ctx_ = ZSTD_createCStream();
        ASSERT(ctx_);
    
        ZSTD_CHECK(ZSTD_initCStream(ctx_, compression_level));
    
        uint64_t read_size = 0, ret = 0;
        bool finished = false;
    
        // Initialize in.size to 0, as no data has been read yet
        ZSTD_inBuffer in{.src = s_inbuf.data(), .size = 0, .pos = 0};
        ZSTD_outBuffer out{.dst = s_outbuf.data(), .size = s_outbuf.size(), .pos = 0};
    
        while (true) {
            if (in.size < s_inbuf.size()) { // Check if there is space to read more data
                std::tie(read_size, finished) =
                    stream_in(s_inbuf.data() + in.size, s_inbuf.size() - in.size);
                in.size += read_size;
                ASSERT(read_size > 0 || finished);
            }
    
            ret = ZSTD_compressStream2(
                ctx_, &out, &in, finished ? ZSTD_e_end : ZSTD_e_continue);
            ZSTD_CHECK(ret);
    
            if (in.pos > 0) {
                std::memmove(s_inbuf.data(), s_inbuf.data() + in.pos, in.size - in.pos);
                in.size -= in.pos;
                in.pos = 0;
            }
            if (out.pos > 0) {
                stream_out(s_outbuf.data(), out.pos);
                out.pos = 0;
            }
            if (finished && ret == 0) {
                ASSERT(in.size == 0);
                break;
            }
        }
    
        ZSTD_check(ZSTD_freeCStream(ctx_));
    }
    

    Adjusting the Decompression Function

    void ZipMiniStream::ZipMiniDeCompressZstd(
            std::function<std::pair<uint64_t, bool>(uint8_t * dst, uint64_t size)> stream_in,
            std::function<void(uint8_t const * dst, uint64_t size)> stream_out) {
    
        ZSTD_DStream *ctx_ = ZSTD_createDStream();
        ASSERT(ctx_);
    
        ZSTD_CHECK(ZSTD_initDStream(ctx_));
    
        uint64_t read_size = 0, ret = 0;
        bool finished = false;
    
        // Initialize in.size to 0, as no data has been read yet
        ZSTD_inBuffer in{.src = s_inbuf.data(), .size = 0, .pos = 0};
        ZSTD_outBuffer out{.dst = s_outbuf.data(), .size = s_outbuf.size(), .pos = 0};
    
        while (true) {
            if (in.size < s_inbuf.size()) { // Check if there is space to read more data
                std::tie(read_size, finished) =
                    stream_in(s_inbuf.data() + in.size, s_inbuf.size() - in.size);
                in.size += read_size;
                ASSERT(read_size > 0 || finished);
            }
    
            ret = ZSTD_decompressStream(ctx_, &out, &in);
            ZSTD_CHECK(ret);
    
            if (in.pos > 0) {
                std::memmove(s_inbuf.data(), s_inbuf.data() + in.pos, in.size - in.pos);
                in.size -= in.pos;
                in.pos = 0;
            }
            if (out.pos > 0) {
                stream_out(s_outbuf.data(), out.pos);
                out.pos = 0;
            }
            if (finished && ret == 0) {
                ASSERT(in.size == 0);
                break;
            }
        }
    
        ZSTD_CHECK(ZSTD_freeDStream(ctx_));
    }
    

    Key Changes:

    1. Initialize in.size to 0: Start with no data read, allowing the first read to accurately set the number of bytes available.

    2. Update in.size Based on Actual Data Read: After each read operation, increment in.size by the number of bytes actually read (read_size), not the buffer's total capacity.

    3. Check Available Space Before Reading More Data: Ensure there's space in the buffer before attempting to read more data by comparing in.size with s_inbuf.size().

    4. Handle in.pos Correctly: After processing, move any remaining data to the beginning of the buffer and reset positions appropriately.

    5. Additional Recommendations

    • Validate Buffer Reads: Always ensure that the number of bytes read does not exceed the buffer's capacity to prevent buffer overflows.

    • Error Handling: Ensure that all ZSTD function calls are checked for errors, and handle them gracefully.

    • Buffer Sizes: Re-evaluate whether a 1MB buffer is necessary for both input and output. Depending on your use case, smaller buffers might be more efficient and reduce memory usage.

    • Testing: After making these changes, thoroughly test with various input sizes to ensure that compression and decompression behave as expected.

    By ensuring that ZSTD_inBuffer.size accurately reflects the actual number of bytes of data rather than the buffer's capacity, you should see correct compression and decompression behavior, eliminating the issue of unexpectedly large decompressed data.


    References:


    Note: Always ensure that buffers are managed correctly when working with low-level APIs like ZSTD to prevent data corruption and unexpected behavior.

    Summary: The compressed data likely includes unintended bytes because the input buffer’s size is set to the entire buffer capacity (1 MB) instead of the actual data size (7200 bytes). To fix this, ensure that ZSTD_inBuffer.size is set to the number of bytes read (7200), not the buffer’s total size, in both compression and decompression functions.