Using C library 'miniz' I have two helper C++ functions for streaming compression and…
- User
Using C library 'miniz' I have two helper C++ functions for streaming compression and decompression. They work incredibly slow, can you fix them?
void ZipMiniCompress( std::function<std::pair<uint64_t, bool>(uint8_t * dst, uint64_t size)> stream_in, std::function<void(uint8_t const * dst, uint64_t size)> stream_out, uint64_t compression_level) { using namespace zip_mini_internal;
std::vector<uint8_t> s_inbuf(BUF_SIZE); std::vector<uint8_t> s_outbuf(BUF_SIZE); mz_stream stream; // Init stream. MEMZERO(stream); stream.next_in = s_inbuf.data(); stream.avail_in = 0; stream.next_out = s_outbuf.data(); stream.avail_out = BUF_SIZE; MZ_CHECK(mz_deflateInit(&stream, compression_level)); uint64_t readed_size = 0; bool finished = false; while (true) { int status = 0; if (!stream.avail_in) { std::tie(readed_size, finished) = stream_in(s_inbuf.data(), BUF_SIZE); stream.next_in = s_inbuf.data(); stream.avail_in = readed_size; ASSERT(readed_size > 0 || finished); } status = mz_deflate(&stream, finished ? MZ_FINISH : MZ_NO_FLUSH); if ((status == MZ_STREAM_END) || (!stream.avail_out)) { // Output buffer is full, or compression is done, so write buffer to output file. uint32_t n = BUF_SIZE - stream.avail_out; stream_out(s_outbuf.data(), n); stream.next_out = s_outbuf.data(); stream.avail_out = BUF_SIZE; } if (status == MZ_STREAM_END) break; else { MZ_CHECK(status); } } MZ_CHECK(mz_deflateEnd(&stream));}
void ZipMiniDeCompress( std::function<std::pair<uint64_t, bool>(uint8_t * dst, uint64_t size)> stream_in, std::function<void(uint8_t const * dst, uint64_t size)> stream_out, uint64_t * compressed_size) { using namespace zip_mini_internal;
std::vector<uint8_t> s_inbuf(BUF_SIZE); std::vector<uint8_t> s_outbuf(BUF_SIZE); mz_stream stream; // Init stream. MEMZERO(stream); stream.next_in = s_inbuf.data(); stream.avail_in = 0; stream.next_out = s_outbuf.data(); stream.avail_out = BUF_SIZE; MZ_CHECK(mz_inflateInit(&stream)); uint64_t readed_size = 0, total_readed_size = 0; bool finished = false; while (true) { int status = 0; if (!stream.avail_in) { std::tie(readed_size, finished) = stream_in(s_inbuf.data(), BUF_SIZE); total_readed_size += readed_size; stream.next_in = s_inbuf.data(); stream.avail_in = readed_size; ASSERT(readed_size > 0 || finished); } status = mz_inflate(&stream, MZ_SYNC_FLUSH); if ((status == MZ_STREAM_END) || (!stream.avail_out)) { // Output buffer is full, or compression is done, so write buffer to output file. uint32_t n = BUF_SIZE - stream.avail_out; stream_out(s_outbuf.data(), n); stream.next_out = s_outbuf.data(); stream.avail_out = BUF_SIZE; } if (finished) { ASSERT(status == MZ_STREAM_END); } if (status == MZ_STREAM_END) { if (compressed_size) *compressed_size = total_readed_size - stream.avail_in; break; } else { MZ_CHECK(status); } } MZ_CHECK(mz_inflateEnd(&stream));}
- Assistant
The performance issues you're experiencing with your
ZipMiniCompressandZipMiniDeCompressfunctions are likely due to a combination of factors, primarily:- Use of
std::function:std::functionintroduces type erasure and can incur significant overhead, especially when used in performance-critical loops. - Buffer Size (
BUF_SIZE): IfBUF_SIZEis relatively small, frequent calls tostream_inandstream_outcan degrade performance due to increased function call overhead and less efficient data processing. - Excessive Assertions and Error Checks: While important for debugging, excessive runtime checks can slow down release builds if not properly optimized.
To address these issues, here's a revised version of your compression and decompression functions with the following optimizations:
- Template-Based Callbacks: Replacing
std::functionwith templated callable objects (like function pointers or lambda expressions) to eliminate the overhead associated withstd::function. - Increased Buffer Size: Using a larger buffer size (e.g., 1 MB) to reduce the number of I/O operations.
- Optimized Loop Structure: Minimizing condition checks inside the main loop.
- Inlining and Compiler Optimizations: Ensuring that functions can be inlined and optimized effectively by the compiler.
Revised
ZipMiniCompress#include <vector> #include <utility> #include <cstdint> #include <cassert> #include "miniz.h" namespace zip_mini_internal { constexpr size_t BUF_SIZE = 1 << 20; // 1 MB inline void MEMZERO(mz_stream& stream) { memset(&stream, 0, sizeof(mz_stream)); } inline void MZ_CHECK(int status) { if (status != MZ_OK && status != MZ_STREAM_END) { throw std::runtime_error("Miniz error"); } } inline void ASSERT(bool condition) { assert(condition); } } template <typename StreamIn, typename StreamOut> void ZipMiniCompress( StreamIn stream_in, StreamOut stream_out, uint64_t compression_level) { using namespace zip_mini_internal; std::vector<uint8_t> s_inbuf(BUF_SIZE); std::vector<uint8_t> s_outbuf(BUF_SIZE); mz_stream stream; // Initialize stream MEMZERO(stream); stream.next_in = s_inbuf.data(); stream.avail_in = 0; stream.next_out = s_outbuf.data(); stream.avail_out = static_cast<mz_uint>(BUF_SIZE); int status = mz_deflateInit(&stream, static_cast<int>(compression_level)); MZ_CHECK(status); uint64_t read_size = 0; bool finished = false; while (true) { if (stream.avail_in == 0 && !finished) { std::tie(read_size, finished) = stream_in(s_inbuf.data(), BUF_SIZE); stream.next_in = s_inbuf.data(); stream.avail_in = static_cast<mz_uint>(read_size); ASSERT(read_size > 0 || finished); } status = mz_deflate(&stream, finished ? MZ_FINISH : MZ_NO_FLUSH); MZ_CHECK(status); if (stream.avail_out == 0 || status == MZ_STREAM_END) { size_t output_size = BUF_SIZE - stream.avail_out; stream_out(s_outbuf.data(), output_size); stream.next_out = s_outbuf.data(); stream.avail_out = static_cast<mz_uint>(BUF_SIZE); } if (status == MZ_STREAM_END) { break; } } mz_deflateEnd(&stream); }Revised
ZipMiniDeCompress#include <vector> #include <utility> #include <cstdint> #include <cassert> #include "miniz.h" namespace zip_mini_internal { constexpr size_t BUF_SIZE = 1 << 20; // 1 MB inline void MEMZERO(mz_stream& stream) { memset(&stream, 0, sizeof(mz_stream)); } inline void MZ_CHECK(int status) { if (status != MZ_OK && status != MZ_STREAM_END) { throw std::runtime_error("Miniz error"); } } inline void ASSERT(bool condition) { assert(condition); } } template <typename StreamIn, typename StreamOut> void ZipMiniDecompress( StreamIn stream_in, StreamOut stream_out, uint64_t* compressed_size = nullptr) { using namespace zip_mini_internal; std::vector<uint8_t> s_inbuf(BUF_SIZE); std::vector<uint8_t> s_outbuf(BUF_SIZE); mz_stream stream; // Initialize stream MEMZERO(stream); stream.next_in = s_inbuf.data(); stream.avail_in = 0; stream.next_out = s_outbuf.data(); stream.avail_out = static_cast<mz_uint>(BUF_SIZE); int status = mz_inflateInit(&stream); MZ_CHECK(status); uint64_t read_size = 0, total_read_size = 0; bool finished = false; while (true) { if (stream.avail_in == 0 && !finished) { std::tie(read_size, finished) = stream_in(s_inbuf.data(), BUF_SIZE); total_read_size += read_size; stream.next_in = s_inbuf.data(); stream.avail_in = static_cast<mz_uint>(read_size); ASSERT(read_size > 0 || finished); } status = mz_inflate(&stream, MZ_SYNC_FLUSH); MZ_CHECK(status); if (stream.avail_out == 0 || status == MZ_STREAM_END) { size_t output_size = BUF_SIZE - stream.avail_out; stream_out(s_outbuf.data(), output_size); stream.next_out = s_outbuf.data(); stream.avail_out = static_cast<mz_uint>(BUF_SIZE); } if (status == MZ_STREAM_END) { if (compressed_size) { *compressed_size = total_read_size - stream.avail_in; } break; } } mz_inflateEnd(&stream); }Key Improvements Explained
-
Template Parameters Instead of
std::function:By converting the helper functions to templates, you allow the compiler to inline the
stream_inandstream_outcalls, eliminating the overhead associated withstd::function. This can lead to significant speed improvements, especially in tight loops.template <typename StreamIn, typename StreamOut> void ZipMiniCompress(StreamIn stream_in, StreamOut stream_out, uint64_t compression_level); template <typename StreamIn, typename StreamOut> void ZipMiniDecompress(StreamIn stream_in, StreamOut stream_out, uint64_t* compressed_size = nullptr);Usage Example:
// Compression example ZipMiniCompress( [&](uint8_t* dst, uint64_t size) -> std::pair<uint64_t, bool> { // Implement your stream_in logic here }, [&](const uint8_t* src, uint64_t size) { // Implement your stream_out logic here }, compression_level ); -
Increased Buffer Size (
BUF_SIZE):Increasing the buffer size to 1 MB reduces the number of read and write operations, which can significantly improve performance, especially when dealing with large datasets.
constexpr size_t BUF_SIZE = 1 << 20; // 1 MBAdjust this size based on your application's memory constraints and typical data sizes. Larger buffers generally improve performance up to a certain point beyond which memory bandwidth becomes a limiting factor.
-
Minimized Condition Checks:
By restructuring the loop and minimizing the number of condition checks within it, you reduce the overhead during each iteration.
while (true) { if (stream.avail_in == 0 && !finished) { // Load more data } // Perform compression/decompression if (stream.avail_out == 0 || status == MZ_STREAM_END) { // Write out data } if (status == MZ_STREAM_END) { break; } } -
Error Handling Optimizations:
The
MZ_CHECKfunction now throws exceptions on errors, which can be more efficient than handling errors through assertions during runtime. Ensure that exception handling is acceptable in your application context. -
Avoiding Unnecessary Operations:
By setting
stream.avail_outandstream.next_outonly when necessary and avoiding resetting them on every iteration, you reduce the number of operations inside the loop.
Additional Recommendations
-
Profile Your Code:
Before and after applying these optimizations, use profiling tools (like
gprof,Valgrind, or Visual Studio's Performance Profiler) to identify where the actual bottlenecks are. This ensures that you're optimizing the parts of the code that most affect performance. -
Compiler Optimizations:
Ensure that your project is compiled with optimization flags enabled (e.g.,
-O2or-O3for GCC/Clang,/O2for MSVC). Optimizations can significantly impact the performance of tight loops and inlined functions. -
Threading and Parallelism:
If your use case allows, consider processing multiple streams in parallel using multi-threading. However, be cautious with thread safety, especially if using shared resources.
-
Miniz Configuration:
Ensure that the miniz library is configured for optimal performance. This might include enabling specific compression algorithms or tuning internal parameters based on your data.
-
Memory Allocation:
Preallocate large enough buffers and reuse them to avoid dynamic memory allocations during the compression/decompression processes.
Final Thoughts
The primary performance gains in the revised functions come from eliminating the overhead of
std::functionand increasing the buffer size to reduce the number of I/O operations. By making these changes and following additional recommendations, you should observe a significant improvement in the speed of your compression and decompression routines.Feel free to further tailor these optimizations based on the specific characteristics and requirements of your application.
- Use of