Hello. Discussing storage solutions - for example S3 or Object Storage by Google - how…
- User
Hello. Discussing storage solutions - for example S3 or Object Storage by Google - how many bytes of data they keep in memory during writing? Like, few kilobytes, few megabytes? And why?
- Assistant
When you upload an object to “infinite”‐scale stores like S3 or Google Cloud Storage, neither service ever needs to buffer the entire file in RAM. Instead they (and your client‐side SDK) break the stream into smaller chunks, upload each chunk (often in parallel), then reassemble on the back end. The amount of RAM in use at any one time is basically “chunk size × concurrency,” which—by default—is on the order of kilobytes to a few megabytes per concurrent upload.
—————————————————————————
1. AWS S3 (Multipart Upload)
• Chunk (part) size
– Minimum: 5 MiB per part (except the last part can be smaller)
– Default in many AWS SDKs/CLI: 8 MiB (some SDKs use 5 MiB or 16 MiB)
– You can configure it anywhere between 5 MiB and 5 GiB.• Concurrency
– By default the AWS SDKs will upload ~4–10 parts in parallel.
– Total RAM ≈ (part_size) × (number_of_parallel_parts).
Example: 8 MiB × 5 threads = ~40 MiB of buffer under heavy upload.• Why this design?
- Large enough to get good TCP throughput
- Small enough to keep RAM usage predictable
- Allows retrying just failed parts, not the whole file
—————————————————————————
2. Google Cloud Storage (Resumable Uploads)
• Chunk size
– Default in the official client libraries: often 256 KiB or 1 MiB per chunk
– You can set it to anything from a few KiB up to dozens of MiB.• Concurrency
– Typically single–chunk–at–a–time per stream (but you can run multiple streams yourself)
– Total RAM ≈ chunk_size × number_of_simultaneous_streams• Why this design?
- Keeps each HTTP request lean
- Eases recovery on network hiccups (only the last chunk must be re-sent)
- Balances memory use vs. round-trip overhead
————————————————————————
3. Bottom Line
-
“How many bytes in memory?”
– Roughly KBs→a few MB per concurrent upload.
– You’ll never see “hundreds of megabytes” or gigabytes unless you explicitly bump chunk sizes or parallelism way up. -
“Why those limits?”
- To optimize network throughput
- To bound memory usage on both client and server
- To allow fine-grained retries on failure
—
💡 If you have a specific SDK or language in mind, you can usually tweak two knobs:- chunk/part size,
- parallelism
to hit your sweet spot of throughput vs. RAM consumption.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.