I'm loading many large numpy arrays from multiple npz files. I don't want all of them in…
- User
I'm loading many large numpy arrays from multiple npz files. I don't want all of them in memory at the same time as I will run into OOM issues. How do I load them into memory only when I want to use them for a function and release them from memory after I'm done?
- Assistant
Here are a few patterns you can use to load one large
.npz(or.npy) at a time, process it, and then free the memory before moving on to the next file.-
Lazy‐loading via NumPy memmaps
• If your.npzarchive was built from plain.npyfiles (the default), you can doimport numpy as np def process_npz(path): # mmap_mode='r' means no full copy into RAM, pages get pulled in on demand with np.load(path, mmap_mode='r') as archive: # assume key 'arr' inside your npz arr = archive['arr'] result = do_something(arr) # exiting the with‐block closes the memmap filehandle # 'arr' is now a view to a closed memmap—if you need the data, copy it: # arr = arr.copy() return result for fn in list_of_npz_files: r = process_npz(fn) print(fn, "→", r)• Because you used
mmap_mode='r'you never pulled the entire file into RAM at once, only pages you actually touch.
• Exiting thewithblock closes the file handle. No need to calldelorgc.collect()unless you hold onto big copies. -
Manual load & delete
If you’d rather pull it fully into RAM but then explicitly free it:import numpy as np, gc def process_npz(path): data = np.load(path) # full in-memory load arr = data['arr'] result = do_something(arr) # cleanup del data, arr gc.collect() # force Python GC return result for fn in list_of_npz_files: print(process_npz(fn)) -
Offloading work to a subprocess
Sometimes Python’s memory allocator won’t return freed pages to the OS immediately. A foolproof way is to spawn a child process to handle each file:from multiprocessing import Pool import numpy as np def worker(path): data = np.load(path) result = do_something(data['arr']) return result if __name__ == "__main__": with Pool(processes=1) as p: # pool of size 1 so you only ever have one file in memory results = p.map(worker, list_of_npz_files) print(results)Once the worker finishes, its process dies, and the OS reclaims all its RAM immediately.
-
Higher-level chunked libraries
If your workflow is more “slice big arrays, do some compute, repeat,” you might consider:• Dask arrays (they also support lazy loading of
.npy/.npzvia memmap under the hood)
• Zarr (chunked, compressed arrays on disk with very fine-grained I/O)
• HDF5 via h5py
— Pick the pattern that best fits your needs. For a quick win, the
np.load(..., mmap_mode='r')+with‐block approach will let you touch arbitrarily large files one at a time without blowing your RAM. -
- User
What does allow_pickle = True do?
- Assistant
When you call
np.load(filename, allow_pickle=True)you’re telling NumPy that it’s ok to reconstruct (“unpickle”) any Python objects that were stored inside your
.npyor.npzfile. Here’s what that means in practice:-
Why pickle shows up in
.npy/.npz- By default, NumPy can save and load plain numeric arrays (dtype=int, float, bool, etc.) without pickling.
- If you save an array with dtype=object (e.g. an array of Python lists, tuples, custom classes, or mixed types), NumPy will serialize each element with Python’s pickle protocol.
- On load, it must unpickle those objects to re-create your original data structures.
-
allow_pickle=False (the safer default)
- Blocks any attempt to unpickle.
- If your file contains object-arrays or anything that wasn’t a “raw” numeric buffer, you’ll see an error:
ValueError: Object arrays cannot be loaded when allow_pickle=False - Protects you from maliciously crafted pickle data (since pickle can execute arbitrary code).
-
allow_pickle=True
- Permits unpickling of stored Python objects.
- You only need it if you originally saved object-dtype arrays or non-NumPy data.
- Example:
# Suppose arr is an object-dtype array: arr = np.array([{"foo": 1}, {"bar": 2}], dtype=object) np.save("my_objects.npy", arr) # Later, to load it back: loaded = np.load("my_objects.npy", allow_pickle=True) print(type(loaded[0])) # → <class 'dict'>
-
Security implications
- Never use
allow_pickle=Trueon files from untrusted sources. - A crafted pickle payload can run arbitrary code on your machine.
- Never use
-
When you typically don’t need it
- If you’re only saving/loading numeric arrays (e.g. floats, ints, bools), you can leave
allow_pickle=False(or omit it, since in NumPy ≥1.16 the default is False).
- If you’re only saving/loading numeric arrays (e.g. floats, ints, bools), you can leave
Summary
- Use
allow_pickle=Trueonly when you know your file contains Python objects that were pickled. - Keep it False (the safer option) for pure numeric data.
-
- User
Let's say I load the npz files like this:
print('loading standard preds') standard_unet_preds = np.load(os.path.join(standard_save_path, 'standard_unet_preds.npz')) standard_squeezeformer_preds = np.load(os.path.join(standard_save_path, 'standard_squeezeformer_preds.npz')) standard_pure_resLSTM_preds = np.load(os.path.join(standard_save_path, 'standard_pure_resLSTM_preds.npz')) standard_pao_model_preds = np.load(os.path.join(standard_save_path, 'standard_pao_model_preds.npz')) standard_convnext_preds = np.load(os.path.join(standard_save_path, 'standard_convnext_preds.npz')) standard_encdec_lstm_preds = np.load(os.path.join(standard_save_path, 'standard_encdec_lstm_preds.npz')) temp_results = my_function(standard_unet_preds['seed_7'])Once I get temp_results, how do I release standard_unet_preds['seed_7'] from memory? Is there no way to put the genie back in the bottle?
- Assistant
Here are a few ways to “put the genie back in the bottle” after you’ve pulled out one slice from your
.npz.- Use a context‐manager (
with np.load…)
def run_on_seed(path, seed_key): with np.load(path) as archive: # ← opens & will auto‐close arr = archive[seed_key] # ← lazy memmap if you used mmap_mode result = my_function(arr) # at this point archive is closed # `arr` is just a numpy array (or memmap) in your local scope return result # example res = run_on_seed("standard_unet_preds.npz", "seed_7") # no lingering file handles- Manually close + delete
import numpy as np, gc f = np.load("standard_unet_preds.npz", mmap_mode="r") # lazy-load pages arr = f["seed_7"] temp_results = my_function(arr) # now tear down f.close() # closes the zip‐file handle del f, arr # drop references gc.collect() # nudge the GC (optional—Python may not return pages to OS)- Force a fresh Python process per file
Because Python’s allocator often holds on to freed memory instead of returning it to the OS, the bulletproof trick is to spawn a subprocess so that all its RAM is fully reclaimed when it exits:
from multiprocessing import Pool def worker(path, key): with np.load(path) as f: return my_function(f[key]) if __name__ == "__main__": files_and_keys = [("standard_unet_preds.npz","seed_7"), ("standard_squeezeformer_preds.npz","seed_7"), # … ] # pool of size 1 guarantees only one file in RAM at a time with Pool(1) as P: results = P.starmap(worker, files_and_keys)- Why you sometimes can’t “give back” RAM
- Even after
del+gc.collect(), the C-heap that NumPy used may stay with your Python process—ready for future allocations instead of going back to the OS. - If you’re hitting system-wide OOM, the safest is the subprocess approach or using chunked on-disk stores (memmap, Zarr, HDF5, Dask, …).
Bottom line
- If you just want to drop that one slice: close your
.npz(f.close()),deleverything that held it, and callgc.collect(). - If you need true OS-level reclamation between loads, spin a short-lived worker process (or use a context manager + subprocess).
- Use a context‐manager (
- User
Can I put the genie back in the bottle by replacing the np.load statements with lambda functions?
- Assistant
You won’t get any magic “free memory” trick just by sticking your
np.loadcalls into lambdas—lambdas only defer when you callnp.load, they don’t change how the array is kept in RAM once loaded. What can help you is a little bit of indirection so that:- You only keep file‐paths (or loader callables) in your main namespace
- You load the file inside a small scope (e.g. a function or a
with‐block) - You immediately close/delete/GC it before returning
Here’s a quick recipe showing how you could wrap each file in a tiny loader‐function (or lambda) and then only ever hold in memory the slice you actually need:
import os, numpy as np, gc from functools import partial standard_save_path = "/…/wherever/you/keep/your/npzs" # 1) build a dict of lazy loaders def make_npz_loader(path, mmap_mode="r"): """ Returns a function that, when called with a key, will: - open the .npz - grab the array at that key - close the file - return the array """ def loader(key): # open the archive with np.load(path, mmap_mode=mmap_mode) as archive: arr = archive[key] # If you need to keep arr beyond the `with`‐block, you must .copy() # otherwise `arr` can still refer to the mmap which closes here. return arr.copy() return loader # set up your loaders load_unet = make_npz_loader(os.path.join(standard_save_path, 'standard_unet_preds.npz')) load_sqm = make_npz_loader(os.path.join(standard_save_path, 'standard_squeezeformer_preds.npz')) # … and so on for the rest # 2) when you actually want to run your function, load, use, then drop def process_one_seed(loader, seed_key): arr = loader(seed_key) # npz is opened, sliced, then closed automatically out = my_function(arr) # do whatever you need with the slice # Clean up del arr # lose your reference gc.collect() # hint to the collector return out # 3) example usage if __name__ == "__main__": result = process_one_seed(load_unet, "seed_7") print(result) # at this point you have no lingering npz filehandles # and arr was .copy()ed into a fresh array that you immediately del-eted.Why this works:
- By delaying the
np.loadinside a function, you never have all the arrays in memory at the same time—only the one you call. - By using a
with np.load(...)block, the file handle is closed as soon as you leave that block. - By copying immediately (or by using
mmap_mode="r"if you only need read-through), you control exactly how much RAM you touch. - Finally,
del+gc.collect()nudges Python to free the array’s C-heap if you’re running tight on RAM.
If you really need bullet-proof OS-level reclamation between loads, you can still wrap your
process_one_seedinside a short-lived subprocess (viamultiprocessingorsubprocess), but 99% of the time the pattern above is enough. - User
Is the lambda solution the most elegant one thus far?
- Assistant
The short answer
• A lambda (or small function) that closes over the file‐path and only does the load when called is already pretty neat—you get one “loader” per .npz and you never have all your arrays in memory at once.
• You’re really just trading your big NumPy arrays for a few tiny loader objects that point at disk.Is it the “most elegant”? It depends on how you define elegant – you might prefer any of these variants:
- The minimalist lambda approach
load_unet = lambda key: np.load("standard_unet_preds.npz")["seed_" + str(key)] # usage: arr = load_unet(7) # arr is in memory now, close/GC when you’re done...Pros: 1-liner, no extra imports.
Cons: you have to remember todel arr; gc.collect()(and the file‐handle stays open until Python’s GC).- A little factory‐function that uses a with-block
def make_loader(path, mmap_mode="r"): def loader(key): with np.load(path, mmap_mode=mmap_mode) as z: return z[key].copy() # copy if you need a standalone np.array return loader load_unet = make_loader("standard_unet_preds.npz") arr = load_unet("seed_7") # z is closed automatically, arr is yours del arr # free itPros: ensures the .npz zip‐file is closed immediately. Still just a few lines.
- A tiny LazyNPZ class (even more “pythonic”)
class LazyNPZ: def __init__(self, path, mmap_mode="r"): self.path = path self.mmap_mode = mmap_mode def __getitem__(self, key): with np.load(self.path, mmap_mode=self.mmap_mode) as archive: arr = archive[key] # if you need arr past this block, unwrap its data return arr.copy() # usage unet = LazyNPZ("standard_unet_preds.npz") arr = unet["seed_7"] # do work… del arrPros: you get an indexable object (
unet[...]), no lambdas, clear intent.- Higher-level tool (Dask/Zarr/HDF5)
If you’re slicing huge arrays frequently, it may be worth switching to a chunked-on-disk format (Dask-array, Zarr, HDF5), which give you true lazy evaluation and very fine‐grained I/O.
Bottom line
– Yes, “lambda loaders” are fine for 90% of quick-and-dirty use-cases.
– If you want a bit more structure, wrap that lambda in a small function or class that uses awith np.load(…)so you never accidentally leave file handles or memmaps dangling.
– For bullet-proof OS-level frees you’ll still want todel array; gc.collect()or even isolate the work in a subprocess. - User
Is it possible to have a multi line lambda function/
- Assistant
Python lambdas are strictly limited to a single expression—you can’t put multiple statements (e.g. assignments, loops,
if …: …blocks) inside them. If you find yourself needing more than “one expression,” it’s time to write a normaldef-function.That said, you can break that one expression across lines for readability by using parentheses or
\line-continuations. For example:# still a single expression, just split across 3 lines f = lambda x: ( x*2 + x**3 - 5 ) print(f(3)) # 3*2 + 3**3 - 5 = 6 + 27 - 5 = 28—or a tiny hack to execute two calls in one expression and return the second result:
# hack: tuple-evaluation, return the last element g = lambda x: (print("logging", x), x**2)[1] res = g(4) # logs "logging 4" # res == 16But these tricks are generally harder to read and maintain.
If you really need multiple steps, just do:def nice_loader(path, key): with np.load(path, mmap_mode="r") as archive: arr = archive[key] # maybe copy() if you need an independent array… return arrThat’s clearer, debuggable, and lets you do as many statements (logging, error-checks, cleanup) as you like. 😉
- User
A multi line lambda solution is not elegant, but I was wondering if i needed to explicitly use gc.collect
- Assistant
You almost never need to sprinkle
gc.collect()after every load—Python’s reference‐counting will immediately free a NumPy array as soon as you drop its last reference. Here’s the typical pattern:# 1) load and use in a tight scope with np.load("standard_unet_preds.npz", mmap_mode="r") as z: arr = z["seed_7"].copy() # now `arr` is a regular in-memory ndarray result = my_function(arr) # 2) teardown del arr # drops the last reference to the ndarray data bufferOnce you
del arr, CPython sees the reference count go to zero and releases that buffer. You do not have to callgc.collect()in this case because there are no reference cycles involved.When you might consider
gc.collect():• If you’ve built up more complex object graphs with cyclical references,
then a manualgc.collect()would force the cycle detector to run.
• If you’re watching your process in a system monitor and don’t see memory drop,
remember that the Python allocator often holds onto freed C-heap pages
for future allocations rather than returning them to the OS.
Evengc.collect()won’t force those pages back to the OS.If you absolutely must see memory go back to the OS between loads, the most bullet-proof solution is to isolate each load/use into its own short-lived subprocess (via
multiprocessingorsubprocess). When that process dies, all of its RAM is reclaimed by the OS.In summary:
- Just
del arr(and close any open memmaps or file handles) and you’re done. - You only need manual
gc.collect()if you have true reference‐cycle garbage to clean up—rare with plain NumPy arrays—and it still won’t necessarily return the freed pages to the OS.
- Just
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.