Build a GPU memory fragmentation diagnosis toolkit: telemetry capture, repro harness,…
- User
Build a GPU memory fragmentation diagnosis toolkit: telemetry capture, repro harness, heuristics, and code change recommendations.
- Assistant
GPU Memory Fragmentation Diagnosis Toolkit
This toolkit is designed for diagnosing GPU memory fragmentation issues, primarily targeting NVIDIA CUDA (with notes for extensibility to ROCm/Vulkan). Fragmentation occurs when free memory is abundant but scattered into small, non-contiguous blocks, preventing large allocations.
The toolkit includes:
- Telemetry Capture: Real-time and historical GPU memory metrics collection.
- Repro Harness: Minimal CUDA code to reproduce fragmentation scenarios.
- Heuristics: Python-based analyzer for diagnosing fragmentation from telemetry.
- Code Change Recommendations: Actionable fixes with code snippets.
Prerequisites
- NVIDIA GPU with CUDA 11+ (tested on RTX 40xx/A100).
- Python 3.8+,
numpy,pandas,matplotlib. - CUDA Toolkit.
- Install deps:
pip install numpy pandas matplotlib pynvml.
1. Telemetry Capture
Capture detailed memory stats: total/used/free memory, largest free block, fragmentation index (largest_free / total_free).
Python Telemetry Collector (
telemetry_capture.py)Uses
pynvmlfor system-level stats and custom CUDA interceptor for app-level.import pynvml import time import json import threading from ctypes import cdll, c_int, c_size_t # Custom CUDA memory tracker (intercept cudaMalloc/cudaFree) cuda_lib = cdll.LoadLibrary('libcuda.so') # Linux; use 'cudart64_*.dll' on Windows cudaMalloc = cuda_lib.cudaMalloc cudaFree = cuda_lib.cudaFree # Global trackers allocations = [] # list of (ptr, size) lock = threading.Lock() def tracked_cudaMalloc(ptr, size): res = cudaMalloc(ptr, size) with lock: allocations.append((ptr.value, size)) return res def tracked_cudaFree(ptr): res = cudaFree(ptr) with lock: allocations[:] = [(p, s) for p, s in allocations if p != ptr.value] return res # Hook (simplified; use LD_PRELOAD in prod for full intercept) # export LD_PRELOAD=./libmemhook.so # Compile hooks separately def capture_telemetry(gpu_id=0, duration=60, interval=0.1, output='telemetry.json'): pynvml.nvmlInit() handle = pynvml.nvmlDeviceGetHandleByIndex(gpu_id) data = [] start_time = time.time() while time.time() - start_time < duration: mem_info = pynvml.nvmlDeviceGetMemoryInfo(handle) # Estimate fragmentation (heuristic: query via custom API or approx) total_free = mem_info.free # largest_free approx via sorted free blocks (from allocations tracker) with lock: if allocations: used = sum(s for _, s in allocations) largest_free_est = max(0, total_free - (sum(s for _, s in allocations) % 1024*1024)) # Rough else: largest_free_est = total_free frag_index = largest_free_est / total_free if total_free > 0 else 0 data.append({ 'timestamp': time.time(), 'total': mem_info.total / 1024**2, # MB 'used': mem_info.used / 1024**2, 'free': mem_info.free / 1024**2, 'largest_free_est': largest_free_est / 1024**2, 'frag_index': frag_index, 'num_allocs': len(allocations) }) time.sleep(interval) pynvml.nvmlShutdown() with open(output, 'w') as f: json.dump(data, f) print(f"Telemetry saved to {output}") if __name__ == "__main__": capture_telemetry(duration=30)Usage:
- Run standalone:
python telemetry_capture.py - Integrate into app: Call
capture_telemetry()in a thread. - For precise tracking, compile a LD_PRELOAD hook (advanced; see NVIDIA docs).
Advanced: Use NVIDIA DCGM for production telemetry:
dcgmi diag -r 1 -j --fields gpuname,memory_used,memory_total # JSON output2. Repro Harness
Minimal CUDA program to reproduce fragmentation (
repro_harness.cu). Allocates small/large blocks in a pattern that fragments memory.#include <cuda_runtime.h> #include <stdio.h> #include <stdlib.h> int main() { const int num_small = 10000; const int small_size = 1024 * 1024; // 1MB small allocs const int large_size = 512 * 1024 * 1024; // 512MB large void *small_ptrs[num_small]; void *large_ptr = nullptr; size_t free_pre, largest_free_pre; cudaMemGetInfo(&free_pre, &largest_free_pre); // Dummy for baseline // Phase 1: Fragment with small allocs printf("Allocating %d small blocks...\n", num_small); for (int i = 0; i < num_small; ++i) { cudaMalloc(&small_ptrs[i], small_size); if (i % 1000 == 0) printf("Allocated %d small blocks\n", i); } // Phase 2: Free every other to create holes printf("Freeing every other small block...\n"); for (int i = 0; i < num_small; i += 2) { cudaFree(small_ptrs[i]); } // Phase 3: Try large alloc (likely fails due to frag) printf("Attempting large alloc (%ld MB)...\n", large_size / 1024 / 1024); cudaError_t err = cudaMalloc(&large_ptr, large_size); if (err != cudaSuccess) { printf("FAILED: %s (Fragmentation!)\n", cudaGetErrorString(err)); } else { printf("SUCCEEDED (low frag)\n"); cudaFree(large_ptr); } // Cleanup for (int i = 1; i < num_small; i += 2) cudaFree(small_ptrs[i]); size_t free_post, largest_free_post; cudaMemGetInfo(&free_post, &largest_free_post); printf("Post-free: Free=%ld MB, LargestFree=%ld MB\n", free_post / 1024 / 1024, largest_free_post / 1024 / 1024); return 0; }Build & Run:
nvcc repro_harness.cu -o repro_harness ./repro_harness # Run with telemetry capture in parallelExpected: Large alloc fails despite ~5GB free on 8GB GPU.
3. Heuristics Analyzer
Python script to analyze telemetry JSON (
heuristics_analyzer.py). Computes fragmentation score and flags issues.import json import pandas as pd import matplotlib.pyplot as plt import sys def analyze_frag(telemetry_file): with open(telemetry_file) as f: data = json.load(f) df = pd.DataFrame(data) # Heuristics df['frag_ratio'] = df['largest_free_est'] / df['free'] df['high_frag'] = df['frag_ratio'] < 0.5 # <50% largest vs total free = fragmented df['many_allocs'] = df['num_allocs'] > 1000 # Many small allocs df['crit_frag'] = (df['frag_ratio'] < 0.3) & (df['free'] > df['total'] * 0.2) # Alerts alerts = [] if df['crit_frag'].any(): alerts.append("CRITICAL: Severe fragmentation detected (frag_ratio < 0.3 with >20% free)") if df['many_allocs'].any() & (df['frag_ratio'] < 0.7).any(): alerts.append("WARNING: Many allocations correlating with fragmentation") print("Alerts:", alerts) print(df[['timestamp', 'free', 'largest_free_est', 'frag_ratio', 'high_frag']].tail()) # Plot fig, ax = plt.subplots(2,1, figsize=(12,8)) ax[0].plot(df['timestamp'], df['free'], label='Free MB') ax[0].plot(df['timestamp'], df['largest_free_est'], label='Largest Free MB') ax[0].legend() ax[1].plot(df['timestamp'], df['frag_ratio'], label='Frag Ratio') ax[1].axhline(0.5, color='r', linestyle='--', label='High Frag Threshold') ax[1].legend() plt.savefig('frag_analysis.png') plt.show() if __name__ == "__main__": analyze_frag(sys.argv[1])Usage:
python heuristics_analyzer.py telemetry.json4. Code Change Recommendations
Based on diagnosis:
Issue Recommendation Code Snippet Many small allocs Use memory pools (cudaMallocAsync + streams) or arenas. cuda<br>cudaStream_t stream;<br>cudaStreamCreate(&stream);<br>void *pool;<br>cudaMallocAsync(&pool, pool_size, stream); // Reuse slices<br>Interleaved alloc/free Batch allocs upfront; use unified memory sparingly. cuda<br>// Pre-alloc array of buffers<br>void *buffers[MAX_BATCH];<br>for(int i=0; i<MAX_BATCH; i++) cudaMalloc(&buffers[i], buf_size);<br>// Index-based reuse<br>High frag_ratio drop Align allocations (cudaMallocPitch); defrag periodically. cuda<br>size_t pitch;<br>cudaMallocPitch(&ptr, &pitch, width*sizeof(float), height);<br>OOM despite free mem Pre-allocate largest expected tensor; use cudaMemPool. cuda<br>cudaMemPool_t pool;<br>cudaMemPoolCreate(&pool, cudaMemPoolType::cudaMemPoolReuseStream);<br>cudaMemPoolSetAttribute(pool, cudaMemPoolAttrReleaseThreshold, threshold);<br>void *ptr; cudaMemPoolAlloc(&ptr, pool, size);<br>Multi-GPU Pin to specific GPUs; use MPS for sharing. CUDA_VISIBLE_DEVICES=0 ./appGeneral Best Practices:
- Monitor with
compute-sanitizer --tool memcheck ./app. - Use Triton/MIG for fine-grained control.
- For PyTorch/TensorFlow:
torch.cuda.empty_cache()sparingly; prefertorch.cuda.caching_allocator.
Toolkit Usage Workflow
- Run repro harness + telemetry capture concurrently.
- Analyze:
python heuristics_analyzer.py telemetry.json. - Apply recommendations to your codebase.
- Validate with reduced frag_ratio >0.8.
Extend for ROCm: Replace
pynvmlwithpyrocm, CUDA APIs with HIP. Questions? Provide your telemetry JSON for custom analysis.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.