Skip to content
TheChatArchiveThe Chat Archive
Grok 4.1 Fast· xaiHugging Face Frontier Reasoning Traces

Build a GPU memory fragmentation diagnosis toolkit: telemetry capture, repro harness,…

  1. User

    Build a GPU memory fragmentation diagnosis toolkit: telemetry capture, repro harness, heuristics, and code change recommendations.

  2. Assistant

    GPU Memory Fragmentation Diagnosis Toolkit

    This toolkit is designed for diagnosing GPU memory fragmentation issues, primarily targeting NVIDIA CUDA (with notes for extensibility to ROCm/Vulkan). Fragmentation occurs when free memory is abundant but scattered into small, non-contiguous blocks, preventing large allocations.

    The toolkit includes:

    1. Telemetry Capture: Real-time and historical GPU memory metrics collection.
    2. Repro Harness: Minimal CUDA code to reproduce fragmentation scenarios.
    3. Heuristics: Python-based analyzer for diagnosing fragmentation from telemetry.
    4. Code Change Recommendations: Actionable fixes with code snippets.

    Prerequisites

    • NVIDIA GPU with CUDA 11+ (tested on RTX 40xx/A100).
    • Python 3.8+, numpy, pandas, matplotlib.
    • CUDA Toolkit.
    • Install deps: pip install numpy pandas matplotlib pynvml.

    1. Telemetry Capture

    Capture detailed memory stats: total/used/free memory, largest free block, fragmentation index (largest_free / total_free).

    Python Telemetry Collector (telemetry_capture.py)

    Uses pynvml for system-level stats and custom CUDA interceptor for app-level.

    import pynvml
    import time
    import json
    import threading
    from ctypes import cdll, c_int, c_size_t
    
    # Custom CUDA memory tracker (intercept cudaMalloc/cudaFree)
    cuda_lib = cdll.LoadLibrary('libcuda.so')  # Linux; use 'cudart64_*.dll' on Windows
    cudaMalloc = cuda_lib.cudaMalloc
    cudaFree = cuda_lib.cudaFree
    
    # Global trackers
    allocations = []  # list of (ptr, size)
    lock = threading.Lock()
    
    def tracked_cudaMalloc(ptr, size):
        res = cudaMalloc(ptr, size)
        with lock:
            allocations.append((ptr.value, size))
        return res
    
    def tracked_cudaFree(ptr):
        res = cudaFree(ptr)
        with lock:
            allocations[:] = [(p, s) for p, s in allocations if p != ptr.value]
        return res
    
    # Hook (simplified; use LD_PRELOAD in prod for full intercept)
    # export LD_PRELOAD=./libmemhook.so  # Compile hooks separately
    
    def capture_telemetry(gpu_id=0, duration=60, interval=0.1, output='telemetry.json'):
        pynvml.nvmlInit()
        handle = pynvml.nvmlDeviceGetHandleByIndex(gpu_id)
        
        data = []
        start_time = time.time()
        
        while time.time() - start_time < duration:
            mem_info = pynvml.nvmlDeviceGetMemoryInfo(handle)
            
            # Estimate fragmentation (heuristic: query via custom API or approx)
            total_free = mem_info.free
            # largest_free approx via sorted free blocks (from allocations tracker)
            with lock:
                if allocations:
                    used = sum(s for _, s in allocations)
                    largest_free_est = max(0, total_free - (sum(s for _, s in allocations) % 1024*1024))  # Rough
                else:
                    largest_free_est = total_free
            
            frag_index = largest_free_est / total_free if total_free > 0 else 0
            
            data.append({
                'timestamp': time.time(),
                'total': mem_info.total / 1024**2,  # MB
                'used': mem_info.used / 1024**2,
                'free': mem_info.free / 1024**2,
                'largest_free_est': largest_free_est / 1024**2,
                'frag_index': frag_index,
                'num_allocs': len(allocations)
            })
            
            time.sleep(interval)
        
        pynvml.nvmlShutdown()
        with open(output, 'w') as f:
            json.dump(data, f)
        print(f"Telemetry saved to {output}")
    
    if __name__ == "__main__":
        capture_telemetry(duration=30)
    

    Usage:

    • Run standalone: python telemetry_capture.py
    • Integrate into app: Call capture_telemetry() in a thread.
    • For precise tracking, compile a LD_PRELOAD hook (advanced; see NVIDIA docs).

    Advanced: Use NVIDIA DCGM for production telemetry:

    dcgmi diag -r 1 -j --fields gpuname,memory_used,memory_total  # JSON output
    

    2. Repro Harness

    Minimal CUDA program to reproduce fragmentation (repro_harness.cu). Allocates small/large blocks in a pattern that fragments memory.

    #include <cuda_runtime.h>
    #include <stdio.h>
    #include <stdlib.h>
    
    int main() {
        const int num_small = 10000;
        const int small_size = 1024 * 1024;  // 1MB small allocs
        const int large_size = 512 * 1024 * 1024;  // 512MB large
    
        void *small_ptrs[num_small];
        void *large_ptr = nullptr;
    
        size_t free_pre, largest_free_pre;
        cudaMemGetInfo(&free_pre, &largest_free_pre);  // Dummy for baseline
    
        // Phase 1: Fragment with small allocs
        printf("Allocating %d small blocks...\n", num_small);
        for (int i = 0; i < num_small; ++i) {
            cudaMalloc(&small_ptrs[i], small_size);
            if (i % 1000 == 0) printf("Allocated %d small blocks\n", i);
        }
    
        // Phase 2: Free every other to create holes
        printf("Freeing every other small block...\n");
        for (int i = 0; i < num_small; i += 2) {
            cudaFree(small_ptrs[i]);
        }
    
        // Phase 3: Try large alloc (likely fails due to frag)
        printf("Attempting large alloc (%ld MB)...\n", large_size / 1024 / 1024);
        cudaError_t err = cudaMalloc(&large_ptr, large_size);
        if (err != cudaSuccess) {
            printf("FAILED: %s (Fragmentation!)\n", cudaGetErrorString(err));
        } else {
            printf("SUCCEEDED (low frag)\n");
            cudaFree(large_ptr);
        }
    
        // Cleanup
        for (int i = 1; i < num_small; i += 2) cudaFree(small_ptrs[i]);
        
        size_t free_post, largest_free_post;
        cudaMemGetInfo(&free_post, &largest_free_post);
        printf("Post-free: Free=%ld MB, LargestFree=%ld MB\n", 
               free_post / 1024 / 1024, largest_free_post / 1024 / 1024);
    
        return 0;
    }
    

    Build & Run:

    nvcc repro_harness.cu -o repro_harness
    ./repro_harness  # Run with telemetry capture in parallel
    

    Expected: Large alloc fails despite ~5GB free on 8GB GPU.

    3. Heuristics Analyzer

    Python script to analyze telemetry JSON (heuristics_analyzer.py). Computes fragmentation score and flags issues.

    import json
    import pandas as pd
    import matplotlib.pyplot as plt
    import sys
    
    def analyze_frag(telemetry_file):
        with open(telemetry_file) as f:
            data = json.load(f)
        
        df = pd.DataFrame(data)
        
        # Heuristics
        df['frag_ratio'] = df['largest_free_est'] / df['free']
        df['high_frag'] = df['frag_ratio'] < 0.5  # <50% largest vs total free = fragmented
        df['many_allocs'] = df['num_allocs'] > 1000  # Many small allocs
        df['crit_frag'] = (df['frag_ratio'] < 0.3) & (df['free'] > df['total'] * 0.2)
        
        # Alerts
        alerts = []
        if df['crit_frag'].any():
            alerts.append("CRITICAL: Severe fragmentation detected (frag_ratio < 0.3 with >20% free)")
        if df['many_allocs'].any() & (df['frag_ratio'] < 0.7).any():
            alerts.append("WARNING: Many allocations correlating with fragmentation")
        
        print("Alerts:", alerts)
        print(df[['timestamp', 'free', 'largest_free_est', 'frag_ratio', 'high_frag']].tail())
        
        # Plot
        fig, ax = plt.subplots(2,1, figsize=(12,8))
        ax[0].plot(df['timestamp'], df['free'], label='Free MB')
        ax[0].plot(df['timestamp'], df['largest_free_est'], label='Largest Free MB')
        ax[0].legend()
        ax[1].plot(df['timestamp'], df['frag_ratio'], label='Frag Ratio')
        ax[1].axhline(0.5, color='r', linestyle='--', label='High Frag Threshold')
        ax[1].legend()
        plt.savefig('frag_analysis.png')
        plt.show()
    
    if __name__ == "__main__":
        analyze_frag(sys.argv[1])
    

    Usage: python heuristics_analyzer.py telemetry.json

    4. Code Change Recommendations

    Based on diagnosis:

    IssueRecommendationCode Snippet
    Many small allocsUse memory pools (cudaMallocAsync + streams) or arenas.cuda<br>cudaStream_t stream;<br>cudaStreamCreate(&stream);<br>void *pool;<br>cudaMallocAsync(&pool, pool_size, stream); // Reuse slices<br>
    Interleaved alloc/freeBatch allocs upfront; use unified memory sparingly.cuda<br>// Pre-alloc array of buffers<br>void *buffers[MAX_BATCH];<br>for(int i=0; i<MAX_BATCH; i++) cudaMalloc(&buffers[i], buf_size);<br>// Index-based reuse<br>
    High frag_ratio dropAlign allocations (cudaMallocPitch); defrag periodically.cuda<br>size_t pitch;<br>cudaMallocPitch(&ptr, &pitch, width*sizeof(float), height);<br>
    OOM despite free memPre-allocate largest expected tensor; use cudaMemPool.cuda<br>cudaMemPool_t pool;<br>cudaMemPoolCreate(&pool, cudaMemPoolType::cudaMemPoolReuseStream);<br>cudaMemPoolSetAttribute(pool, cudaMemPoolAttrReleaseThreshold, threshold);<br>void *ptr; cudaMemPoolAlloc(&ptr, pool, size);<br>
    Multi-GPUPin to specific GPUs; use MPS for sharing.CUDA_VISIBLE_DEVICES=0 ./app

    General Best Practices:

    • Monitor with compute-sanitizer --tool memcheck ./app.
    • Use Triton/MIG for fine-grained control.
    • For PyTorch/TensorFlow: torch.cuda.empty_cache() sparingly; prefer torch.cuda.caching_allocator.

    Toolkit Usage Workflow

    1. Run repro harness + telemetry capture concurrently.
    2. Analyze: python heuristics_analyzer.py telemetry.json.
    3. Apply recommendations to your codebase.
    4. Validate with reduced frag_ratio >0.8.

    Extend for ROCm: Replace pynvml with pyrocm, CUDA APIs with HIP. Questions? Provide your telemetry JSON for custom analysis.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.