Skip to content
TheChatArchiveThe Chat Archive
o3-2025-04-16· OpenAIHugging Face LMArena Human Preference 140K

Guide me what to do when NCU shows the below when profiling CUDA kernels - low memory…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with grok-4-0709’s answer.
  1. User

    Guide me what to do when NCU shows the below when profiling CUDA kernels

    • low memory throughput
    • low DRAM throughput
    • Low L1 throughput
    • Low l2 throughput
    • Compute (SM) throughput
    • low achieved occupancy -> achieved occupancy is lower than theoretical occupancy
  2. Assistant

    Interpreting Nsight Compute (NCU) “Low-Throughput” Warnings

    Below is a pragmatic checklist you can walk through whenever NCU warns about low throughput on various memory/cache levels, SM utilization, and low achieved occupancy. The rules of thumb work for Ampere/Hopper, but 95 % of them also apply to Volta–Pascal hardware.

    NCU WarningWhy it happensFirst things to try
    Low global-memory (DRAM) throughput• Not enough concurrent warps to fully saturate memory pipes. • Strided / un-coalesced accesses (≤ 32 B per transaction). • Read-modify-write patterns causing RMW serialization.1. Make sure each thread warp accesses consecutive 32/64/128 B chunks.2. Benchmark bigger thread-blocks (e.g., 128→256→512).3. Prefetch with asyncCopy / cp.async or __prefetch_global().4. Use 64-bit or 128-bit loads when possible (int2, float4).
    Low L2 throughput• Repeatedly loading the same data from DRAM because of poor temporal locality. • Large working-set > L2 capacity or conflict misses.1. Reorder loops so that re-use fits in L2.2. Use tiling / blocking techniques.3. Combine loads into shared memory tile.
    Low L1 throughput• Working-set > L1 (approx. 128 kB/SM). • Mixed reads & writes on same lines → write-allocate traffic.• Consecutive diverging warps pollute cache.1. Tile at thread-block level so each TB works on a subset that fits in L1.2. Separate read-only (__ldg, const) and read-write data.3. Use __restrict__ + const so compiler puts data in read-only cache.
    Low SM (Compute) throughput• Kernel is memory-bound (SMs stall on data). • Instruction-level-parallelism (ILP) is low ⇒ cycles wasted between dependent ops.1. After fixing memory issues, examine arithmetic intensity; fuse kernels or add FLOPs where useful.2. Interleave independent computations (increase ILP).3. Check for divergent branches that force warp serialisation.
    Achieved occupancy ‹ Theoretical• Too many registers or shared memory per thread-block. • Launch grid too small. • Poor block size (e.g., 33 threads means only one warp).1. Inspect ptxas line: reg: 76 smem: 8KB. Reduce registers (__launch_bounds__, fewer inlined vars, shuffle instead of store).2. Shrink shared-memory per block or switch to dynamic shared memory only where needed.3. Round blockDim to 128/256 threads; launch (#blocks ≥ 4× #SMs).4. Re-profile. If achieved occupancy > ~40 % and throughput rises, you’re done; otherwise keep iterating.

    A Structured Optimization Workflow

    1. Collect the facts

    1. Record an NCU session with:
      ncu --set full --kernel-id yourKernel
      
    2. Use the Roofline view (Nsight Compute > Menu > Roofline) to see if the dot for your kernel is: • Below the red “memory-bound” ceiling → bandwidth problem
      • Below the green “compute ceiling” → SM latency problem
      • Stuck in lower-left corner → occupancy / latency hiding problem

    2. Fix Occupancy First

    Achieved occupancy < 25 % generally means the GPU cannot hide memory latency; bandwidth and compute numbers will look bad even if your ALU or DRAM is perfectly fine.

    Action plan:

    A. Check kernel registers (ptxas –v)
       - If > 120 on Ampere, force a register cap:
         __launch_bounds__(BLOCK_SIZE, 2)
    B. Re-tune block sizes: 128-256 threads
    C. Ensure you launch ≥ 4 × (#SM) blocks
    

    Re-profile; if throughput increases you know you were latency-bound.

    3. Repair Memory Access Patterns

    1. Visualize “Memory Workload Analysis” in NCU. Look at the “Access Pattern” pie: • 100 % coalesced – good
      • Many 32-byte or 8-byte requests – your accesses are strided.

    2. Remedies: • Make structures of arrays (SoA) instead of AoS.
      • Pad rows so each warp lands in the same 128 B segment.
      • Combine scalar loads into float4 / int4 loads.
      • Switch to cudaMemcpyAsync + cudaMemPrefetchAsync for page-migration issues on UVM.

    4. Cache-Friendly Blocking

    Shared memory is your manual L1:

    __shared__ float tile[TILE_Y][TILE_X+1]; // +1 avoids bank conflict
    for (int m = 0; m < M; m += TILE_X)
    {
        cp.async.bulk(tile, &gmem[m], TILE_X);  // Hopper/Ampere
        __syncthreads();
        // compute
    }
    

    A 32×8 tile (256 threads) often fills both occupancy and coalescing requirements.

    5. Balance Compute vs. Memory

    If, after memory tuning, DRAM ∼ 70 % of peak but SM only 10 %, your kernel is genuinely memory-bound: stop optimizing compute and start algorithm-level work (compression, mixed precision, etc.).
    Conversely, if SM ∼ 80 % but DRAM is 30 %, you’re compute-bound → drop precision (Tensor Cores), leverage math libraries (cuBLAS, CUTLASS).


    Quick Reference Cheat-Sheet

    SymptomOne-liner Fix
    Achieved Occupancy < 30 %“Use fewer registers, bigger blocks, more blocks.”
    DRAM thruput < 50 %“Make loads coalesced, prefetch, double-buffer in shared memory.”
    L2 > 70 % hit but SM low“Increase ILP or fuse kernels — you’re latency-bound.”
    L1 hit rate < 20 %“Tile to reuse data inside block, or bypass with ld.global.cs.”
    Warps waiting on barrier“Reduce __syncthreads(), use warp-level primitives.”
    Divergent branches > 25 %“Refactor: make two kernels or use predication.”

    Final sanity checks

    1. Compile with -Xptxas=-v -lineinfo -dlcm=ca and confirm compiler did not spill registers to local memory (look for .local in SASS).
    2. Make sure ECC errors or PCIe throttling are not the real culprit (nvidia-smi -i 0 -q -d ERRORS).

    Tackle items in the order shown: Occupancy → Memory pattern → Cache locality → Compute balance. Re-profile after each step; small, iterative changes almost always beat giant rewrites 🚀.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.