Guide me what to do when NCU shows the below when profiling CUDA kernels - low memory…
- User
Guide me what to do when NCU shows the below when profiling CUDA kernels
- low memory throughput
- low DRAM throughput
- Low L1 throughput
- Low l2 throughput
- Compute (SM) throughput
- low achieved occupancy -> achieved occupancy is lower than theoretical occupancy
- Assistant
Interpreting Nsight Compute (NCU) “Low-Throughput” Warnings
Below is a pragmatic checklist you can walk through whenever NCU warns about low throughput on various memory/cache levels, SM utilization, and low achieved occupancy. The rules of thumb work for Ampere/Hopper, but 95 % of them also apply to Volta–Pascal hardware.
NCU Warning Why it happens First things to try Low global-memory (DRAM) throughput • Not enough concurrent warps to fully saturate memory pipes. • Strided / un-coalesced accesses (≤ 32 B per transaction). • Read-modify-write patterns causing RMW serialization. 1. Make sure each thread warp accesses consecutive 32/64/128 B chunks.2. Benchmark bigger thread-blocks (e.g., 128→256→512).3. Prefetch with asyncCopy/cp.asyncor__prefetch_global().4. Use 64-bit or 128-bit loads when possible (int2,float4).Low L2 throughput • Repeatedly loading the same data from DRAM because of poor temporal locality. • Large working-set > L2 capacity or conflict misses. 1. Reorder loops so that re-use fits in L2.2. Use tiling / blocking techniques.3. Combine loads into shared memory tile. Low L1 throughput • Working-set > L1 (approx. 128 kB/SM). • Mixed reads & writes on same lines → write-allocate traffic.• Consecutive diverging warps pollute cache. 1. Tile at thread-block level so each TB works on a subset that fits in L1.2. Separate read-only ( __ldg,const) and read-write data.3. Use__restrict__+constso compiler puts data in read-only cache.Low SM (Compute) throughput • Kernel is memory-bound (SMs stall on data). • Instruction-level-parallelism (ILP) is low ⇒ cycles wasted between dependent ops. 1. After fixing memory issues, examine arithmetic intensity; fuse kernels or add FLOPs where useful.2. Interleave independent computations (increase ILP).3. Check for divergent branches that force warp serialisation. Achieved occupancy ‹ Theoretical • Too many registers or shared memory per thread-block. • Launch grid too small. • Poor block size (e.g., 33 threads means only one warp). 1. Inspect ptxas line: reg: 76 smem: 8KB. Reduce registers (__launch_bounds__, fewer inlined vars, shuffle instead of store).2. Shrink shared-memory per block or switch to dynamic shared memory only where needed.3. Round blockDim to 128/256 threads; launch(#blocks ≥ 4× #SMs).4. Re-profile. If achieved occupancy > ~40 % and throughput rises, you’re done; otherwise keep iterating.
A Structured Optimization Workflow
1. Collect the facts
- Record an NCU session with:
ncu --set full --kernel-id yourKernel - Use the Roofline view (Nsight Compute > Menu > Roofline) to see if the dot for your kernel is:
• Below the red “memory-bound” ceiling → bandwidth problem
• Below the green “compute ceiling” → SM latency problem
• Stuck in lower-left corner → occupancy / latency hiding problem
2. Fix Occupancy First
Achieved occupancy < 25 % generally means the GPU cannot hide memory latency; bandwidth and compute numbers will look bad even if your ALU or DRAM is perfectly fine.
Action plan:
A. Check kernel registers (ptxas –v) - If > 120 on Ampere, force a register cap: __launch_bounds__(BLOCK_SIZE, 2) B. Re-tune block sizes: 128-256 threads C. Ensure you launch ≥ 4 × (#SM) blocksRe-profile; if throughput increases you know you were latency-bound.
3. Repair Memory Access Patterns
-
Visualize “Memory Workload Analysis” in NCU. Look at the “Access Pattern” pie: • 100 % coalesced – good
• Many 32-byte or 8-byte requests – your accesses are strided. -
Remedies: • Make structures of arrays (SoA) instead of AoS.
• Pad rows so each warp lands in the same 128 B segment.
• Combine scalar loads intofloat4/int4loads.
• Switch tocudaMemcpyAsync + cudaMemPrefetchAsyncfor page-migration issues on UVM.
4. Cache-Friendly Blocking
Shared memory is your manual L1:
__shared__ float tile[TILE_Y][TILE_X+1]; // +1 avoids bank conflict for (int m = 0; m < M; m += TILE_X) { cp.async.bulk(tile, &gmem[m], TILE_X); // Hopper/Ampere __syncthreads(); // compute }A 32×8 tile (256 threads) often fills both occupancy and coalescing requirements.
5. Balance Compute vs. Memory
If, after memory tuning, DRAM ∼ 70 % of peak but SM only 10 %, your kernel is genuinely memory-bound: stop optimizing compute and start algorithm-level work (compression, mixed precision, etc.).
Conversely, if SM ∼ 80 % but DRAM is 30 %, you’re compute-bound → drop precision (Tensor Cores), leverage math libraries (cuBLAS, CUTLASS).
Quick Reference Cheat-Sheet
Symptom One-liner Fix Achieved Occupancy < 30 % “Use fewer registers, bigger blocks, more blocks.” DRAM thruput < 50 % “Make loads coalesced, prefetch, double-buffer in shared memory.” L2 > 70 % hit but SM low “Increase ILP or fuse kernels — you’re latency-bound.” L1 hit rate < 20 % “Tile to reuse data inside block, or bypass with ld.global.cs.”Warps waiting on barrier “Reduce __syncthreads(), use warp-level primitives.”Divergent branches > 25 % “Refactor: make two kernels or use predication.”
Final sanity checks
- Compile with
-Xptxas=-v -lineinfo -dlcm=caand confirm compiler did not spill registers to local memory (look for.localin SASS). - Make sure ECC errors or PCIe throttling are not the real culprit (
nvidia-smi -i 0 -q -d ERRORS).
Tackle items in the order shown: Occupancy → Memory pattern → Cache locality → Compute balance. Re-profile after each step; small, iterative changes almost always beat giant rewrites 🚀.
- Record an NCU session with:
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.