Hey, can you explain what's different between these two kernels and what is going on…
- User
Hey, can you explain what's different between these two kernels and what is going on here? #include <assert.h> #include <stdlib.h>
// CUDA runtime #include "helper.h" #include <cublas_v2.h> #include <cuda_runtime.h>
// a = mxk, b = kxn template <int BLOCK, int STRIDE> global void sgemm(int m, int n, int k, float *a, int lda, float *b, int ldb, float *c, int ldc) { // blockIdx control subpanel matrix constexpr int STEP = BLOCK * STRIDE; const int tx = threadIdx.x * STRIDE; const int ty = threadIdx.y * STRIDE; const int bx = blockIdx.x * STEP; const int by = blockIdx.y * STEP;
float *begin_a = a + by * k; float *begin_b = b + bx; float *end_a = begin_a + k;
float sum[STRIDE][STRIDE] = {0.f};
shared float ashare[STEP][2 * STEP]; shared float bshare[2 * STEP][STEP]; // bigger split for (float *a_ptr = begin_a, *b_ptr = begin_b; a_ptr < end_a; a_ptr += 2 * STEP, b_ptr += 2 * STEP * n) {
for (int i = 0; i < STRIDE; ++i) { for (int j = 0; j < STRIDE; ++j) { ashare[ty + i][tx + j] = a_ptr[(ty + i) * k + tx + j]; ashare[ty + i][tx + j + STEP] = a_ptr[(ty + i) * k + tx + j + STEP]; bshare[ty + i][tx + j] = b_ptr[(ty + i) * n + tx + j]; bshare[ty + i + STEP][tx + j] = b_ptr[(ty + i + STEP) * n + tx + j]; } } __syncthreads(); for (int i = 0; i < STRIDE; ++i) { for (int j = 0; j < STRIDE; ++j) { for (int kk = 0; kk < 2 * STEP; ++kk) { sum[i][j] += ashare[ty + i][kk] * bshare[kk][tx + j]; } } } __syncthreads();}
#pragma unroll for (int i = 0; i < STRIDE; ++i) { for (int j = 0; j < STRIDE; ++j) { c[(by + ty + i) * n + bx + tx + j] = sum[i][j]; } } }
void MY_MMult(cublasHandle_t handle, int m, int n, int k, float *d_A, int lda, float *d_B, int ldb, float *d_C, int ldc) {
constexpr int BLOCK = 16; constexpr int STRIDE = 2; // every thread calc STRIDExSTRIDE result dim3 block(BLOCK, BLOCK); dim3 grid((m + BLOCK - 1) / BLOCK / STRIDE, (n + BLOCK - 1) / BLOCK / STRIDE);
sgemm<BLOCK, STRIDE><<<grid, block>>>(m, n, k, d_A, lda, d_B, ldb, d_C, ldc); }#include <assert.h> #include <stdlib.h>
// CUDA runtime #include "helper.h" #include <cublas_v2.h> #include <cuda_runtime.h>
// a = mxk, b = kxn global void sgemm(int m, int n, int k, float *a, int lda, float *b, int ldb, float *c, int ldc) { const int tx = (threadIdx.x % 16) * 2; const int ty = threadIdx.x / 16 * 2; const int bx = blockIdx.x * 64; const int by = blockIdx.y * 64;
float *begin_a = a + by * k; float *begin_b = b + bx; float *end_a = begin_a + k;
shared float ashare[64][64]; shared float bshare[64][64]; float sum0[2][2] = {0}; float sum1[2][2] = {0}; float sum2[2][2] = {0}; float sum3[2][2] = {0};
// bigger split for (float *a_ptr = begin_a, *b_ptr = begin_b; a_ptr < end_a; a_ptr += 64, b_ptr += 64 * n) {
#pragma unroll for (int i = 0; i < 2; ++i) { for (int j = 0; j < 2; ++j) { ashare[ty + i][tx + j] = a_ptr[(ty + i) * k + tx + j]; ashare[ty + i][tx + j + 32] = a_ptr[(ty + i) * k + tx + j + 32]; ashare[ty + i + 32][tx + j] = a_ptr[(ty + 32 + i) * k + tx + j]; ashare[ty + i + 32][tx + j + 32] = a_ptr[(ty + 32 + i) * k + tx + j + 32];
bshare[ty + i][tx + j] = b_ptr[(ty + i) * n + tx + j]; bshare[ty + i][tx + j + 32] = b_ptr[(ty + i) * n + tx + j + 32]; bshare[ty + i + 32][tx + j] = b_ptr[(ty + i + 32) * n + tx + j]; bshare[ty + i + 32][tx + j + 32] = b_ptr[(ty + i + 32) * n + tx + j + 32]; } } __syncthreads();#pragma unroll for (int i = 0; i < 2; ++i) { for (int j = 0; j < 2; ++j) { for (int subk = 0; subk < 64; ++subk) { sum0[i][j] += ashare[ty + i][subk] * bshare[subk][tx + j]; sum1[i][j] += ashare[ty + i][subk] * bshare[subk][tx + j + 32]; sum2[i][j] += ashare[ty + i + 32][subk] * bshare[subk][tx + j]; sum3[i][j] += ashare[ty + i + 32][subk] * bshare[subk][tx + j + 32]; } } } __syncthreads(); }
#pragma unroll for (int i = 0; i < 2; ++i) { for (int j = 0; j < 2; ++j) { c[(by + ty + i) * n + bx + tx + j] = sum0[i][j]; c[(by + ty + i) * n + bx + tx + 32 + j] = sum1[i][j]; c[(by + ty + i + 32) * n + bx + tx + j] = sum2[i][j]; c[(by + ty + i + 32) * n + bx + tx + 32 + j] = sum3[i][j]; } } }
void MY_MMult(cublasHandle_t handle, int m, int n, int k, float *d_A, int lda, float *d_B, int ldb, float *d_C, int ldc) {
dim3 block(256); dim3 grid(m / 64, n / 64);
sgemm<<<grid, block>>>(m, n, k, d_A, lda, d_B, ldb, d_C, ldc); }
- Assistant
The two CUDA kernels you've provided implement single-precision general matrix multiplication (SGEMM) but use different strategies in terms of tiling, thread mapping, data loading, and loop unrolling to optimize performance. Here's a detailed explanation of the differences between the two kernels and what's going on in each.
First Kernel (
sgemmwith template parameters)Key Features:
-
Templated Parameters: The kernel uses template parameters
BLOCKandSTRIDEto define tile sizes and thread strides, offering flexibility in configuring the computation. -
Thread Block Configuration:
- Uses a 2D thread block with dimensions
(BLOCK, BLOCK)(e.g., 16 x 16 threads). - Each thread computes a sub-block of the output matrix of size
STRIDE x STRIDE(e.g., 2 x 2 elements).
- Uses a 2D thread block with dimensions
-
Shared Memory Usage:
- Allocates shared memory tiles:
ashare[STEP][2 * STEP], whereSTEP = BLOCK * STRIDE(e.g., 32 x 64 elements).bshare[2 * STEP][STEP](e.g., 64 x 32 elements).
- Total shared memory per block is relatively smaller (e.g., 16 KB) compared to the second kernel.
- Allocates shared memory tiles:
-
Data Loading Strategy:
- Each thread loads multiple elements into shared memory:
- Loads elements from
aintoashareusing the thread'stxandty, offset bySTRIDE. - Similarly loads elements from
bintobshare.
- Loads elements from
- The data loading loops iterate over
iandjfrom0toSTRIDE - 1.
- Each thread loads multiple elements into shared memory:
-
Computation:
- Performs matrix multiplication using nested loops over
i,j, andkk. - Accumulates partial sums in
sum[i][j]for each thread.
- Performs matrix multiplication using nested loops over
-
Flexibility:
- The use of templates allows this kernel to be tuned for different hardware or matrix sizes by adjusting
BLOCKandSTRIDE.
- The use of templates allows this kernel to be tuned for different hardware or matrix sizes by adjusting
Second Kernel (
sgemmwith fixed sizes)Key Features:
-
Fixed Tile Sizes:
- The kernel uses fixed tile sizes, specifically designed for a tile of size 64 x 64.
-
Thread Block Configuration:
- Uses a 1D thread block with 256 threads (
blockDim.x = 256). - Thread indices
txandtyare calculated using:tx = (threadIdx.x % 16) * 2ty = (threadIdx.x / 16) * 2
- This effectively maps the 1D thread block into a 2D arrangement with strides.
- Uses a 1D thread block with 256 threads (
-
Shared Memory Usage:
- Allocates larger shared memory tiles:
ashare[64][64]andbshare[64][64].
- Total shared memory per block is larger (e.g., 32 KB).
- Allocates larger shared memory tiles:
-
Data Loading Strategy:
- Each thread loads multiple elements (up to 8) into shared memory:
- Loads elements from
aintoashareat different offsets to cover all sub-tiles. - Similarly loads elements from
bintobshare.
- Loads elements from
- Uses unrolled loops and manual indexing to efficiently utilize memory bandwidth.
- Each thread loads multiple elements (up to 8) into shared memory:
-
Computation:
- Maintains multiple sum arrays (
sum0,sum1,sum2,sum3) to hold partial results. - Performs computation over larger tiles, unrolling loops to increase instruction-level parallelism.
- The nested loops over
i,j, andsubkaccumulate the results into the sum arrays.
- Maintains multiple sum arrays (
-
Optimization Techniques:
- Aggressive loop unrolling (using
#pragma unroll) to reduce loop overhead. - Increased per-thread workload to improve computational intensity (higher arithmetic operations per memory operation).
- Aggressive loop unrolling (using
What's Going On Here?
Different Tiling Strategies:
- First Kernel: Uses smaller tiles (e.g., 32 x 32) with flexibility in adjusting the tile size via templates.
- Second Kernel: Uses larger fixed tiles (64 x 64) to increase data reuse and reduce the number of iterations needed over the input matrices.
Thread Mapping Differences:
- First Kernel: Employs a 2D thread block where each thread is responsible for computing a small sub-block of the result matrix.
- Second Kernel: Maps a 1D block of threads into a 2D arrangement with custom calculations for
txandty, effectively handling a larger portion of the computation per thread.
Data Loading Techniques:
-
First Kernel:
- Each thread loads fewer elements into shared memory, spreading the data loading across more threads.
- Uses stride-based indexing to cover the necessary elements.
-
Second Kernel:
- Each thread loads more elements (up to 8), decreasing the total number of threads needed for data loading.
- By loading more data per thread, it aims to improve memory throughput.
Loop Unrolling and Computation:
-
First Kernel:
- Uses standard nested loops for computation with partial sums accumulated in
sum[i][j].
- Uses standard nested loops for computation with partial sums accumulated in
-
Second Kernel:
- Employs loop unrolling (with
#pragma unroll) and multiple sum arrays to increase instruction-level parallelism. - By unrolling loops, it reduces loop control overhead and allows the compiler to optimize the computation better.
- Employs loop unrolling (with
Shared Memory Utilization:
- First Kernel: Uses less shared memory per block, which might allow for more concurrent blocks on a streaming multiprocessor (SM) but might result in less data reuse within a block.
- Second Kernel: Uses more shared memory to increase data reuse within a block, potentially improving performance if the hardware supports it.
Performance Trade-offs:
-
First Kernel:
- Offers more flexibility and might perform better on devices with limited shared memory per SM.
- The smaller tile size may result in more iterations and potentially less efficient use of computational resources.
-
Second Kernel:
- Optimized for performance with larger tiles, more aggressive loop unrolling, and increased shared memory usage.
- May achieve higher performance on devices with ample shared memory and compute capabilities.
Conclusion
The two kernels represent different optimization strategies for implementing matrix multiplication on CUDA-capable GPUs:
-
The first kernel focuses on flexibility and balance between resource usage and computational workload, making it adaptable to various hardware configurations.
-
The second kernel aims for maximum performance by increasing computational intensity and data reuse at the expense of higher shared memory usage.
By employing different tiling sizes, thread mappings, data loading patterns, and loop unrolling techniques, each kernel attempts to optimize the use of GPU resources to achieve efficient matrix multiplication. The differences highlight the trade-offs between flexibility, resource utilization, and performance optimization in GPU programming.
-