cublasLtGroupedMatmulPlan_t groupPlans[3]; for (int i = 0; i < groupCount; i++) { cublasLtGroupedMatmulPlanInit(handle, matmulDesc, &groupPlans[i], CUDA_R_16F, CUDA_R_16F, CUDA_R_16F, CUDA_R_32F, m_arr[i], n, k); }
In the world of High-Performance Computing (HPC) and Deep Learning (DL), the General Matrix Multiply (GEMM) operation is the undisputed king. From large language models (LLMs) to scientific simulations, performance often hinges on how efficiently you can compute C = α*A*B + β*C . cublaslt grouped gemm
If you're building a transformer-based model, a recommender system, or any application that requires many small, independent matrix multiplications, Grouped GEMM should be your default choice. As NVIDIA continues to optimize cuBLASLt for Hopper and future architectures, the performance gap between irregular and regular workloads will only shrink further. For implementation details, refer to the NVIDIA cuBLASLt Developer Guide (CUDA 12.x and later). cublasLtGroupedMatmulPlan_t groupPlans[3]; for (int i = 0; i