FMHA 101: From Transformer Attention to a CUDA Flash Attention Kernel
A beginner-friendly, code-oriented introduction to Transformer attention, multi-head attention, online softmax, and a tiled CUDA FMHA forward kernel.
Read More
NVIDIA GPU Operator: A Comprehensive Deep Dive
An in-depth technical exploration of NVIDIA GPU Operator, covering its architecture, components, lifecycle, and advanced features. Learn how Driver Manager evolved from 264 to 876+ lines, understand the JIT compilation model, and master production deployment strategies.
Read More