Cross-Attention in Multimodal AI: When One Stream Needs to Read Another
A practical guide to cross-attention, from Q/K/V to Stable Diffusion’s text-conditioned U-Net and cross-modal dimension design.
Read More
The Decision Engine Behind Fused Attention: How Transformer Engine Orchestrates cuDNN on Blackwell
A source-grounded tour of Transformer Engine’s fused-attention dispatch: cuDNN Graph API, versioned backend eligibility, Blackwell paths, kernel-pipelining context, and deterministic training constraints.
Read MoreFMHA 101: From Transformer Attention to a CUDA Flash Attention Kernel
A beginner-friendly, code-oriented introduction to Transformer attention, multi-head attention, online softmax, and a tiled CUDA FMHA forward kernel.
Read More