
Training Transformer Attention Entirely in FP4 with CroissAttn
Lower-precision training can reduce memory requirements, but rounding errors make the attention backward pass difficult to get right.
In this recorded Nebius Science Papers Club session, Alexander Johansen, PhD student in Computer Science at Stanford University, presents CroissAttn: A Complete FP4 Attention Backward. The method uses NVIDIA’s NVFP4 format for all six attention matrix multiplications across the forward and backward passes, while retaining higher precision where needed for normalization and accumulation.
Alexander introduces floating-point formats and stochastic rounding, then explains how CroissAttn adapts FlashAttention’s computation for 4-bit arithmetic. He discusses the observed training-quality tradeoffs, memory savings and the kernel optimization work still needed for practical adoption.
What you’ll learn
- Why rounding and quantization complicate the attention backward pass
- How stochastic rounding and softmax row correction address gradient bias
- How CroissAttn reduces memory transfers through quantized storage and recomputation
- What the experiments show and what remains under development
Who should watch
ML researchers and engineers working on low-precision training, attention kernels or memory-constrained language-model workloads. Familiarity with transformer attention and backpropagation will help; the session introduces the FP4 fundamentals.
Recommended reading: CroissAttn: A Complete FP4 Attention Backward
About Nebius Science Paper Club
Nebius Science Paper Club is a webinar series led by Nebius researchers. Each session brings together paper authors and practitioners to discuss new ideas, research, and discoveries in AI.
The session is open to researchers, engineers, students, and anyone curious about the paper.