Training transformer attention entirely in FP4 with CroissAttn
Can transformer attention be trained entirely in 4-bit arithmetic without compromising training stability?
Lower-precision training can reduce memory and compute requirements, but the attention backward pass is particularly sensitive to rounding errors. In this live Nebius Science Papers Club session, Alexander Johansen, PhD student in Computer Science at Stanford University, will present CroissAttn: A Complete FP4 Attention Backward.
CroissAttn extends native FP4 execution to all six attention matrix multiplications in the forward and backward passes on NVIDIA Blackwell GPUs. Alexander will introduce the fundamentals of floating-point formats, NVFP4 and stochastic rounding before explaining the five modifications to FlashAttention behind the method.
He’ll also present training results across models from 181M to 8B parameters, along with speed and memory measurements for long-context workloads.
What you’ll learn
- Why low-precision arithmetic makes the attention backward pass difficult
- How stochastic rounding and softmax row correction work in CroissAttn
- What the method’s unbiasedness guarantee does—and does not—mean
- How FP4 training affects model quality, speed and memory use
- What CroissAttn could mean for FP4 deployment and NVIDIA GB300 GPUs
Who should attend
This session is designed for ML researchers and engineers working on low-precision training, attention kernels, model optimization or long-context systems.
Familiarity with transformer attention and backpropagation will help you follow the technical discussion. Previous experience with FP4 or stochastic rounding is useful but not required, as the session will introduce the core concepts.
Recommended reading: CroissAttn: A Complete FP4 Attention Backward
About Nebius Science Paper Club
Nebius Science Paper Club is a webinar series led by Nebius researchers. Each session brings together paper authors and practitioners to discuss new ideas, research, and discoveries in AI.
The session is open to researchers, engineers, students, and anyone interested in knowledge distillation, model compression, and computer vision.