Training Transformer Attention Entirely in FP4 with CroissAttn

Lower-precision training can reduce memory requirements, but rounding errors make the attention backward pass difficult to get right.

In this recorded Nebius Science Papers Club session, Alexander Johansen, PhD student in Computer Science at Stanford University, presents CroissAttn: A Complete FP4 Attention Backward. The method uses NVIDIA’s NVFP4 format for all six attention matrix multiplications across the forward and backward passes, while retaining higher precision where needed for normalization and accumulation.

Alexander introduces floating-point formats and stochastic rounding, then explains how CroissAttn adapts FlashAttention’s computation for 4-bit arithmetic. He discusses the observed training-quality tradeoffs, memory savings and the kernel optimization work still needed for practical adoption.

What you’ll learn

  • Why rounding and quantization complicate the attention backward pass
  • How stochastic rounding and softmax row correction address gradient bias
  • How CroissAttn reduces memory transfers through quantized storage and recomputation
  • What the experiments show and what remains under development

Who should watch

ML researchers and engineers working on low-precision training, attention kernels or memory-constrained language-model workloads. Familiarity with transformer attention and backpropagation will help; the session introduces the FP4 fundamentals.

Recommended reading: CroissAttn: A Complete FP4 Attention Backward

About Nebius Science Paper Club

Nebius Science Paper Club is a webinar series led by Nebius researchers. Each session brings together paper authors and practitioners to discuss new ideas, research, and discoveries in AI.

The session is open to researchers, engineers, students, and anyone curious about the paper.

Follow how AI and science co-evolve →

Try Nebius AI Cloud console today

Get immediate access to NVIDIA GPUs, along with CPU resources, storage and additional services through our user-friendly self-service console.