Recraft V4: Scaling image gen with the NVIDIA HGX B200 cluster in Nebius AI Cloud

Recraft has an extensive history with Nebius that spans successful deployments of NVIDIA Ampere, Hopper and Blackwell platforms. They were one of the first customers on the Nebius AI Cloud platform and early adopters of NVIDIA Blackwell GPUs at Nebius. After successfully adopting NVIDIA HGX B200, they have moved on to testing NVIDIA HGX B300 instances on their workloads.

They emphasize model quality through scale, building large foundation models for creating high-quality visual content. “For image generation, the bigger the model, the better the quality, ” explains Pavel Ostyakov, Head of AI at Recraft. “When we talk about model size, we mean more than just the number of parameters. In our experience, the most important factor is the quantity of operations, or the number of FLOPS the model uses internally in order to generate the output. With NVIDIA Blackwell GPUs, we were able to scale the model significantly in both operations and parameters — in this case, we’re talking about dozens of billions of parameters.”

This new architecture offers enhanced training performance and inference capabilities, with LLM training up to four times faster than on previous generation. To power Recraft’s model training, Nebius provided a cluster with NVIDIA Blackwell GPUs, interconnected via NVIDIA Quantum-2 InfiniBand.

The transition from NVIDIA Hopper to NVIDIA Blackwell was straightforward for Recraft. Their PyTorch-based software stack was compatible with NVIDIA containers, which provide precompiled libraries that work out of the box. Distributed training workloads didn’t require any changes to the GPU communication software stack, with the InfiniBand network providing the same reliable performance as on previous generations of GPUs.

The Recraft team made a few customizations to their training setup to improve production performance on the new architecture, such as rewriting a portion of code to switch from compiling networks with TensorRT to using torch.compile directly.

“If you have simple code, you can switch to the new GPUs without doing anything special, ” Ostyakov noted.

Nebius infrastructure facilitates hardware upgrades with an optimized software stack and preventative maintenance. The Nebius AI Cloud platform provides improved cluster observability through robust monitoring services and detailed logs, enabling Recraft to quickly identify and resolve performance issues.

Recraft engineers experienced minimal training interruptions due to efficient checkpoint and restart procedures. This can be attributed to Nebius’s automated maintenance systems and fault detection. When an InfiniBand link or GPU goes offline, the system automatically replaces the node and restarts training from the last checkpoint, minimizing disruption to multi-day training runs.

Recraft also collaborated with the in-house engineering team at Nebius throughout the transition to NVIDIA Blackwell GPUs, relying on their experience with hardware upgrades and deep expertise in this newest architecture to help solve issues that came up during early access testing.

This latest generation of GPUs presents new alternatives for training precision, with support for both FP8 and FP4 quantization potentially enabling faster performance.

Recraft selects training precision formats based on workload-specific quality requirements. They primarily use bfloat16 for model training, where precision is critical for image generation quality. However, they deploy FP8 for running the text encoder component in their model pipeline — since these weights remain frozen during pre-training, the lower precision enables faster processing of samples without compromising output quality.

For LLM inference workloads, Recraft deploys FP8 precision, consistent with their NVIDIA Hopper implementation. The team is now experimenting with NVFP4, NVIDIA Blackwell’s native 4-bit format, to optimize inference speed.

• Easy upgrade to frontier chips: Migrated from NVIDIA Hopper to Blackwell with minimal code changes, running distributed training out of the box.

• Scaling image generation: Successfully deployed NVIDIA Blackwell GPUs for scaling their model, validating this hardware for large-scale training of image generation models.

• Training precision optimization: Deployed workload-specific precision formats on NVIDIA Blackwell GPUs — BF16 for the majority of training, FP8 to accelerate training the text encoder and experiments with NVFP4 for faster inference.

• Reliable infrastructure and hands-on support: Minimized training interruptions through automated fault detection and checkpoint recovery in the Nebius AI Cloud. Nebius engineers were on hand to solve technical challenges for faster onboarding to the new generation.