Inside the Nebius + PyTorch DeepSeek V3 recipe: NVSHMEM and DeepEP for wide expert parallelism

In 2026, nearly all frontier models are mixture-of-experts (MoE). Expert parallel (EP) is the most natural and efficient scheme to split the model up during training. However, enabling EP in training quickly reveals a key bottleneck: all-to-all communication that shuffles tokens between experts on every layer quickly dominates runtime. The irregular traffic pattern proves to be poorly suited to standard NCCL collectives, and most of the efficiency that makes MoE attractive disappears into communication overhead.

The alternative is GPU-initiated RDMA. NVSHMEM and DeepEP let GPUs read and write each other’s memory directly: across nodes, from inside CUDA kernels, with minimal CPU involvement.

On Nebius, adding DeepEP to a DeepSeek V3 training run on 256 B200 GPUs lifted model FLOPs utilization (MFU) by up to ~65% with no meaningful memory cost. This guide covers why MoE breaks collectives, the libraries that fix it, what we measured, and how to set it up on Nebius.

Why ML labs are switching to RDMAWhy ML labs are switching to RDMA

Collective communication, where multiple GPUs participate in a single coordinated data exchange, is the default for distributed ML. NCCL operations like all-reduce and all-gather deliver high throughput for predictable workloads, which makes them a natural fit for dense LLMs using data, tensor, and pipeline parallelism.

MoE breaks that assumption: the router decides at each step how much data goes where, so transfer sizes and destinations change from step to step. Forcing that into strict collectives is inefficient, and ML labs are increasingly turning to RDMA point-to-point instead.

Mixture-of-experts and wide expert parallelismMixture-of-experts and wide expert parallelism

In an MoE model, the dense feed-forward network (FFN) at the end of each Transformer block is replaced with a sparse block split into experts. On each token, only a small subset of experts (top-K) activates, which keeps compute cost low while preserving quality.

Left: dense LLM. Right: MoE, where the FFN becomes a set of routed experts

That sparsity makes MoEs cheap to serve but heavy on parameters, and it maps naturally onto expert parallelism (EP): partition the experts across GPUs so each holds only a subset. Modern MoEs are sparse enough that it’s often most efficient to scale EP across many nodes at a high EP degree — a technique known as Wide-EP.

DeepSeek V3 is a good example: 671B total parameters, but only 37B activate per token across 256 experts. With EP32 and DP32:

1. Each GPU holds just 8 of the 256 experts, using far less HBM.
2. With a high DP value, more ranks send tokens to fewer experts per GPU, so each expert processes more tokens per step, raising arithmetic intensity.
3. Each GPU loads weights simultaneously, increasing aggregate HBM bandwidth 4x versus a single-node deployment and easing the memory bottleneck proportionally.

The catch: the higher the EP degree, the more cross-node communication dominates runtime. Every MoE layer needs two all-to-all exchanges: a dispatch phase that sends tokens to their assigned experts, and a combine phase that returns expert outputs to their original positions.

Because each rank must talk to every other rank, message count grows quadratically with the number of workers. At a high EP degree, all-to-all becomes the bottleneck and the gains above go to waste, which is exactly why any communication improvement pays off end to end. NVSHMEM and DeepEP are how we capture it.

The building blocks: NVSHMEM, RDMA, GPUDirect, IBGDA, GDRCopyThe building blocks: NVSHMEM, RDMA, GPUDirect, IBGDA, GDRCopy

RDMA-based point-to-point lets GPUs read from and write to remote GPU memory directly, with minimal CPU involvement. Because it skips collective setup and synchronization, it can deliver much lower latency for the small, dynamic GPU-to-GPU transfers MoE generates.

NVSHMEM is NVIDIA’s library for GPU-initiated RDMA. It exposes a Partitioned Global Address Space (PGAS): each GPU maps a symmetric memory region that every other GPU can address directly, reading or writing any peer’s memory — within or across nodes — from inside CUDA kernels, without CPU involvement.

It builds on two primitives. RDMA lets a NIC access DRAM directly without CPU orchestration; GPUDirect RDMA extends that to read and write GPU HBM directly. NVSHMEM picks the fastest available fabric automatically: NVLink within a node, InfiniBand between nodes.

For MoE, where many small messages cross InfiniBand, latency is far higher than for the same operations over NVLink. NVSHMEM narrows the gap with InfiniBand GPUDirect Async (IBGDA). Standard GPUDirect RDMA still leaves the CPU in the loop to tell the NIC what to do; IBGDA removes that dependency and lets the GPU drive the NIC directly.

Both NVSHMEM and DeepEP also use GDRCopy, which gives the CPU RDMA access to GPU memory to speed up small CPU↔GPU transfers.

DeepEPDeepEP

DeepEP attacks the communication bottleneck directly, with purpose-built GPU-initiated kernels for dispatch and combine, built on NVSHMEM’s symmetric memory and RDMA. It ships two kernel families for different regimes: high-throughput kernels for training and prefill, and low-latency pure-RDMA kernels for decode. That low-latency path is especially useful for inference that combines Wide-EP with prefill/decode disaggregation. DeepEP also supports low-precision operations, including FP8. It was the first library of its kind, and it’s still the one others are measured against.

Benchmarking DeepEP on NebiusBenchmarking DeepEP on Nebius

We worked with PyTorch to build a high-performance DeepSeek V3 training recipe on TorchTitan, running on 256 B200 GPUs managed by Soperator.

The early numbers, for DeepSeek-16B (64 experts) and DeepSeek V3-671B (256 experts):