Serving world models: Request latency and node throughput tradeoffs

Serving NVIDIA Cosmos 3 Super as eight single-GPU replicas on one NVIDIA HGX B200 node produced 42.5% more valid video per node-hour than one replica spanning all 8 GPUs, but at 5.5 times the mean request latency: 378.1 seconds versus 68.5. At the same hourly node price, that is 30% lower node cost per video-second.

We measured the text-to-video generator path of NVIDIA Cosmos 3 Super (nvidia/Cosmos3-Super), served through vLLM-Omni, across four serving topologies on NVIDIA HGX B200 and NVIDIA HGX H200 nodes on Nebius AI Cloud.

Video-generation benchmarks need a node-level output metricVideo-generation benchmarks need a node-level output metric

A video-generation serving benchmark needs two metrics: request latency and node throughput. LLM inference stacks, such as those behind coding assistants, are typically optimized for time to first token and tokens per second. A synchronous video endpoint returns nothing until the clip is generated, encoded, and delivered as a finished file, so request latency runs from dispatch to the completed file.

NVIDIA’s Cosmos accelerates synthetic video generation and dataset augmentation workflows for robotics and autonomous systems. Because these workflows generate many clips, request latency tells only part of the story. A serving benchmark also needs to measure how much valid video the node produces over the full job. We define node throughput as technically validated video-seconds per node-hour.

We calculate node throughput by adding the duration of every clip that passes the validation gate, dividing that total by the job’s wall-clock time, and scaling the result to one hour. The clock starts when the first request is sent and stops when the final request finishes. Failed or rejected attempts add no video-seconds, but their time still counts toward the wall clock and lowers throughput.

A clip passes the gate only if the request succeeds and the resulting MP4 is non-empty, decodes fully, has the expected shape, and clears the blank-frame and frozen-frame checks. This gate confirms file integrity and basic motion. A separate quality evaluation is needed to judge prompt adherence, visual quality, temporal consistency, physical plausibility, realism, and training utility.

Comparable benchmarks require a defined control setComparable benchmarks require a defined control set

Latency and node-throughput results are comparable only when the benchmark controls are clear. Video generation introduces many variables that can change both metrics, including the model, prompt, output shape, sampling settings, software stack, request schedule, timing boundary, and validation gate. For this benchmark, we fixed the model, software, input, and generation settings:

• Model: nvidia/Cosmos3-Super at one pinned revision

• Software: the vllm/vllm-omni:cosmos3 container (vLLM 0.25.0), pinned by digest

• Input: one fixed prompt and negative prompt, with three fixed random seeds used equally

• Generation: BF16, 1280 × 720 output, 189 frames at 24 frames per second, 35 denoising steps, and guardrails disabled

For each topology run, we sent 24 measured requests, each generating one clip from that prompt. Each replica completed one warmup request, then processed one request at a time. The measured window included routing, generation, MP4 encoding, and time spent waiting for the last replica to finish, and excluded startup, model loading, and warmup.

Our public benchmark method provides the exact workload, validation gate, model revision, container digest, software stack, and host configuration behind these runs. The full HGX B200 and HGX H200 records together contain 408 attempts, including the concurrency tests and repeat runs. All 408 passed the video-validation checks, so the gate did not change the throughput reported here.

Serving topology controls how the node divides workServing topology controls how the node divides work

A serving topology specifies how many replicas run on the node, which GPUs each replica uses, and how each replica is parallelized. We tested four topologies, each using all 8 GPUs with a different replica count and parallel configuration.

The 8-GPU replica uses the layout the NVIDIA Cosmos 3 Super model card recommends for 8-GPU nodes, combining CFG parallelism, Ulysses-SP, and HSDP. The 4-GPU and 2-GPU replicas use tensor parallelism, as NVIDIA’s published Cosmos 3 benchmarks do for multi-GPU runs. Within a multi-GPU replica, each parallelism method divides a different part of the work:

• Tensor parallelism (TP) shards supported diffusion transformer (DiT) weights across GPUs. In these vLLM-Omni layouts, the text encoder remains replicated.

• Classifier-free guidance (CFG) parallelism runs the positive and negative guidance branches on separate GPUs.

• Ulysses sequence parallelism (Ulysses-SP) splits the video-token sequence across GPUs and redistributes attention tensors through all-to-all communication.

• Hybrid Sharded Data Parallel (HSDP) shards model weights across GPUs and gathers them as needed during each forward pass.

Figure 1. Every replica ran in its own vLLM-Omni container with a separate endpoint and a non-overlapping set of GPUs.

vLLM-Omni implements these layouts through its diffusion parallelism methods. A multi-GPU replica can lower per-request latency or pool enough GPU memory to serve a single request. Additional replicas let the node process more independent requests at once, while multi-GPU replicas add communication overhead between GPUs.

Benchmark results: higher-output topologies had higher latencyBenchmark results: higher-output topologies had higher latency

Both platforms showed the same ordering: topologies with more replicas delivered more total video output but had higher request latency.

Moving from one 8-GPU replica to eight single-GPU replicas raised node throughput by 42.5% on Blackwell and 25.7% on Hopper. Mean request latency was 5.5 times as high on Blackwell and 6.3 times as high on Hopper.

Figure 2. Each point is one 24-request run on one node. The table below lists the exact values.