SGLang Omni: Serving Omni & Multimodal Models | Nebius Science Paper Club

Multimodal generation combines components that process different data and have different scheduling needs.

In the Qwen3-Omni example discussed during the session, an audio encoder processes input, the Thinker generates tokens, the Talker produces audio codec frames, and a vocoder converts them into speech.

Streaming between stages allows audio generation to begin before the full response is ready.

Components that communicate on every decoding step can be grouped into one stage to reduce overhead.

For example, the Talker and its supporting completion head exchange data frequently, making separation across processes expensive.

Components with looser dependencies can run independently, with their own schedulers and memory budgets.

SGLang Omni makes the mapping between stages, processes and GPUs configurable.

Developers can change how components are grouped, where they run and how many replicas are deployed without rewriting the model implementation.

The right configuration depends on communication frequency, resource bottlenecks and memory requirements.

The system must balance throughput with low latency and continuous audio delivery.

Efficient batching alone is not enough: the pipeline also needs to start responding promptly and keep producing audio without interruptions.

The speakers emphasized dedicated streaming strategies that coordinate model execution with network delivery.

The team evaluates both output accuracy and serving performance, then reruns benchmarks in continuous integration.

Tests cover different combinations of input and output modalities, including audio and video input with text and audio output.

Performance thresholds are recalibrated as the system improves to help detect regressions in later changes.

Its multi-stage architecture is relevant to workloads that combine several interacting model components.

The team described world models and physical AI as development directions beyond the audio examples presented.

Support depends on the specific model architecture and its scheduling requirements; the discussion did not establish universal support for these workloads.