Explaining Soperator, Nebius’ open-source Kubernetes operator for Slurm

My name is Mikhail Mokrushin, I’m the Managed Schedulers Team Leader at Nebius. In this article, I’ll provide details about our latest open project: Soperator. I won’t rehash what Slurm and Kubernetes are, their differences, or how to use them for computational jobs — all of that was covered in the previous post. Instead, I’ll focus on methods for combining Slurm and K8s and on the architectural approach we’ve chosen to bring these two systems together. Soperator is available on GitHub under the Apache 2.0 license.

Marrying Slurm and KubernetesMarrying Slurm and Kubernetes

Both Slurm and Kubernetes can be used for distributed model training and high-performance computing (HPC) in general. Like any system, they each have their strengths and weaknesses. In this case, though, the trade-offs are pretty significant.

It’s a shame there’s no easy way to get the best of both worlds: Slurm’s efficient scheduling and granular hardware control, combined with Kubernetes’ universality, autoscaling and self-healing capabilities. Some ML engineers working in big tech don’t even have a choice — Kubernetes is their company’s default infrastructure layer, and there’s no support for a separate model training system. Those determined enough to manage a separate Slurm cluster face a new challenge: they can’t share expensive computational resources between the two workload managers. That’s why many companies wrestle with how to make these two play nice together.

Let’s look at a few approaches that could theoretically unite Slurm and Kubernetes, taking the best from both.

Kubernetes-firstKubernetes-first

The most obvious approach is to run Slurm nodes directly in a Kubernetes cluster (by implementing a Kubernetes operator). While there have been several attempts, none have been successful (at least in the open-source world).

The problem is that these solutions usually force users to change how they interact with Slurm. Kubernetes is built for orchestrating “cloud-native” containerized applications, but Slurm doesn’t fit that mold. In a typical Slurm setup, node filesystems are usually persistent: people install software, create Linux users and groups and store configs, batch scripts, job outputs and all sorts of other stuff there. But in Kubernetes, container filesystems are ephemeral. Many changes require rebuilding the container image and restarting the entire HPC cluster (if Slurm nodes are represented as Kubernetes Pods), which is a nightmare for model training. These solutions often force users to run their jobs in containers, too. Unfortunately, not all HPC job submitters are well-versed in containers and the cloud native philosophy.

Slurm-firstSlurm-first

The second approach involves running Kubernetes nodes as Slurm jobs. It sounds weird, and it is. It’s not really feasible (at least not well) without patching Slurm, because Slurm jobs have a maximum execution time of one year.

My engineering intuition also tells me this approach isn’t the best: it might be fine to run finite workloads in an environment designed for infinite ones, but not the other way around.

Another downside of this approach is that it’s hardly compatible with the many managed solutions for Kubernetes out there — not everyone’s brave enough to maintain an on-premise installation.

PeeringPeering

This is theoretically the cleanest solution, requiring the least number of workarounds.

Instead of running one system inside the other, it integrates them so they can share the computational resources. For example, thanks to Slurm’s modular design, it’s possible to create a scheduling plugin that connects Slurm to Kubernetes and launches jobs there as Pods. This leaves the base Slurm setup untouched.

SchedMD, the folks behind Slurm, are already working on such a solution called Slinky. As I write this, it hasn’t been released yet (so I may be off on some details), but we’re looking forward to it.

While this approach is neat and quite universal, it still requires Slurm users to change how they interact with Slurm, which isn’t ideal for everyone.

Our solutionOur solution

At Nebius, we decided to go with the Kubernetes-first approach. At the same time, we’ve designed the system to preserve the familiar user experience while fitting into cloud-native limitations. This has allowed us to add some features that many users found lacking in vanilla Slurm installations.

Here’s a high-level view of our solution, showing how a Slurm cluster is represented using Kubernetes resources:

Figure 1. Top-level architecture of Soperator

At first glance, our operator works like you’d expect. It takes your desired Slurm cluster state (specified in YAML) and brings the Kubernetes resources in line. During this process, it might bootstrap new Slurm nodes, change container images, update config files, mount new volumes and so on.

The Slurm clusters we create have three types of nodes: Login, Controller and Worker (a.k.a. Compute), all represented as Kubernetes Pods.

Login nodes are load-balanced — each time a user connects to the cluster via SSH, they’re directed to a random node.

While all containers have ephemeral filesystems (changes are lost after node restarts), we use Kubernetes Persistent Volumes to keep data around.

The real magic of Soperator lies in how we use “jail” Persistent Volume. This shared filesystem is mounted on all Worker and Login nodes and contains another root directory, possibly of a different Linux distribution.

When users interact with the Slurm cluster (like browsing the filesystem in an SSH session or running a Slurm job), they actually see this shared filesystem as their root “/” directory. This trick not only gives users a familiar Slurm experience but also actively makes their lives easier: they no longer have to keep filesystems identical across all nodes. They can install new software packages, create Linux users, write job outputs or checkpoint their model on one node, and those changes automatically show up on all the other nodes.

Features and reasoningFeatures and reasoning

Below, you’ll find a detailed description of all the features we bring to Slurm through our integration with Kubernetes.

Shared root filesystemShared root filesystem

As I mentioned, all Login and Worker nodes share the same root filesystem from the user’s perspective. Combined with K8s containers, this takes away the headache of keeping nodes in sync. We call the environment where users do their work the “jail”.

“Host” environments are separate for each Slurm node, but they’re just pod containers — if something goes wrong, you can restart them and get back to square one.

If you’re into recursion, you can still run your jobs in another layer of containerization. The current version of the operator supports Pyxis plugin for this.

How we implemented thisHow we implemented this

For Linux gurus, this solution may sound a bit out there. Linux isn’t built to share its root filesystem. That’s true, which is why the filesystem isn’t entirely shared — some directories are local to each node. These include /dev, containing device drivers; /tmp, containing temporary files stored in RAM; /proc, containing information about running processes; /run, containing runtime variable data; and others. Some low-level libraries, device drivers and system configs are also not shared, though they’re identical — propagated (bind-mounted) from each host’s environment.

Here’s a diagram that tries to explain this concept visually:

Figure 2. How we implemented shared root filesystem

While this approach won’t work for sharing the rootfs of an arbitrary Linux system, it should cover what Slurm users typically need. They usually don’t create new daemons or install device drivers between job submissions. Our Kubernetes operator can explicitly support such operations, but what users need often is always available.

Another consideration is the performance of shared filesystems, which is significantly worse than that of local ones. This can be solved by storing sensitive data on non-shared (or even in-memory) local filesystems mounted to subpaths inside the jail’s root, leaving other data on shared storage. Our operator supports such setups.

To move Slurm job processes from the host to the jail environment, we wrote a SPANK plugin that runs after forking the Slurm worker daemon slurmd, but before launching the job’s command. Our routine creates and enters new Linux namespaces, then changes the root directory the job process sees (using the pivot_root system call). All existing container runtimes, such as Docker or containerd, use a similar, though more complex, isolation method.

This plugin isn’t bound to our setup and could theoretically work in typical (non-Kubernetes) Slurm installations.

We use a different approach to change the root directory for user-established SSH sessions to Login nodes. We simply use OpenSSH’s ChrootDirectory feature.

Surprisingly (even for us!), it actually works.

In fact, the jail environment doesn’t have to be shared. It’s theoretically possible to have a separate persistent volume for each node while keeping all other approaches (Slurm plugin & ChrootDirectory). Users would again need to maintain nodes in an identical state, but the filesystem would be snappier. We’ll add support for this if there’s demand.

GPU health checksGPU health checks

This feature is specific to computations that use modern ML-tailored GPUs. It addresses the fact that these devices aren’t the most reliable and tend to fail under constant load.

This is especially true for large clusters. Check out Meta’s story, where a small issue with one GPU led to a 50% performance drop across their entire 16,000-GPU cluster for several hours. Imagine the cost. That’s why it’s so important to monitor hardware and take suspicious hosts offline quickly.

We tackle this by running periodic GPU health checks. If any Slurm node fails the check, our operator “drains” it, effectively removing the node from the pool for new job scheduling.

In the future, we’re planning to integrate with Nebius to not only drain problematic Slurm nodes but replace them with fresh ones. We’ll also try to leave room for integration with other cloud providers.

How we implemented thisHow we implemented this

There are two kinds of health checks in our solution:

1. Quick checks that take up to 10 seconds and don’t interfere with user workloads when run in parallel. These are implemented as Slurm’s HealthCheckProgram. They make sure the system recognizes GPUs correctly and there are no critical software or hardware issues.

2. Longer checks that take several minutes and run as regular Slurm jobs. Currently, we only have one such check that runs NCCL benchmarks. These checks queue up with user jobs and don’t interfere with them.

Here’s a diagram explaining how longer checks work. It’s not 100% accurate (especially the bash scripts), but it gives you the general idea: