$ sky check nebius
Checking credentials to enable clouds for SkyPilot.
  Nebius: enabled [compute, storage]
To enable a cloud, follow the hints above and rerun: sky check 
If any problems remain, refer to detailed docs at: https://docs.skypilot.co/en/latest/getting-started/installation.html
🎉 Enabled clouds 🎉
  Nebius [compute, storage]
Using SkyPilot API server: http://127.0.0.1:46580

import os
import openai

ENDPOINT = os.getenv("ENDPOINT")
client = openai.Client(base_url=f"http://{ENDPOINT}/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Llama-4-Scout-17B-16E-Instruct",
    messages=[
        {"role": "user", "content": "Tell me about Nebius AI"},
    ],
    temperature=0
)

print(response.choices[0].message.content)
> Nebius AI! After conducting research, I found that Nebius AI is a relatively new player in the artificial intelligence (AI) landscape. 
...

Multi-node deployment for larger models

For larger models like Llama 4 Maverick (which has 128 experts and 400B parameters), we need a multi-node setup as shown below. Save this as llama4-multinode.yaml.

It’s important to note that the Maverick model is 4x the size of the Scout model, so we need to increase both the disk size and the amount of shared memory.

resources:
  accelerators: H100:8
  region: eu-north1
  cloud: nebius
  disk_size: 1024
  ports: 8000

num_nodes: 2  # Specify number of nodes to launch

envs:
  HF_TOKEN: 
  MODEL_NAME: meta-llama/Llama-4-Maverick-17B-128E-Instruct

setup: |
  uv pip install "sglang[all]>=0.4.5" --find-links https://flashinfer.ai/whl/cu124/torch2.5/flashinfer-python
  uv pip install hf_xet
  # Set up shared memory for better performance
  sudo bash -c "echo 'vm.max_map_count=655300' >> /etc/sysctl.conf"
  sudo sysctl -p
run: |
  export GLOO_SOCKET_IFNAME=(ip -o -4 route show to default | awk '{print5}')
  MASTER_ADDR=$(echo "$SKYPILOT_NODE_IPS" | head -n1)
  TOTAL_GPUS=$(($SKYPILOT_NUM_GPUS_PER_NODE * $SKYPILOT_NUM_NODES))

  python -m sglang.launch_server \
  --model-path $MODEL_NAME \
  --tp $TOTAL_GPUS \
  --dp 8 \
  --enable-dp-attention \
  --dist-init-addr ${MASTER_ADDR}:5000 \
  --nnodes ${SKYPILOT_NUM_NODES} \
  --node-rank ${SKYPILOT_NODE_RANK} \
  --trust-remote-code \
  --torch-compile-max-bs 8 \
  --host 0.0.0.0 \
  --port 8000 \
  --chat-template llama-4

Launch with:

export HF_TOKEN=your_huggingface_token_here
sky launch -c llama4-maverick llama4-multinode.yaml --env HF_TOKEN

SkyPilot and SGLang will automatically distribute the model across the two nodes, to set up the proper communication channels between them. Once the model is loaded, you can use the sky status command to get its endpoint for querying:

  ENDPOINT=$(sky status --endpoint 8000 llama4-maverick)
    curl http://$ENDPOINT/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
    "model": "meta-llama/Llama-4-Maverick-17B-128E-Instruct",
    "messages": [
      {
        "role": "user",
        "content": "Tell me about Nebius AI"
      }
    ]
  }' | jq .
  > {
    "id": "98a393272d9544b2a9731bc8990615d8",
    "object": "chat.completion",
    "created": 1745262008,
    "model": "meta-llama/Llama-4-Maverick-17B-128E-Instruct",
    "choices": [
      {
        "index": 0,
        "message": {
        "role": "assistant",
        "content": "Nebius AI! That's a relatively new player in the AI landscape, and I'm happy to provide an overview.\n\n**What is Nebius AI?**\n\nNebius AI is an artificial intelligence company
        ...

The service configuration for SkyServe

Now, let’s use SkyServe to deploy our model as a scalable service with multiple replicas, load balancing and autoscaling. The main difference between sky launch and sky serve is that:

1. sky launch creates a single cluster that stays running until you stop it.
2. sky serve creates a service with multiple replicas that can scale based on load.

The only change you need to make to the YAML file is to add the service section.

For example, the following is the service configuration for the Maverick model. Add this at the end of the llama4-multinode.yaml file:

service:
  readiness_probe:
    path: /health
    initial_delay_seconds: 5400 # 1.5 hours because the model is large
    timeout_seconds: 20
  replica_policy:
    min_replicas: 1
    max_replicas: 2
    target_qps_per_replica: 2.5
    upscale_delay_seconds: 300
    downscale_delay_seconds: 1200

To deploy as a service:

export HF_TOKEN=your_huggingface_token_here
sky serve up -n llama4-maverick-serve llama4-multinode.yaml --env HF_TOKEN

This command:

• Creates a SkyServe controller if one doesn’t exist yet. A controller is a small on-demand CPU VM that manages the deployment of your service, monitors the status of your service and routes traffic to your service replicas.

• Deploys the model by using SGLang.

• Sets up autoscaling based on our replica policy.

You’ll see output like this:

$ export HF_TOKEN=your_huggingface_token_here
$ sky serve up -n llama4-maverick-serve llama4-multinode.yaml --env HF_TOKEN
Start streaming logs for task job of replica 1...
Job ID not provided. Streaming the logs of the latest job.
├── Waiting for task resources on 2 nodes.
└── Job started. Streaming logs... (Ctrl-C to exit log streaming; job will not be killed)
(setup pid=8313) Using Python 3.10.13 environment at: /home/ubuntu/miniconda3
(setup pid=5833, ip=192.168.0.72) Using Python 3.10.13 environment at: /home/ubuntu/miniconda3
(setup pid=8313) Resolved 137 packages in 2.95s
(setup pid=8313) Downloading pillow (4.4MiB)
(setup pid=8313) Downloading pycountry (6.0MiB)
(setup pid=8313) Downloading llguidance (13.3MiB)
...
(head, rank=0, pid=8313) [2025-04-21 18:50:09 DP0 TP0] Load weight begin. avail mem=76.96 GB
(head, rank=0, pid=8313) [2025-04-21 18:50:10 DP0 TP3] Using model weights format ['*.safetensors']
(head, rank=0, pid=8313) [2025-04-21 18:50:10 DP0 TP1] Using model weights format ['*.safetensors']
...
(worker1, rank=1, pid=5833, ip=192.168.0.72) [2025-04-21 18:50:10 DP3 TP13] Using model weights format ['*.safetensors']
(worker1, rank=1, pid=5833, ip=192.168.0.72) [2025-04-21 18:50:10 DP3 TP12] Using model weights format ['*.safetensors']
(worker1, rank=1, pid=5833, ip=192.168.0.72) [2025-04-21 18:50:11 DP2 TP10] Using model weights format ['*.safetensors']
...
Loading safetensors checkpoint shards:  96% Completed | 53/55 [06:21<00:03,  1.52s/it]
Loading safetensors checkpoint shards:  98% Completed | 54/55 [06:23<00:01,  1.45s/it]
(head, rank=0, pid=8490) [2025-04-21 20:14:11 DP1 TP4] Load weight end. type=Llama4ForConditionalGeneration, dtype=torch.bfloat16, avail mem=25.12 GB, mem usage=51.83 GB.

To check service status:

$ sky serve status
Services
NAME                   VERSION  UPTIME      STATUS  REPLICAS  ENDPOINT                     
llama4-maverick-serve  1        1h 58m 25s  READY   1/1       http://89.169.102.185:30001  

Service Replicas
SERVICE_NAME           ID  VERSION  ENDPOINT                    LAUNCHED   RESOURCES               STATUS  REGION     
llama4-maverick-serve  1   1        http://89.169.112.127:8000  2 hrs ago  2x Nebius({'H100': 8})  READY   eu-north1

Once the status shows READY, get your service endpoint:

ENDPOINT=$(sky serve status --endpoint llama4-maverick-serve)
curl $ENDPOINT/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
  "model": "meta-llama/Llama-4-Maverick-17B-128E-Instruct",
  "messages": [
    {
      "role": "user",
      "content": "Tell me about Nebius AI"
    }
  ]
}' | jq .

What’s immediately clear is that Scout actually outperforms Maverick in raw throughput, when comparing a single 8xH100 node to two 8xH100 nodes. This makes sense when you consider the communication overhead between nodes for the distributed Maverick setup (due to tensor parallelism).

Looking at the input and output token throughput separately provides a more nuanced view of performance, as input token processing is typically much faster than output token generation. The Time to First Token (TTFT) and Inter-Token Latency (ITL) metrics are particularly important for real-time applications, as they directly impact user experience. Scout’s 69.17 ms TTFT and 15.80 ms ITL are excellent figures that rival commercial APIs and contribute to smooth conversational experiences.

When we look at the Nebius monitoring dashboard for one of the Maverick nodes, we can see some interesting patterns:

Click to expand

The port data throughput graph shows how the InfiniBand traffic rapidly escalates at the 20:02:00 mark when the model starts loading and distributing weights across nodes. That steep curve represents the significant cross-node communication needed for MoE models.

Click to expand

The NVIDIA NVLink usage shows active communication climbing to ~5.6 GiB. This indicates heavy memory traffic, which is exactly what we’d expect when running a distributed MoE model.

Click to expand

Notice how the power usage jumps from ~130W to ~300W once inference begins. The temperature rises correspondingly, but stays within safe operating ranges.

Click to expand

Perhaps most revealing is the Used Frame Buffer metric, which jumps from 78.4 GB to 79 GB when inference starts. That relatively modest increase highlights one of the key advantages of MoE models — they are memory-efficient during inference because only a subset of experts is activated for any given token. The ~80% utilization shows the system is working hard but not maxed out.

These numbers could definitely be improved by fine-tuning the --tp and --dp parameters of the SGLang server. For example, I could experiment with different tensor parallelism and data parallelism settings to better balance memory usage and computation. The current setup prioritizes stability over maximum performance, which is usually the right choice for production environments.

In practice for most applications, I recommend starting with Scout on a single node. It offers a really good performance-to-cost ratio and avoids the complexity of multi-node setups. However, if your use case requires the extra intelligence that Maverick provides, the distributed setup is definitely viable — just be prepared for about 20% lower throughput and higher infrastructure costs.

Production readiness: From experiment to deployment

So far, we’ve focused on getting Llama 4 models deployed and running. But what does it take to make this production-ready for a real application like a customer support chatbot or code completion engine?

Security for your Llama 4 deployment

When moving from development to production, you’ll want to consider adding some security enhancements such as:

1. API key authentication: Protect your model API endpoints by implementing API key authentication. SGLang supports this via the --api-key parameter, which you can add to your YAML configuration:

envs:
  HF_TOKEN: 
  MODEL_NAME: meta-llama/Llama-4-Scout-17B-16E-Instruct
  AUTH_TOKEN: your_secure_api_key_here  # Store securely, consider using env vars

run: |
  python -m sglang.launch_server \
    --model-path $MODEL_NAME \
    --tp $SKYPILOT_NUM_GPUS_PER_NODE \
    --api-key $AUTH_TOKEN \
    # other parameters as before

2. HTTPS encryption: For production deployments, secure your traffic with HTTPS. SkyServe supports TLS encryption for your endpoints by adding certificate information to your service configuration:

service:
  readiness_probe:
    path: /health
    headers:
      Authorization: Bearer $AUTH_TOKEN
  tls:
    keyfile: path/to/your/private/key.pem
    certfile: path/to/your/certificate.pem
  # Other service configuration as before

3. Authorization headers: Configure your readiness probe and client requests to include authorization headers:

service:
  readiness_probe:
    path: /health
    headers:
      Authorization: Bearer $AUTH_TOKEN
  # Rest of service config

When making requests to your secured endpoint, clients need to include the appropriate authorization header:

curl <https://$ENDPOINT/v1/chat/completions> \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -d '{
    "model": "meta-llama/Llama-4-Scout-17B-16E-Instruct",
    "messages": [
      {"role": "user", "content": "Tell me about Nebius AI"}
    ]
  }'

These security enhancements are essential for production deployments, especially when serving models that might have access to sensitive information or when exposing your API to external users.

Persistent deployment with the SkyPilot API server

For reliable production deployments, you should use SkyPilot’s client-server architecture instead of running commands from your laptop. This transforms SkyPilot from a single-user system into a scalable, multi-user platform with several key benefits:

1. Resilient command and control: The SkyPilot API server acts as the central hub for managing all your Llama 4 deployments, ensuring they continue running even if your local machine disconnects or shuts down.

2. Team collaboration: Multiple team members can connect to the same API server, allowing your entire organization to view and manage resources, without sharing credentials or running all commands from the same machine.

3. Centralized management: Get a single view of all running clusters, jobs and services across your organization, making it easier to monitor cost and usage.

4. Fault-tolerant deployment: The SkyPilot API server can be deployed in a cloud-native way with full fault tolerance, eliminating the risk of workload loss.