Boosting dedicated inference on Nebius with warmctl

01 · THE PROBLEMA replica you pay for before it serves

How long does it take to bring a model up for dedicated inference? For a small model, a minute. For a frontier MoE like GLM-5.3-Flash, it is ten to twenty.

Running inference yourself is not hard because of the API server. It is hard because of the cold start. A new vLLM replica of GLM-5.3-Flash on 8×H100 has to load 320 GiB of FP8 weights, compile its kernels, capture CUDA graphs, autotune hundreds of GEMM shapes and warm up. That is 650 seconds before the first token, and it does not count pulling the weights to a fresh node.

Every one of those minutes is a full 8×H100 node you pay for and cannot serve from. Autoscaling reacts ten minutes late, so you over-provision for peaks you could have followed. A rollout or a lost node costs the same ten minutes again, with users waiting.

If you want dedicated inference on Nebius that comes up in under a minute, that is what warmctl is for. Here is how it works, and the numbers from a real cluster.

02 · WHAT IS WARMCTLWarm up once. Restore everywhere.

warmctl is an open-source project, Apache-2.0, on GitHub. It starts the engine once, warms it over every shape it will serve, and then takes a snapshot of the whole warm process: its memory, the GPU memory and CUDA state of every rank, the NCCL communicators and the memory ranks share, and everything the warm-up produced, from compiled kernels to tuned GEMMs. The weights travel next to it, already processed, one file set per rank.

The snapshot ships as an ordinary OCI artifact, like a container image: a manifest and content-addressed blobs that any registry can store, so you push it once and pull it on any host with the same GPUs and driver branch. There, warmctl up maps the processes back with CRIU, re-attaches the GPUs with cuda-checkpoint, reloads the weights to the same addresses and checks a canary token before it opens the port. Nothing is recompiled, recaptured or re-tuned.

03 · LET'S BENCHMARKThree 8×H100 nodes on Nebius

To test it the way you would run it, I created a Nebius Managed Kubernetes cluster with one node group of three gpu-h100-sxm nodes, 8×H100 each, and a shared filesystem mounted on every node. It is a few clicks in the console, and with preemptible instances a test like this is cheap to run.

You do not have to build the snapshot yourself: warmctl publishes ready-made ones for popular open models in its public catalog. The benchmark has three stages. The first two are setup that you do once, the third is what every new replica pays.

Stage 1 · Put the snapshot on the shared filesystem

Once per cluster, a Kubernetes Job pulls the snapshot from registry.warmctl.io and copies it to the share. The pull reads the OCI manifest, then fetches the blobs over 64 parallel connections and checks every digest; large files arrive in 64 MiB pieces and are assembled on disk. It lands on the node's local disk first, because many small writes are slow on any network filesystem, and a wide copy then moves it to the share in 32 streams.

$ warmctl pull glm53-flash:8xh100-80gb-sxm-580.173.02
  pulled 399.76 GiB: 7,126 files in 6,326 blobs, 723 s
$ cp -r … /mnt/filesystem-q1/warmctl/glm53-flash   # 32 streams, 97 s

Fourteen minutes, once. From here on, every node in the cluster reads the same copy, and no node needs local disk for snapshots.

Stage 2 · Install warmctl on the nodes

Once per node, a DaemonSet pulls the warmctl runtime image, which carries warmctl, cuInterpose and the exact vLLM build the snapshot was made with, and sets the one kernel switch CRIU needs, kernel.io_uring_disabled=2. This is node bootstrap, not a replica's critical path, so the timed runs below never wait for an image.

Stage 3 · Cold starts from the share

Every new replica is a pod: the stock one from the warmctl Kubernetes guide, pointed at the directory on the share instead of a registry name.

containers:
  - name: warmctl
    image: registry.warmctl.io/warmctl/runtime:latest
    args: [up, /share/warmctl/glm53-flash]
    securityContext: {privileged: true}
    resources: {limits: {nvidia.com/gpu: 8}}
    readinessProbe: {httpGet: {path: /health, port: 8080}, periodSeconds: 1}
    volumeMounts: [{name: share, mountPath: /share, readOnly: true}]

Cold means cold: before every run, the node's page cache was dropped, so every byte of the 404 GiB came off the shared filesystem, and the node had no copy of the snapshot of its own. I ran it nine times one node at a time, then twice on all three nodes at once.

Starting all three replicas at the same moment did not slow any of them down. Each node still read the share at 7.2–7.8 GiB/s, so the filesystem served about 22 GiB/s in all:

04 · WHY IT IS FASTRead ahead, all the way down

A restore needs two things from storage: the CRIU images, 82 GiB of process memory, and the weights, 320 GiB in 8 rank sets. warmctl starts 32 readers before it starts CRIU. They read the CRIU images first, so CRIU finds them in page cache, then the weights rank by rank, while the GPUs are being restored. By the time a rank reloads its weights, most of them are already in memory.

That only works if the storage keeps up, and the Nebius shared filesystem does. I ran fio on all three nodes at once, 32 threads each, one minute per test:

Each node gets about 7 GiB/s both ways, and three nodes together get 21 GiB/s with no sign of a ceiling. warmctl reads at 7.0–7.8 GiB/s during a restore: it takes nearly everything a node can pull from the share, which is why one replica or three come up in the same minute.

05 · AND THEN IT SERVESStock speed, from the first request

A fast start is worth little if the engine is slower after it. So with the three replicas up, I ran vllm bench serve against each of them: random prompts, fixed output lengths, from one request at a time to hundreds.

A single request decodes at 7.9 ms per token, about 197 tokens/s with MTP. With 128 concurrent long generations the node produces 4,170 output tokens/s, and with 32k-token prompts it processes 21,733 tokens/s. These match what the same profile does as a freshly started vLLM on 8×H100: the restored engine is the warm engine, CUDA graphs and tuned kernels included.

06 · TAKEAWAYSWhat this changes

• A new replica is a minute, not eleven. GLM-5.3-Flash serves in 58.4 s from a shared filesystem, against 650 s for a vLLM cold start.
• One copy serves the cluster. 400 GiB lands on the share once; any node restores from it, three at a time without slowing down.
• Nodes do not need to know the model. The only per-node step is the runtime image, which your node bootstrap can pre-pull.
• Scale with traffic, not ahead of it. With a minute to a new replica, you can follow load instead of paying for idle 8×H100 nodes.