Optimize KServe cold starts: 134 s → 9 s with warmctl

00 · THE QUESTIONIsn’t a kernel cache enough?

Every new vLLM replica starts its processes, loads the weights, compiles the model and captures CUDA graphs before its first token. On Kubernetes each scale-out pays that again. The usual fix is a kernel cache: keep what torch.compile and Triton built, ship it in the image or mount it as a folder. KServe ships one as KernelCache, and you hear that vLLM then starts in seconds.

We measured it on one VM with two L40S, the same vLLM image and flags every time, and then ran the winner as an ordinary KServe LLMInferenceService.

01 · WHERE IT GOESWhat a kernel cache saves, and what it can’t

The kernel cache removes compilation: torch.compile drops from 20 s to under a second. With the weights in memory too, vLLM starts in 33 s. That is the best stock start here, and most of it is what no cache reaches: 23 s to start vLLM’s processes and CUDA, then weights, CUDA graphs and memory profiling.

warmctl skips the start itself. Its snapshot is the engine after all of that, processes and GPU memory included, so it serves in 9.1 s: 3.7× faster than the best kernel-cache start, and at the same speed afterwards (46 tokens/s for one stream, 900 for 32).

A cold disk is the catch. On a node that has never read the model, every start waits for the bytes, and a snapshot is more bytes than the weights: on this 0.44 GiB/s disk warmctl is only 1.7× faster for 8B and 1.1× for 32B. Keep snapshots on the nodes (step 6) or on a fast shared filesystem: GLM-5.3-Flash reads from one at 7 GiB/s and restores on 8×H100 in 58 s, against a 650 s cold start (benchmark).

02 · KSERVEKServe + warmctl =

KServe’s main object for LLMs is the LLMInferenceService, and it is easy to customize: a preset can replace the command its pods run. Swap vllm serve for warmctl up, and every new replica restores a warm engine instead of cold-starting one. Routing, scheduling and autoscaling stay exactly as KServe does them.

03 · HOW IT WORKSHow warmctl works

The idea is simple. Take any model you have and the inference engine you run it on (vLLM, today), and start it once with warmctl. When the engine is warm, with its weights loaded, kernels compiled and CUDA graphs captured, warmctl takes a snapshot of it, processes and GPU memory included, and ships it as an ordinary OCI image. From then on you don’t start replicas, you hand out copies of the warm one.

04 · TUTORIALRun it

Let’s do it on a GPU cluster with two L40S. You need privileged pods allowed for the model and warmctl v2.0.0-beta.11 or newer: earlier runtime images refuse port 8000 (#29). The same steps, kept current, are the Serve on KServe tutorial.

1 · Install KServe

$ curl -sL https://github.com/kserve/kserve/releases/download/v0.21.0/llmisvc-full-install-with-manifests.sh | bash

2 · Add the warmctl preset

Once per cluster. Pods get warmctl’s runtime image and run warmctl up with the snapshot from the model URI. warmctl answers on port 8000 of the pod’s IP, where KServe expects vLLM; root is for CRIU, and pulled snapshots stay on the node.

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceServiceConfig
metadata:
  name: warmctl
  namespace: kserve
spec:
  storageInitializer: {enabled: false}
  router: {gateway: {}, route: {}, scheduler: {}}
  template:
    containers:
      - name: main
        image: registry.warmctl.io/warmctl/runtime:latest
        command: [/usr/local/bin/tini, --, /bin/sh, -c,
          "exec warmctl up --listen ${POD_IP}:8000 ${WARMCTL_SNAPSHOT#warmctl://}"]
        env:
          - name: POD_IP
            valueFrom: {fieldRef: {fieldPath: status.podIP}}
          - name: WARMCTL_SNAPSHOT
            value: "{{ .Spec.Model.URI }}"
        securityContext:
          privileged: true
          allowPrivilegeEscalation: true
          runAsNonRoot: false
          runAsUser: 0
          seccompProfile: {type: Unconfined}
        startupProbe: {periodSeconds: 1, failureThreshold: 3600}
        volumeMounts: [{name: snapshots, mountPath: /var/lib/warmctl}]
    volumes:
      - name: snapshots
        hostPath: {path: /var/lib/warmctl}

$ kubectl apply -f warmctl-preset.yaml

3 · Deploy a model

An ordinary LLMInferenceService with two differences: baseRefs names the preset and the model URI names a snapshot from the catalog.

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: qwen3-8b
spec:
  baseRefs:
    - name: warmctl
  model:
    uri: warmctl://qwen3-8b
    name: qwen3-8b
  template:
    containers:
      - name: main
        resources:
          limits:
            nvidia.com/gpu: "1"

$ kubectl apply -f qwen3-8b.yaml

4 · Send a request

$ URL=$(kubectl get llmisvc qwen3-8b -o jsonpath='{.status.url}')
$ curl -s $URL/v1/chat/completions -H 'Content-Type: application/json' \
    -d '{"model": "qwen3-8b", "messages": [{"role": "user", "content": "Hi"}]}'

5 · Scale

$ kubectl patch llmisvc qwen3-8b --type merge -p '{"spec":{"replicas":2}}'

The second replica was Ready in 14.1 s. spec.scaling with KEDA or HPA works as for stock vLLM.

6 · Keep snapshots on the nodes

Optional. The first pod on a node pulls its snapshot; this pulls it on every node ahead of time and reads it once into memory.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: warmctl-prefetch
spec:
  selector: {matchLabels: {app: warmctl-prefetch}}
  template:
    metadata: {labels: {app: warmctl-prefetch}}
    spec:
      initContainers:
        - name: pull
          image: registry.warmctl.io/warmctl/runtime:latest
          command: [/bin/sh, -c, "warmctl pull qwen3-8b:1xl40s-48gb-580.173.02 &&
            find -L /var/lib/warmctl/pulled -type f -exec cat {} + > /dev/null"]
          volumeMounts: [{name: snapshots, mountPath: /var/lib/warmctl}]
      containers:
        - name: hold
          image: registry.warmctl.io/warmctl/runtime:latest
          command: [/usr/local/bin/tini, --, sleep, infinity]
      volumes:
        - name: snapshots
          hostPath: {path: /var/lib/warmctl}

05 · THROUGH KSERVEThe same starts as LLMInferenceServices

KServe v0.21.0 on k3s on the same VM, one L40S per replica, Qwen3-8B. Stock services run the same vLLM image with the weights on a PersistentVolume; the kernel cache is mounted from a hostPath, as KernelCache does. Scaling a running warmctl service from one to two replicas: 14.1 s to Ready.

Two things to know. KServe’s startup probe runs every 10 s and readiness waits for it, so a pod that restores in 12 s became Ready at 21 s; the preset probes every second, which took the scale-out from 21 s to 14.1 s. And a brand-new service also waits for KServe’s own scheduler to start: 14.8 s more from kubectl apply, paid once.

06 · YOUR MODELSYour models, your registry

The catalog is a shortcut, not a requirement. warmctl warms any model you serve on vLLM: your fine-tunes, your settings, your vLLM image. Build the snapshot once per model and GPU type in your own pipeline, push it to the registry you already run, and point model.uri at it. Nothing leaves your network, so it works the same on-premises.

$ sudo warmctl build --profile my-model.yaml --model-dir /models/my-model \
    --output /snaps/my-model
$ sudo warmctl push /snaps/my-model registry.example.com/llm/my-model

Then uri: warmctl://registry.example.com/llm/my-model in the service. A step-by-step build shows the profile and what goes into the image.

07 · NEXTWhat does not work yet

What we are improving next:

• Unprivileged pods. CRIU needs a privileged pod today; a node agent that restores into a regular pod is next.
• HTTPS to the pod. With KServe’s enableLLMInferenceServiceTLS on, probes and traffic use HTTPS. The gateway serves it with KServe’s certificate from v2.0.0-beta.12 (Serve over HTTPS); upstream KServe’s own gateway still connects to the pods in plain HTTP.
• Cold nodes. The restore waits until the whole snapshot is read. Restoring processes while the weights still stream in should shorten the cold path.
• Name lookup. warmctl up qwen3-8b asks the registry for 3–4 s even when the snapshot is already on the node.
• One node per replica. Tensor and expert parallelism within a node; no prefill/decode split or multi-node workers yet.

warmctl is open source on GitHub. If you run KServe and try it, tell us how it went.