---
title: "Optimize KServe cold starts: 134 s → 9 s with warmctl"
description: "A kernel cache cuts a vLLM start on an L40S from 134 s to 33 s; warmctl restores the same Qwen3-8B in 9 s, and a new KServe replica serves in 14 s. Measured, and how to set it up."
author: Kirill Bespalov
date: 2026-10-24
url: https://warmctl.io/blog/kserve-llm-replicas-in-seconds/
---

# Optimize KServe cold starts: 134 s → 9 s with warmctl

> Kernel caches are the usual answer to slow vLLM starts on KServe. We measured what they save and what they cannot, then made every new LLMInferenceService replica a restore of a warm engine. One preset, three lines in your service.

**Key results**

- Qwen3-8B on one NVIDIA L40S with vLLM 0.30, start to /health 200: cold start 134 s; weights in the page cache 70 s; kernel cache (torch.compile, Triton and inductor caches) 103 s from disk and 33 s with the weights in memory; warmctl restore 9.1 s from memory and 63 s from a 0.44 GiB/s disk. Qwen3-32B on 2×L40S: 251 s cold, 47 s with a kernel cache and the weights in memory, 13.8 s restored.
- A kernel cache removes torch.compile; it cannot remove starting vLLM’s processes and CUDA (23 s with every file in memory), loading the weights, capturing CUDA graphs or profiling memory.
- Under KServe v0.21.0, pod created to Ready on a warm node: stock vLLM 70 s, with a kernel cache 40 s, warmctl 21 s; scaling a warmctl LLMInferenceService from 1 to 2 replicas took 14.1 s with a 1-second startup probe.

vLLM cold start134 s kernel cache, warm node33 s warmctl restore9.1 sQwen3-8B on one L40S, the same vLLM each time, start to `/health` 200.

- 14.1 s a new KServe replica, restored by warmctl
- 9.1 s warmctl restore, start to /health 200
- 33 s best stock start: kernel cache, weights in memory
- 134 s vLLM cold start, same GPU

##
Isn’t a kernel cache enough?

  Every new vLLM replica starts its processes, loads the weights, compiles the model and captures CUDA graphs before its first token. On Kubernetes each scale-out pays that again. The usual fix is a **kernel cache**: keep what `torch.compile` and Triton built, ship it in the image or mount it as a folder. KServe ships one as [KernelCache](https://github.com/kserve/kserve/tree/master/kernelcache), and you hear that vLLM then starts in seconds.

We measured it on one VM with two L40S, the same vLLM image and flags every time, and then ran the winner as an ordinary KServe LLMInferenceService.

##
What a kernel cache saves, and what it can’t

  **The kernel cache removes compilation**: `torch.compile` drops from 20 s to under a second. With the weights in memory too, vLLM starts in 33 s. That is the best stock start here, and most of it is what no cache reaches: 23 s to start vLLM’s processes and CUDA, then weights, CUDA graphs and memory profiling.

**warmctl skips the start itself.** Its snapshot is the engine after all of that, processes and GPU memory included, so it serves in **9.1 s**: 3.7× faster than the best kernel-cache start, and at the same speed afterwards (46 tokens/s for one stream, 900 for 32).

| Start, to /health 200 | Qwen3-8B · 1×L40S | Qwen3-32B · 2×L40S |
|---|---|---|
| vLLM, cold | 134 s | 251 s |
| vLLM, weights in memory | 70 s | 87 s |
| kernel cache, cold | 103 s | 210 s |
| kernel cache, weights in memory | 33 s | 47 s |
| warmctl, cold | 63 s | 183 s |
| warmctl, snapshot in memory | 9.1 s | 13.8 s |

**A cold disk is the catch.** On a node that has never read the model, every start waits for the bytes, and a snapshot is more bytes than the weights: on this 0.44 GiB/s disk warmctl is only 1.7× faster for 8B and 1.1× for 32B. Keep snapshots on the nodes (step 6) or on a fast shared filesystem: GLM-5.3-Flash reads from one at 7 GiB/s and restores on 8×H100 in 58 s, against a 650 s cold start ([benchmark](https://warmctl.io/blog/boosting-dedicated-inference/)).

##
KServe + warmctl =

  KServe’s main object for LLMs is the **LLMInferenceService**, and it is easy to customize: a preset can replace the command its pods run. Swap `vllm serve` for `warmctl up`, and every new replica restores a warm engine instead of cold-starting one. Routing, scheduling and autoscaling stay exactly as KServe does them.

##
How warmctl works

  The idea is simple. Take any model you have and the inference engine you run it on (vLLM, today), and start it once with warmctl. When the engine is warm, with its weights loaded, kernels compiled and CUDA graphs captured, warmctl takes a snapshot of it, processes and GPU memory included, and ships it as an ordinary OCI image. From then on you don’t start replicas, you hand out copies of the warm one.

##
Run it

  Let’s do it on a GPU cluster with two L40S. You need privileged pods allowed for the model and warmctl v2.0.0-beta.11 or newer: earlier runtime images refuse port 8000 ([#29](https://github.com/TensorRails/warmctl/issues/29)). The same steps, kept current, are the [Serve on KServe](https://warmctl.io/docs/tutorials/kserve/) tutorial.

### 1 · Install KServe

$ curl -sL https://github.com/kserve/kserve/releases/download/v0.21.0/llmisvc-full-install-with-manifests.sh | bash

### 2 · Add the warmctl preset

  Once per cluster. Pods get warmctl’s runtime image and run `warmctl up` with the snapshot from the model URI. warmctl answers on port 8000 of the pod’s IP, where KServe expects vLLM; root is for CRIU, and pulled snapshots stay on the node.

  warmctl-preset.yaml
apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceServiceConfig
metadata:
  name: warmctl
  namespace: kserve
spec:
  storageInitializer: {enabled: false}
  router: {gateway: {}, route: {}, scheduler: {}}
  template:
    containers:
      - name: main
        image: registry.warmctl.io/warmctl/runtime:latest
        command: [/usr/local/bin/tini, --, /bin/sh, -c,
          "exec warmctl up --listen ${POD_IP}:8000 ${WARMCTL_SNAPSHOT#warmctl://}"]
        env:
          - name: POD_IP
            valueFrom: {fieldRef: {fieldPath: status.podIP}}
          - name: WARMCTL_SNAPSHOT
            value: "{{ .Spec.Model.URI }}"
        securityContext:
          privileged: true
          allowPrivilegeEscalation: true
          runAsNonRoot: false
          runAsUser: 0
          seccompProfile: {type: Unconfined}
        startupProbe: {periodSeconds: 1, failureThreshold: 3600}
        volumeMounts: [{name: snapshots, mountPath: /var/lib/warmctl}]
    volumes:
      - name: snapshots
        hostPath: {path: /var/lib/warmctl}
  $ kubectl apply -f warmctl-preset.yaml

### 3 · Deploy a model

  An ordinary LLMInferenceService with two differences: `baseRefs` names the preset and the model URI names a snapshot from the [catalog](https://warmctl.io/catalog/).

  qwen3-8b.yaml
apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: qwen3-8b
spec:
  baseRefs:
    - name: warmctl
  model:
    uri: warmctl://qwen3-8b
    name: qwen3-8b
  template:
    containers:
      - name: main
        resources:
          limits:
            nvidia.com/gpu: "1"
  $ kubectl apply -f qwen3-8b.yaml

### 4 · Send a request

  $ URL=$(kubectl get llmisvc qwen3-8b -o jsonpath='{.status.url}')
$ curl -s $URL/v1/chat/completions -H 'Content-Type: application/json' \
    -d '{"model": "qwen3-8b", "messages": [{"role": "user", "content": "Hi"}]}'

### 5 · Scale

  $ kubectl patch llmisvc qwen3-8b --type merge -p '{"spec":{"replicas":2}}'
  The second replica was Ready in **14.1 s**. `spec.scaling` with KEDA or HPA works as for stock vLLM.

### 6 · Keep snapshots on the nodes

Optional. The first pod on a node pulls its snapshot; this pulls it on every node ahead of time and reads it once into memory.

  prefetch.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: warmctl-prefetch
spec:
  selector: {matchLabels: {app: warmctl-prefetch}}
  template:
    metadata: {labels: {app: warmctl-prefetch}}
    spec:
      initContainers:
        - name: pull
          image: registry.warmctl.io/warmctl/runtime:latest
          command: [/bin/sh, -c, "warmctl pull qwen3-8b:1xl40s-48gb-580.173.02 &&
            find -L /var/lib/warmctl/pulled -type f -exec cat {} + > /dev/null"]
          volumeMounts: [{name: snapshots, mountPath: /var/lib/warmctl}]
      containers:
        - name: hold
          image: registry.warmctl.io/warmctl/runtime:latest
          command: [/usr/local/bin/tini, --, sleep, infinity]
      volumes:
        - name: snapshots
          hostPath: {path: /var/lib/warmctl}

## The same starts as LLMInferenceServices

| Pod created → Ready | warm node | cold node |
|---|---|---|
| stock vLLM | 70 s | 131 s |
| stock vLLM + kernel cache | 40 s | 100 s |
| warmctl | 21 s | 70 s |

  KServe v0.21.0 on k3s on the same VM, one L40S per replica, Qwen3-8B. Stock services run the same vLLM image with the weights on a PersistentVolume; the kernel cache is mounted from a hostPath, as KernelCache does. Scaling a running warmctl service from one to two replicas: **14.1 s** to Ready.

Two things to know. KServe’s startup probe runs every 10 s and readiness waits for it, so a pod that restores in 12 s became Ready at 21 s; the preset probes every second, which took the scale-out from 21 s to 14.1 s. And a brand-new service also waits for KServe’s own scheduler to start: 14.8 s more from `kubectl apply`, paid once.

##
Your models, your registry

  The catalog is a shortcut, not a requirement. warmctl warms any model you serve on vLLM: your fine-tunes, your settings, your vLLM image. Build the snapshot once per model and GPU type in your own pipeline, push it to the registry you already run, and point `model.uri` at it. Nothing leaves your network, so it works the same on-premises.

$ sudo warmctl build --profile my-model.yaml --model-dir /models/my-model \
    --output /snaps/my-model
$ sudo warmctl push /snaps/my-model registry.example.com/llm/my-model
  Then `uri: warmctl://registry.example.com/llm/my-model` in the service. A [step-by-step build](https://warmctl.io/blog/qwen38-27b-11-second-start-rtx-pro-6000/) shows the profile and what goes into the image.

##
What does not work yet

  What we are improving next:

    - **Unprivileged pods.** CRIU needs a privileged pod today; a node agent that restores into a regular pod is next.

    - **HTTPS to the pod.** With KServe’s `enableLLMInferenceServiceTLS` on, probes and traffic use HTTPS. The gateway serves it with KServe’s certificate from v2.0.0-beta.12 ([Serve over HTTPS](https://warmctl.io/docs/guides/https/#kserve)); upstream KServe’s own gateway still connects to the pods in plain HTTP.

    - **Cold nodes.** The restore waits until the whole snapshot is read. Restoring processes while the weights still stream in should shorten the cold path.

    - **Name lookup.** `warmctl up qwen3-8b` asks the registry for 3–4 s even when the snapshot is already on the node.

    - **One node per replica.** Tensor and expert parallelism within a node; no prefill/decode split or multi-node workers yet.

warmctl is open source on [GitHub](https://github.com/TensorRails/warmctl). If you run KServe and try it, tell us how it went.