Tutorial

Serve on KServe

Add one preset to KServe, and every LLMInferenceService replica restores a warm vLLM engine instead of starting one. About half an hour, most of it installing KServe and the first download.

Level 3 of 3. One GPU host → Kubernetes → KServe. On one L40S under KServe v0.21.0, a Qwen3-8B pod is Ready in 13 s against 70 s for stock vLLM; the measurements.

You need:

A Kubernetes cluster with a GPU node that fits Qwen3-8B: one free H100, H200, B200 or L40S, on a driver branch the catalog lists for it. The NVIDIA device plugin or GPU Operator.
A namespace that allows privileged pods: CRIU and cuda-checkpoint need root and every checkpoint capability.
Nodes that can pull from registry.warmctl.io (anonymous, no account).
kubectl, plus helm and yq for KServe's installer.

Install KServe

bash

$ curl -sL https://github.com/kserve/kserve/releases/download/v0.21.0/llmisvc-full-install-with-manifests.sh | bash
$ kubectl -n kserve get llminferenceserviceconfigs

The installer brings the LLMInferenceService controller, Gateway API with Envoy Gateway, and KServe's own presets, which the second command lists. Skip this step on a cluster that already runs KServe v0.21 or later with LLMInferenceService.

Add the warmctl preset

warmctl-preset.yaml

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceServiceConfig
metadata: {name: warmctl, namespace: kserve}
spec:
  storageInitializer: {enabled: false}
  router: {gateway: {}, route: {}, scheduler: {}}
  template:
    containers:
      - name: main
        image: registry.warmctl.io/warmctl/runtime:latest
        command: [/usr/local/bin/tini, --, /bin/sh, -c,
          "exec warmctl up --listen ${POD_IP}:8000{{ if .GlobalConfig.EnableTLS }} --tls-cert /var/run/kserve/tls/tls.crt --tls-key /var/run/kserve/tls/tls.key{{ end }} ${WARMCTL_SNAPSHOT#warmctl://}"]
        env:
          - name: POD_IP
            valueFrom: {fieldRef: {fieldPath: status.podIP}}
          - name: WARMCTL_SNAPSHOT
            value: "{{ .Spec.Model.URI }}"
        securityContext:
          privileged: true
          allowPrivilegeEscalation: true
          runAsNonRoot: false
          runAsUser: 0
          seccompProfile: {type: Unconfined}
        startupProbe: {periodSeconds: 1, failureThreshold: 3600}
        volumeMounts: [{name: snapshots, mountPath: /var/lib/warmctl}]
    volumes:
      - name: snapshots
        hostPath: {path: /var/lib/warmctl, type: DirectoryOrCreate}

bash

$ kubectl apply -f warmctl-preset.yaml

KServe merges a service's presets over its default vLLM template. This one swaps the container for the warmctl runtime image and its command for warmctl up, makes the pod privileged, probes every second and keeps pulled snapshots on the node. The Deployment, workload Service, InferencePool, scheduler and HTTPRoute stay KServe's.

The gateway takes port 8000 on the pod's IP, where KServe sends probes and traffic; the restored engine keeps 127.0.0.1:8000 behind it. The {{ if }} passes KServe's certificate when HTTPS is on (below).

Deploy a model

qwen3-8b.yaml

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata: {name: qwen3-8b, namespace: llm}
spec:
  baseRefs: [{name: warmctl}]
  model: {uri: "warmctl://qwen3-8b", name: qwen3-8b}
  template:
    containers:
      - name: main
        resources: {limits: {nvidia.com/gpu: "1"}}

bash

$ kubectl create namespace llm
$ kubectl label namespace llm pod-security.kubernetes.io/enforce=privileged
$ kubectl apply -f qwen3-8b.yaml
$ kubectl -n llm get llmisvc qwen3-8b -w

An ordinary LLMInferenceService with two differences: baseRefs names the preset, and model.uri names a snapshot from the catalog, as warmctl up takes it. model.name must be the snapshot's served name, the same word.

The first pod on a node downloads the snapshot, some 23 GiB, then restores; READY turns True when the canary passed. kubectl -n llm logs -f deploy/qwen3-8b-kserve -c main shows the restore and then the engine's own log.

Send a request

bash

$ GW=$(kubectl get svc -A -l gateway.envoyproxy.io/owning-gateway-name=kserve-ingress-gateway \
    -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')
$ curl -s http://$GW/llm/qwen3-8b/v1/chat/completions -H 'content-type: application/json' -d '{
    "model": "qwen3-8b",
    "messages": [{"role": "user", "content": "Name three warm colors."}],
    "max_tokens": 64
  }'

KServe's HTTPRoute serves the model at /NAMESPACE/NAME/v1/… on its gateway, and its scheduler picks the replica. Without a LoadBalancer, kubectl -n envoy-gateway-system port-forward the gateway's Service instead.

Scale out

bash

$ kubectl -n llm patch llmisvc qwen3-8b --type merge -p '{"spec":{"replicas":2}}'
$ kubectl -n llm get pods -w

LLMInferenceService has no scale subresource, so kubectl scale does not work; patch replicas. The new pod finds the snapshot on the node and only restores: on an L40S it served 14 s after the patch. A replica on another node downloads first.

Keep snapshots on the nodes

The preset's hostPath keeps every pulled snapshot in /var/lib/warmctl for the next pod on that node. To have a node ready before the first pod lands, pull ahead with the DaemonSet from Serve in Kubernetes, tag qwen3-8b for that node's platform.

HTTPS

With KServe's enableLLMInferenceServiceTLS, probes, the scheduler's metrics scrape and the workload Service talk HTTPS to the pod, and the preset hands KServe's certificate to warmctl. How to turn it on: Serve over HTTPS.

What you did

Added warmctl to KServe as one LLMInferenceServiceConfig.
Served Qwen3-8B from the catalog as an LLMInferenceService, through KServe's gateway and scheduler.
Scaled it, each new replica a restore instead of a cold start.

Next: serve your own models: build and publish a snapshot into your registry and point model.uri at it, warmctl://registry.example.com/llm/my-model.