Run in Kubernetes

Apply a Deployment and a restored engine serves behind a Service, on a GPU node of the snapshot's platform. The pod runs the same command a host does: warmctl up NAME.

Before you start

• GPU nodes of the snapshot's platform: GPU model and count, a driver of its branch. The NVIDIA GPU Operator, or the device plugin and Container Toolkit.
• A namespace that allows privileged pods: CRIU and cuda-checkpoint need root and every checkpoint capability. With Pod Security admission, label it pod-security.kubernetes.io/enforce: privileged.
• Nodes that can pull from registry.warmctl.io (anonymous, no account), or your own registry with a copy of the runtime image.

1. The runtime image

Pods run the published warmctl runtime image, registry.warmctl.io/warmctl/runtime:latest: the snapshot's vLLM image with warmctl, its vLLM extension, NVIDIA cuInterpose and tini on top. Its marker file, /opt/warmctl/RUNTIME, tells up to restore in place, in the pod's own container, instead of asking Docker for one.

# from the warmctl source tree
$ docker pull registry.warmctl.io/warmctl/runtime:latest

The image's base must be the snapshot's vLLM image; one runtime image serves every snapshot built on that image, and an in-place restore refuses a snapshot taken in another.

:latest follows the newest warmctl release; for a cluster that must not change under you, pin a version tag such as :v2.0.0-beta.5. To run your own copy, or a runtime image on another vLLM base, build it from the repository with make pod-image POD_IMAGE=<your registry>/warmctl/runtime:<tag> and push it there.

2. Prepare the nodes

A DaemonSet turns off io_uring, pulls the runtime image (some 20 GB, minutes, not seconds) and the snapshot ahead of time, and reads it once so the first restore hits the page cache:

apiVersion: apps/v1
kind: DaemonSet
metadata: {name: warmctl-node-prep}
spec:
  selector: {matchLabels: {app: warmctl-node-prep}}
  template:
    metadata: {labels: {app: warmctl-node-prep}}
    spec:
      nodeSelector: {nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3}
      initContainers:
        - name: io-uring
          image: registry.warmctl.io/warmctl/runtime:latest
          command: [/bin/sh, -c, "echo 2 > /proc/sys/kernel/io_uring_disabled"]
          securityContext: {privileged: true}
        - name: pull
          image: registry.warmctl.io/warmctl/runtime:latest
          args: [pull, glm53-flash:8xh100-80gb-sxm-580.173.02]
          volumeMounts: [{name: snapshots, mountPath: /var/lib/warmctl}]
        - name: page-cache
          image: registry.warmctl.io/warmctl/runtime:latest
          command: [/bin/sh, -c, "find /var/lib/warmctl/pulled -type f -exec cat {} + > /dev/null"]
          volumeMounts: [{name: snapshots, mountPath: /var/lib/warmctl}]
      containers:
        - name: hold
          image: registry.warmctl.io/warmctl/runtime:latest
          command: [/usr/local/bin/tini, --, /bin/sleep, infinity]
      volumes:
        - name: snapshots
          hostPath: {path: /var/lib/warmctl, type: DirectoryOrCreate}

Pick the platform tag that matches the nodes: warmctl catalog --all or the model's card in the catalog lists them.

3. Deploy

apiVersion: apps/v1
kind: Deployment
metadata: {name: glm53-flash}
spec:
  replicas: 1
  selector: {matchLabels: {app: glm53-flash}}
  template:
    metadata: {labels: {app: glm53-flash}}
    spec:
      containers:
        - name: warmctl
          image: registry.warmctl.io/warmctl/runtime:latest
          args: [up, glm53-flash]
          securityContext: {privileged: true}
          resources:
            limits: {nvidia.com/gpu: 8}
          readinessProbe:
            httpGet: {path: /health, port: 8080}
            periodSeconds: 1
          volumeMounts: [{name: snapshots, mountPath: /var/lib/warmctl}]
      volumes:
        - name: snapshots
          hostPath: {path: /var/lib/warmctl, type: DirectoryOrCreate}
---
apiVersion: v1
kind: Service
metadata: {name: glm53-flash}
spec:
  selector: {app: glm53-flash}
  ports: [{port: 80, targetPort: 8080}]

$ kubectl apply -f node-prep.yaml -f warmctl.yaml
$ kubectl logs -f deploy/glm53-flash

The gateway listens on :8080 from the start: /health answers 503 while the pod pulls and restores and 200 once the canary passed, so the Service routes only to ready pods. kubectl logs shows the restore and then the engine's own log.

The hostPath volume is only for speed: snapshots pulled there stay for the next pod on the node. Without it each pod downloads into its own layer. Keep it on local disk; a snapshot read from a network share restores several times slower.

4. Scale on queue depth

With Prometheus and KEDA in the cluster, scale on vLLM's waiting requests. A new replica costs a restore, not a cold start:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: {name: glm53-flash}
spec:
  scaleTargetRef: {name: glm53-flash}
  minReplicaCount: 1
  maxReplicaCount: 4
  pollingInterval: 5
  cooldownPeriod: 600
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-operated.monitoring.svc:9090
        query: sum(vllm:num_requests_waiting{model_name="glm53-flash"})
        threshold: "8"
        activationThreshold: "1"

Limits

• One node per pod. A snapshot is one host's engine: tensor parallel within a node, not across nodes.
• No hostNetwork. The restored engine's loopback would be the node's; warmctl refuses it.
• No loopback sidecars. Service meshes that bind loopback ports can collide with the restored sockets.
• Driver upgrades. Moving a node pool to another driver branch invalidates its snapshots: pull the platform tag of the new branch.

Every detail of the in-place restore, its preflight and the open risks: docs/kubernetes.md in the source tree.