Tutorial

Serve and scale vLLM on RunPod

Bake a warm Qwen3-8B snapshot on one RunPod pod, keep it on a network volume, and start every new pod from it in about 20 seconds instead of a minute and a half. About forty minutes, most of it the model download and the bake.

Experimental. A RunPod pod is a container without the privileges CRIU needs, so warmctl runs there with its dmtcp backend, which works in user space. Why and how: Solving vLLM cold starts on RunPod. It ships in warmctl v2.0.0-beta.16; use that release’s runtime image below.

You need:

A RunPod account with Secure Cloud access and credit for two L40S pods at once.
For the last step only, a RunPod API key in RUNPOD_API_KEY, if you want to start pods from a script.

Nothing else: the warmctl runtime image is public at registry.warmctl.io, and Qwen3-8B is public on Hugging Face.

Create a network volume

In the RunPod console, open Storage → New Network Volume. Pick a data center that has L40S GPUs (we measured in EU-NL-1) and give it 100 GB: the model takes 16 GB, a snapshot 14 to 24 GB.

Every pod that restores from this volume runs in that data center. That is the point: a network volume next to the GPUs reads at 0.8–1.0 GiB/s per pod, about twice a Global Volume, and the restore takes as long as reading the snapshot.

Bake the snapshot on one pod

Deploy a pod with these settings. The pod downloads the model, warms vLLM once, snapshots it, and copies the snapshot to the volume, all from its start command.

GPU: 1 × L40S, Secure Cloud. In the filters, CUDA version 13.2.
Container image: registry.warmctl.io/warmctl/runtime:v2.0.0-beta.16
Container disk: 80 GB. The bake builds the snapshot here before copying it.
Network volume: the one you created, with the mount path set to /nv. Not /workspace: warmctl keeps the engine’s own /workspace as part of the snapshot.
Container start command:

Container start command

{"entrypoint": ["/usr/local/bin/tini", "--", "/bin/sh", "-c"], "cmd": ["hf download Qwen/Qwen3-8B --revision b968826d9c46dd6066d109eabc6255188de91218 --local-dir /nv/models/qwen3-8b && warmctl build --in-place --profile qwen3-8b-l40s-tp1 --model-dir /nv/models/qwen3-8b --output /root/snap/qwen3-8b --compress && mkdir -p /nv/snap/qwen3-8b && tar -C /root/snap/qwen3-8b --exclude=./manifest.json -cf - . | tar -C /nv/snap/qwen3-8b -xf - && cp /root/snap/qwen3-8b/manifest.json /nv/snap/qwen3-8b/ && warmctl verify /nv/snap/qwen3-8b; sleep infinity"]}

Follow it in the pod’s Logs. The bake takes about ten minutes, and these lines mark its progress:

log

process snapshot backend: dmtcp (in place)
donor qwen3-8b started; loading the model
published /root/snap/qwen3-8b: 37 files, 24.16 GiB (weights 15.28 GiB), 13.60 GiB stored
ok qwen3-8b: profile qwen3-8b-l40s-tp1, 1 ranks on l40s-48gb, 24.16 GiB (37 files)

The last line is warmctl verify reading the copy on the volume. Then terminate the pod: the model and the snapshot stay on the volume.

Two details of that command matter:

--compress stores the snapshot as zstd frames, 24 GiB in 14. A new pod reads fewer bytes from the volume and decodes them on its CPUs as they arrive.
manifest.json is copied last. warmctl treats it as the snapshot’s commit point, so a pod that starts during the copy sees no snapshot rather than half of one.

Start a replica

Deploy a second pod in the same data center, with the same GPU, CUDA filter, image, container disk and volume mounted at /nv, and:

Expose HTTP ports: 8080
Container start command: up /nv/snap/qwen3-8b --mode in-place. The image’s entrypoint is already warmctl.

In its logs, the snapshot is copied to the pod’s disk while the engine restores from it, and the last line says when it serves:

log

snapshot staged in /var/lib/warmctl/staged/qwen3-8b while it restores
rank 0: /dev/nvidia3 links to this pod's /dev/nvidia0
weights reloaded and fingerprints match: 15.28 GiB
serving started 21.4s

Ask it something through RunPod’s proxy, with your pod’s ID:

bash

$ curl -s https://POD_ID-8080.proxy.runpod.net/v1/chat/completions -H 'Content-Type: application/json' -d '{"model": "qwen3-8b", "messages": [{"role": "user", "content": "What is the capital of France?"}], "max_tokens": 64}'

warmctl serves vLLM’s OpenAI-compatible API on port 8080. The pod’s /health answers 503 until the engine is restored and 200 after; use it as the readiness check of whatever sends traffic.

Scale out

Every pod deployed with the replica’s settings restores the same snapshot in about 20 seconds. From a script, the same pod through RunPod’s API:

bash

$ curl -s https://v2-rest.runpod.io/v2/pods -H "Authorization: Bearer $RUNPOD_API_KEY" -H 'Content-Type: application/json' -d '{
  "name": "qwen3-8b-replica",
  "image": "registry.warmctl.io/warmctl/runtime:v2.0.0-beta.16",
  "args": "up /nv/snap/qwen3-8b --mode in-place",
  "gpu": {"id": "NVIDIA L40S", "count": 1, "allowedCudaVersions": ["13.2"]},
  "cloud": "SECURE", "dataCenterIds": ["EU-NL-1"], "disk": 80, "ports": ["8080/http"],
  "mounts": {"network": [{"volumeId": "YOUR_VOLUME_ID", "path": "/nv"}]}
}'

A pod that restarts its container keeps the copy on its disk and restores from it in about 11 seconds. A stopped pod loses its container disk and starts from the volume again.

A 180B model on four GPUs

The same steps serve Qwen3.8-Flash-Next, a 180B mixture-of-experts model in FP8, split over four RTX PRO 6000 GPUs. What changes:

GPU: 4 × RTX PRO 6000, Secure Cloud, CUDA 13.2. We used EU-RO-1; four of them in one data center are not always free, so create the volume where you find them.
Network volume: 500 GB: the model takes 180 GB, the snapshot 245 GB (228 GiB).
Container disk: 400 GB, for the bake and, on every replica, the pod’s own copy.
Profile: qwen38-flash-next-rtx6000-tep4, the layout from vLLM’s recipe for this model: tensor and expert parallel over the four GPUs, FP8 KV cache, multi-token prediction, the full 262K context.

The bake pod’s start command:

Container start command

{"entrypoint": ["/usr/local/bin/tini", "--", "/bin/sh", "-c"], "cmd": ["hf download Qwen/Qwen3.8-Flash-Next-FP8 --revision 236dfdf285828023ca3bcd3f37366c58a3469b13 --local-dir /nv/models/qwen38-flash-next --max-workers 32 && warmctl build --in-place --profile qwen38-flash-next-rtx6000-tep4 --model-dir /nv/models/qwen38-flash-next --output /root/snap/flash-next && cd /root/snap/flash-next && find . -type d -exec mkdir -p /nv/snap/flash-next/{} \\; && find . -type f ! -name manifest.json | xargs -P 32 -I{} cp {} /nv/snap/flash-next/{} && cp manifest.json /nv/snap/flash-next/ && warmctl verify /nv/snap/flash-next; sleep infinity"]}

The bake took 16 minutes, 13 of them vLLM loading and compiling the model once. The copy runs 32 at a time: the volume took a single stream at about 0.1 GiB/s and 32 at 8.8 GiB/s.

Replicas start with up /nv/snap/flash-next --mode in-place. What we measured on new pods in EU-RO-1:

vLLM, cold: 751 s, 586 of them loading the model from the volume.
vLLM with compile caches on the volume: 668 s.
warmctl from the volume: 215 s, nearly all of it reading 228 GiB at about 1 GiB/s; from the pod’s own copy: 21 s.

The engine runs NVIDIA’s cuInterpose, because its all-reduce shares GPU memory between the four processes; the runtime image ships it, and nothing has to be set for it.

If a pod does not start

warmctl checks before it restores and says what is wrong. The messages you may meet on RunPod:

“driver …, the snapshot was taken on …; cuda-checkpoint needs the same driver”: the pod landed on a host with another NVIDIA driver. Deploy it again with the CUDA filter the bake pod had.
“a DMTCP snapshot reopens the model’s files where the donor had them”: the volume is mounted at another path than on the bake pod. Mount it at /nv on every pod.
“/workspace holds files no warmctl run left there”: something is mounted at /workspace, usually the volume itself. Mount the volume at /nv.
“another warmctl run is in place in this container”: a pod runs one engine. For a second model or a second replica, deploy another pod.

What you did

Baked a warm Qwen3-8B snapshot on a RunPod pod, without privileges, and kept it on a network volume.
Started a replica from it in about 20 seconds, where vLLM takes 83 with compile caches and 124 without.
Scaled out, each new pod a restore instead of a cold start.

Next: the measurements behind these numbers, and why CRIU cannot run in a pod, are in Solving vLLM cold starts on RunPod. The backend’s design and limits are in docs/dmtcp.md.