Solving vLLM cold starts on RunPod: why CRIU doesn’t work there

00 · THE PROBLEMWhere the time goes during a cold start

Every time you scale out on RunPod, a new pod boots vLLM in a fresh container with the model on a network volume. Before it can serve its first token, it goes through three steps:

• Load the weights: read the checkpoint from the volume into GPU memory — 16 GB for Qwen3-8B, 180 GB for Qwen3.8-Flash-Next.
• Compile the model: torch.compile fuses operations into GPU kernels, costing tens of seconds on every cold boot.
• Capture CUDA graphs & warm up: vLLM records GPU execution graphs across batch sizes and runs dummy profiling passes.

Keeping compiled kernels on the volume helps, cutting Qwen3-8B’s start from 124 s to 83 s. But every new replica still pays for weight loading, graph capture and warmup — and on Qwen3.8-Flash-Next (180B on four GPUs) a warm-cache start still takes 668 s (751 s cold), 586 s of which is just reading weights off the volume.

What it means. Cold start is a tax paid on every new replica: while a pod boots, traffic waits or you pay for idle GPUs kept warm. Compile caches shave the tax; they don’t remove it.

01 · WARMCTLBoot once, restore everywhere

warmctl is an open-source tool (Apache-2.0) that warms an inference engine once and ships the result as a snapshot. A snapshot here is the whole running engine at the moment it became ready: the memory of its processes, their open files and their GPU memory. warmctl packs it as an OCI image, the format container registries already store, so you push it to any registry and every new replica restores it instead of booting.

On a virtual machine it stands on three pieces:

• CRIU (Checkpoint/Restore In Userspace) is the standard Linux tool that freezes a tree of processes, saves their memory, files and sockets to disk, and later recreates them exactly.
• cuda-checkpoint is NVIDIA’s utility that moves a process’s GPU state into ordinary host memory, so it can be saved with the rest, and back after a restore.
• cuInterpose is NVIDIA’s library that lets the GPU processes of one engine, spread over several GPUs, be checkpointed together.

On VMs this restores Qwen3-8B in about 9 seconds (KServe) and a 27B model in 11 (RTX PRO 6000).

Bottom line. On a machine you control, the hard part, the operating system’s view of the engine, is CRIU’s job, and CRIU does it well.

02 · THE WALLRunPod is different: a GPU container is not a virtual machine

Most GPU clouds rent you a virtual machine. You get your own Linux kernel and root on it, so CRIU can do whatever it needs. Container-first clouds such as RunPod rent you a container on a shared host instead. Your processes run on the provider’s kernel, inside limits the provider sets, and three of those limits stop CRIU:

• No capabilities. Linux splits root’s powers into about forty capabilities, and Docker keeps a safe subset by default. CRIU needs CAP_SYS_ADMIN or CAP_CHECKPOINT_RESTORE, and a pod gets neither.
• A system-call filter. seccomp decides which kernel calls a process may make. Docker’s default profile refuses unshare and clone3, which CRIU uses to rebuild processes and their namespaces.
• Read-only /proc/sys. CRIU cannot set the next process ID, so it cannot give restored processes back their original PIDs.

We checked a RunPod pod: exactly Docker’s defaults, and there is no way to ask for more.

What it means. Kernel-level checkpointing needs kernel-level privileges, and a container-first cloud will not hand them out. On RunPod, warmctl’s main backend cannot even start, so the fix has to live in user space.

03 · MULTIPLE BACKENDSMultiple backend support in warmctl: CRIU and DMTCP

Once kernel-level checkpointing is off the table in a container, how else can you bring a multi-GPU vLLM replica up in seconds? Before adding a second backend, we evaluated the alternatives (see our full alternatives comparison):

Rather than compromise on cold-start time, we built multiple process-backend support into warmctl: CRIU and DMTCP behind a single internal process-tree interface (tree):

• CRIU backend (default on VMs & Kubernetes): kernel-level checkpoint/restore with zero runtime overhead when you control the host.
• DMTCP backend (default in unprivileged containers): user-space checkpoint/restore via DMTCP, where processes save and restore themselves through preloaded libc wrappers.

A bake records which backend it used in the snapshot’s manifest (process.backend) and OCI tag (-dmtcp), and warmctl up automatically picks the snapshot backend the current host can restore.

Bottom line. Supporting both CRIU and DMTCP under one roof means warmctl uses the fastest kernel path on VMs and seamlessly falls back to user-space checkpointing in locked-down GPU containers — while GPU checkpointing, weight streaming, and OCI distribution stay identical.

04 · UNDER THE HOODWhat’s the difference: CRIU and DMTCP, side by side

Both tools freeze the same three vLLM processes, but from opposite sides of the operating system boundary:

• CRIU works from outside, through the kernel. A separate criu process stops the engine, reads every mapping, file descriptor and socket straight from kernel tables, and asks the kernel to recreate the tree with its original PIDs — leaving zero wrappers in the restored process, at the cost of kernel privileges.
• DMTCP works from inside, through LD_PRELOAD. dmtcp_launch preloads libdmtcp.so into every process so it tracks its own libc calls, stops its own threads on checkpoint, and rebuilds sockets and virtual PIDs on restart — using only ordinary user permissions.
• The GPU and weights path is identical. Before either dump, cuda-checkpoint moves GPU state into host RAM and restores it afterwards, while weights stream straight onto the GPUs and are checked against chunk fingerprints and a canary prompt.

If DMTCP needs no privileges, why not drop CRIU and use DMTCP everywhere? Because CRIU reads the ground truth from the kernel regardless of how a resource was created, whereas DMTCP only sees what passes through its libc wrappers. In a modern inference stack, plenty bypasses libc: the CUDA driver’s internal eventfds and pipes, Rust libraries (cuInterpose) issuing raw syscalls, io_uring, UCX’s mmap hooks, Python’s fcntl64, and ZMQ sockets that must drain in lockstep. Each of those silently dropped state, hung a dump, or crashed a restart until we patched DMTCP itself (11 commits in TensorRails/dmtcp) and wrote a companion plugin (fdkeep) to rebuild what bypassed libc. CRIU works out of the box on arbitrary process trees with zero runtime overhead; DMTCP is the surgical path for locked-down containers where the kernel door is shut.

What it means. The snapshot format, OCI registry, and warmctl up stay the same. CRIU remains the universal default on VMs, and DMTCP takes over inside containers.

05 · ON RUNPODHow to run it on RunPod

1. Create a network volume in the data center where your pods will run, and start one pod from the warmctl runtime image with the volume mounted at /nv. warmctl uses RunPod’s default /workspace itself.
2. Bake the snapshot on that pod, then copy it to the volume, manifest.json last, so a pod starting meanwhile never sees half a snapshot:

   warmctl build --in-place --profile qwen3-8b-l40s-tp1 \
     --model-dir /nv/models/qwen3-8b --output /root/snap/qwen3-8b --compress

3. Start every new replica from it: any pod with the same image, GPU model and driver, with this as its start command.

   warmctl up /nv/snap/qwen3-8b --mode in-place    # OpenAI-compatible API on :8080

06 · RESULTSWhat a new replica costs now — and how it serves

Running those three steps on a new RunPod pod cuts Qwen3-8B’s cold start to a fifth of warm-cache vLLM:

Smaller models gain even more: Qwen3-0.6B starts in 8 seconds from a network volume and 4 from the pod’s disk, against 34 for vLLM with warm caches; split over two GPUs (TP=2), it serves in 5 to 6 seconds.

Does a restored engine lose throughput or accuracy?

Because DMTCP keeps its LD_PRELOAD wrappers in the process after restart, we benchmarked a restored Qwen3-8B engine head-to-head against stock vLLM and a CRIU-restored engine on the same RTX PRO 6000 GPU (vllm bench serve, 1,024 input / 512 output tokens, greedy):

Throughput and inter-token latency match within noise (0.3%), and every restore verifies chunk fingerprints on the GPU copy stream plus a canary generation check before /health returns 200.

What it means. A restored replica serves at full stock speed — and if you look at the startup table above (11 s from local disk vs 21 s from a Network Volume and 49 s from a Global Volume), the remaining cold-start time is no longer vLLM or DMTCP. It is the storage.

07 · STORAGE BOTTLENECKAnother RunPod problem: slow remote storage

Once DMTCP and cuda-checkpoint restore the processes and GPU state in a few seconds, we hit a second wall on RunPod: remote storage bandwidth. Throughput may vary by GPU tier and region, but across the A40, L40S and RTX PRO 6000 pods we tested, a single reader got only 0.11–0.28 GiB/s, and even with 32 parallel readers a regional RunPod Network Volume (MooseFS over FUSE) topped out at ~0.8–1.0 GiB/s (and a Global Volume at 0.5 GiB/s).

Unlike clouds such as Nebius, Verda, or CoreWeave, where shared network filesystems over RDMA or virtiofs deliver 7 to 15+ GiB/s per node, a RunPod pod spends almost its entire startup time pulling gigabytes off the volume. Squeezing ~1 GiB/s out of RunPod’s FUSE mount forced us to add two things: 32-stream parallel staging (restoring while copying, so dmtcp_restart and cuda-checkpoint run while weights stream in) and a dedicated --compress mode to shrink how many bytes cross the wire:

You can either bake a compressed snapshot directly, compress an existing snapshot, and then restore it on any replica pod:

$ warmctl build --in-place --profile qwen3-8b-l40s-tp1 \
  --model-dir /nv/models/qwen3-8b --output /nv/snap/qwen3-8b --compress
  published /nv/snap/qwen3-8b: 37 files, 23.11 GiB, 13.35 GiB stored

$ warmctl compress /root/snap/qwen3-8b /nv/snap/qwen3-8b
  compressed 37 files into /nv/snap/qwen3-8b (13.35 GiB stored)

$ warmctl up /nv/snap/qwen3-8b --mode in-place
  staged in /var/lib/warmctl/staged/qwen3-8b · serving started 21.4s

Linux process images compress well — 2.84× with independent 32 MiB zstd frames (7.28 → 2.56 GB on Qwen3-8B). Model weights are near-random and barely compress as raw bytes, but splitting BF16 weights into two byte planes (the high sign+exponent byte compresses to 0.37×, while the mantissa stays raw) yields 1.46× on the weights too.

What matters: --compress will hurt you on fast storage. Decompression is heavily CPU-bound — 8 worker threads on a pod decode weights at about 3.1–3.4 GiB/s (and single-stream zstd inside mtcp_restart ran at 0.67 GB/s, slowing a local restore from 10.3 s to 15.4 s). If you have fast local NVMe or a high-speed shared network filesystem like on Nebius (where we measured 7.0–7.8 GiB/s per node and 22 GiB/s across three nodes), Verda, or CoreWeave, storage is far faster than the CPU can decompress, so --compress will only slow your startup down:

What it means. Slow remote storage is a separate bottleneck on RunPod, but --compress offsets it at build time. Just don’t turn it on where your storage is faster than your CPUs.

08 · MULTI-GPU SCALETesting large multi-GPU models

Single-GPU 8B models are the easy case; production deployments split 30B to 180B+ models across multiple GPUs. Beyond dense TP=2 and TP=4 models (Qwen3-32B, Gemma 4 31B), the hardest case we tested on RunPod was Qwen3.8-Flash-Next: a 180B mixture-of-experts model in FP8 on four RTX PRO 6000 cards, running our full catalog recipe — tensor and expert parallel across all four GPUs, FP8 KV cache, multi-token prediction, and the native 262K context.

Stock vLLM needs 751 seconds to start it on a new pod, 586 of them reading 180 GB of weights from the network volume. A replica that takes over twelve minutes to appear cannot absorb a spike: by the time it serves, the spike is over.

What broke, and what we fixed

• The fast all-reduce shares GPU memory. Over PCIe, the profile uses FlashInfer’s all-reduce, which maps the other ranks’ buffers directly, and cuda-checkpoint refuses such a process. We kept the collective and ran the engine under NVIDIA’s cuInterpose, which takes the sharing apart before the checkpoint and puts it back at the same addresses after it.
• cuInterpose and DMTCP had never met. Under DMTCP, cuInterpose could not find its own core library, its control socket did not survive a restart, and the processes vLLM spawns lost it. Each is fixed: two patches in our DMTCP fork, one in warmctl’s DMTCP plugin, and the launch order.
• This model makes vLLM spawn its workers. Its model class touches CUDA when Python imports it, so vLLM starts the engine and workers as fresh programs, and DMTCP’s checkpoint then waited forever on a pipe. The cause was one missing wrapper, fcntl64, the name under which modern glibc compiles fcntl. With it, DMTCP sees how Python duplicates descriptors, and the checkpoint completes.
• The next pod numbered its GPUs differently. The bake pod’s rank 2 was /dev/nvidia7; on the replica that name belonged to rank 3’s GPU, and warmctl refused to restore. It turned out to be harmless: the CUDA restore moves each GPU by its UUID and opens its devices itself, and the weights’ fingerprints and the canary prove it.

• The bake runs once. It took 16 minutes on one pod, and the snapshot is 228 GiB: 183 GiB of weights and 45 GiB of process memory. Every replica afterwards restores it.
• The volume is the clock. Restarting the processes, the GPUs and their shared memory took about 7 seconds of the 215; the rest was reading 228 GiB at 1.06 GiB/s, the rate one pod got from RunPod’s network volume with 32 parallel readers. From the pod’s own disk the same restore takes 21 seconds.
• Compression does not pay for FP8. FP8 weights barely compress (the snapshot went from 228 to 183 GiB, mostly in the process memory), and reading frames was slower, so the compressed snapshot took 283 seconds. For BF16 weights, which compress well, the trade turns the other way.

What it means. A replica of a 180B model on RunPod now serves in under four minutes from the network volume instead of eleven to twelve, and in 21 seconds when its pod restarts with a copy on disk, where vLLM would take the full eleven minutes again. Faster storage next to the GPUs would shrink the first number; nothing in warmctl stands in the way.

09 · STATUSWhere it stands

• Verified across architectures and tensor-parallel sizes. Tested end to end on RunPod Pods (A40, L40S, RTX PRO 6000) and in unprivileged containers on 4×/8×RTX PRO 6000 across Qwen3-0.6B (TP1/TP2/TP4), Qwen3-8B (TP1, raw and --compress), Qwen3.8-27B-FP8 (TP1), Qwen3-32B (TP2/TP4), Gemma 4 31B (TP4), gpt-oss-20b (TP1), and Qwen3.8-Flash-Next 180B (TEP4).
• Multi-GPU collectives that share memory run under cuInterpose inside DMTCP, verified with FlashInfer’s PCIe all-reduce on RTX PRO 6000.
• Same image, GPU model and driver on both sides. A snapshot holds GPU state, so the receiving pod has to match.
• Not tested yet: RunPod Serverless workers.

10 · TRY ITSet it up on RunPod

The step-by-step guide, with every pod setting and the exact start commands, is the tutorial Serve and scale vLLM on RunPod; the design and every limit are in docs/dmtcp.md. warmctl is on GitHub, and the dmtcp backend ships in v2.0.0-beta.16. If you run inference on RunPod and want help with your models, write to us.

11 · FAQQuestions we get

Why is vLLM’s cold start slow on RunPod?

Each new pod loads the weights from a network volume, compiles the model with torch.compile, captures CUDA graphs and warms up: 83–124 s for Qwen3-8B, 11–12.5 minutes for a 180B model on four GPUs. Compile caches save part of it; the weights are read again on every pod.

Can I run CRIU on RunPod?

No. Pods get Docker’s default capabilities and seccomp profile and a read-only /proc/sys; CRIU needs CAP_SYS_ADMIN or CAP_CHECKPOINT_RESTORE, unshare and clone3.

What does warmctl use instead?

DMTCP, which checkpoints processes from inside, in user space, together with NVIDIA’s cuda-checkpoint for GPU memory. warmctl picks it automatically when CRIU cannot run.

Which storage should hold the snapshot?

A network volume in the same data center as your pods. It reads at 0.8–1.0 GiB/s per pod, about twice a Global Volume, and that sets the start time.

Does it work on RunPod Serverless?

Not tested yet; this post covers RunPod Pods.

Sources

• CRIU; DMTCP and the TensorRails/dmtcp fork; NVIDIA cuda-checkpoint.
• RunPod docs: network volumes.
• warmctl PR #61: the dmtcp backend, the measurements and their limits.