vLLM cold start: where the minutes go and how to cut them

The short answer. A vLLM cold start takes from about two minutes for an 8B model on one GPU to ten minutes or more for a large MoE model on eight, because before the first token the server boots its processes, loads the weights, compiles the model, captures CUDA graphs, tunes kernels and warms up. Staged weights, persisted compile caches, a faster weight loader and fewer CUDA graphs cut a small model’s start by 3–4×; what they leave is the work of building a running engine, which every new replica repeats. A snapshot of the warm engine removes that part too.

00 · THE ANATOMYWhy vLLM startup is slow: what happens before the first token

A cold start is the time from launching vllm serve on a machine that has never run this model to the first answered request. It is what every new replica pays when you scale out. Before the server answers, it goes through these steps:

• Boot. Start Python, import vLLM and PyTorch, spawn the API server, the engine and one worker process per GPU, and initialize CUDA. On several GPUs it also starts NCCL, the library through which GPUs exchange data.
• Load the weights. Read the checkpoint, usually safetensors files, from disk or the network into GPU memory, and process it: quantize, split across GPUs.
• Compile. torch.compile traces the model and generates fused GPU kernels for it. vLLM stores the result in a compile cache.
• Profile the KV cache. A dummy forward pass measures how much GPU memory is left, and vLLM reserves it for the KV cache, the memory that holds attention keys and values of running requests.
• Capture CUDA graphs. A CUDA graph is a recorded sequence of GPU kernel launches that replays with one call. vLLM records one per batch size it expects, so serving does not pay the CPU cost of launching kernels one by one.
• Tune and JIT-compile kernels. Libraries such as FlashInfer, Triton and DeepGEMM benchmark or compile kernel variants for this GPU and these shapes. On large mixture-of-experts (MoE) models this takes minutes.
• Warm up. A few dummy requests, so that real ones do not hit lazy compilation.

In a container there are two more steps before all of this: pulling the image and, if nobody staged it, downloading the model. Here is how the steps add up for Qwen3-8B on one L40S, measured on our own VM:

On an 8B model the two biggest pieces are boot and the weights, and both shrink on the second start on the same machine. A large MoE model looks different. GLM-5.3-Flash on eight H100s takes 650 seconds, and the weights are only 7% of it:

Others see the same split. Tensorfuse timed Llama-3.1-8B on an L40S at 294 s: 61 s download, 33 s weights, 52 s compilation, 54 s CUDA graph capture and 94 s imports and engine init (Tensorfuse). At the other end of the scale, H Company saw Qwen3.5 397B FP8 take up to 27 minutes to start on Kubernetes before they rebuilt the whole path (H Company).

The most careful look at the steps is the MLSys 2026 paper Breaking the Ice, which profiled 22 models on H100 and L40S (arXiv 2606.07362). Three of its findings change how you should think about the problem:

• Startup is mostly a CPU job. Of the six steps the authors timed, only KV cache profiling and graph capture run on the GPU. Bootstrapping, the tokenizer, loading and compilation are bound by the CPU and by Python, so a node with a fast GPU and a slow CPU starts slowly.
• Uncached compilation is the biggest single swing. With the compile cache disabled, storing the compiled graphs took 11–21 s across their models; with it, 3–6 s. Even a cache hit is not free: loading cached graphs for a 9B model took 5.66 s.
• Graph capture scales with the number of sizes, not the model. Llama-2-7B captured 3 batch sizes in 0.33 s and 60 sizes in 1.8 s. The capture bill is set by how many batch sizes you ask for.

The same pattern shows at the small end. j9s’s 3B model took 117 s cold, Tensorfuse’s 8B spent 94 of its 294 s in imports and engine init, and in our runs Qwen3-8B’s boot was 55 s of 134 s, before a single weight was read. The fixed costs do not shrink with the model, so the smaller the model, the larger their share.

What it means. On a small model, half the cold start is boot and weights, which disks and caches fix. On a large MoE model, 93% is the engine compiling, capturing and tuning, the work every replica repeats. And most of it waits on the CPU, not the GPU.

01 · MEASURE FIRSTHow to measure vLLM startup time

Time the whole start the way a load balancer sees it: from launch until /health answers. This is the number every figure in this guide reports, and it is the one your autoscaler waits for.

$ t0=$(date +%s.%N)
$ vllm serve Qwen/Qwen3-8B --port 8000 > vllm.log 2>&1 &
$ until curl -sf localhost:8000/health >/dev/null; do sleep 0.2; done
$ echo "ready in $(echo "$(date +%s.%N) - $t0" | bc) s"

Then pull the phases out of the log. vLLM prints a line for each of them, and the wording changes a little between releases, so check your version’s output:

$ grep -E "Loading weights took|Model loading took|torch.compile takes|Graph capturing finished|init engine" vllm.log

What the log does not cover is boot, the time before the first line: subtract the logged phases from the total. For a finer split, the authors of Breaking the Ice published a startup profiler.

Measure on a fresh machine, or make yours look fresh. The second start on the same VM reads the weights from the page cache, the copy of recently read files that Linux keeps in RAM, and reuses the compile caches of the first. In Figure 1 the page cache alone took Qwen3-8B from 134 s to 70 s, and a new node in an autoscaling group never has it. Before a cold run, drop both:

$ rm -rf ~/.cache/vllm ~/.triton ~/.cache/flashinfer ~/.nv   # the caches of earlier starts
$ sync && echo 3 | sudo tee /proc/sys/vm/drop_caches        # the page cache

Bottom line. Time cold starts on a node with an empty page cache and empty caches. A benchmark that restarts on the same machine shows a start that autoscaling never gets.

02 · STOCK FIXESHow to speed up vLLM startup with stock flags

Everything in this section works with stock vLLM and is what most write-ups recommend, from the easiest to the most involved. The numbers come from our runs and from the posts linked in each item.

1Stage the model and skip the Hub check (HF_HUB_OFFLINE=1)

Stage the weights on a local disk or a volume once, and set HF_HUB_OFFLINE=1 so vLLM does not contact the Hugging Face Hub at import time. Tensorfuse’s download step went from 61 s to zero this way; RunPod lists the same variable. Keep the image small, or pre-pull it on the nodes, for the same reason.

2Ship compiled Python bytecode (UV_COMPILE_BYTECODE=1)

Set ENV UV_COMPILE_BYTECODE=1 before the uv install steps of your Dockerfile, so Python does not compile vLLM’s sources on every new container. The vLLM team measured about 18 s saved per start on an H100 server, 17.6 s for Qwen3-8B (vLLM blog).

3Persist the torch.compile cache and the kernel caches (VLLM_CACHE_ROOT)

vLLM writes compiled graphs under VLLM_CACHE_ROOT, ~/.cache/vllm by default. Triton, FlashInfer and the CUDA driver keep their own JIT caches in ~/.triton, ~/.cache/flashinfer and ~/.nv. Put them on a volume or bake them into the image:

$ docker run --gpus all \
    -v /cache/vllm:/root/.cache/vllm -v /cache/triton:/root/.triton \
    -v /cache/flashinfer:/root/.cache/flashinfer -v /cache/nv:/root/.nv \
    -v /models:/models -e HF_HUB_OFFLINE=1 vllm/vllm-openai --model /models/qwen3-8b

In our runs this cut torch.compile from 20 s to about a second, and the whole start from 134 s to 103 s. Others report the same order: Tensorfuse’s compilation went from 42 to 13 s, route179’s from 39 to 10 s, and j9s took a 3B model from 117 to 27 s with caches alone (j9s).

The cache is keyed on the vLLM version, the configuration, the model code and the GPU, so copy the whole directory and expect a recompile after any upgrade (vLLM docs). Two practical notes from teams that run this at scale: a cache on a network volume costs storage and bandwidth every month (Tensorfuse puts it at $0.30 per GB-month on EFS, against a free but slower download), and the caches can be shared by every replica of the same build, which is what H Company does with its FP8 compile and JIT kernel caches.

4Load the weights faster: prefetch, Run:ai Model Streamer, InstantTensor

A plain read is bound by the disk: 16 GB at the 0.44 GiB/s of our VM disk is half a minute, and our weights step took 48 s cold against 4.5 s from the page cache. vLLM has three faster paths:

• --safetensors-load-strategy prefetch reads files ahead in parallel. RunPod’s Qwen3-32B-FP8 on two H200s loaded in about 21 s with it.
• --load-format runai_streamer streams tensors with many threads, from local disk or object storage; tune --model-loader-extra-config '{"concurrency":8}' to your storage. route179’s weights step went from 142 to 59 s.
• --load-format instanttensor adds distributed loading and direct I/O. The vLLM docs show Qwen3-30B-A3B on an H200 going from 57.4 to 1.77 s, and DeepSeek-R1 on eight H200s from 160 to 15.3 s (vLLM docs).

Whichever loader you pick, measure its throughput rather than trusting the flag. H Company moved a 397B model’s weights from S3 straight into eight GPUs with NVIDIA ModelExpress and then between GPUs over RDMA, and found the RDMA path quietly running over plain TCP at 1.8 Gbit/s because of a library mismatch. Once fixed, the GPU-to-GPU transfer fell from 41.4 s to 0.22 s. A loader that falls back silently looks like a slow disk.

5Reduce the CUDA graph capture size (--max-cudagraph-capture-size)

By default vLLM captures graphs for a range of batch sizes capped at 512, or 1024 on data-center Blackwell GPUs. If a replica never runs more than 64 requests at once, say so:

$ vllm serve Qwen/Qwen3-8B --max-num-seqs 64 --max-cudagraph-capture-size 64

Tensorfuse’s capture went from 54 to 7 s. Keep the cap at or above --max-num-seqs: batches outside the captured sizes run without graphs and lose throughput. Since v0.10.0 vLLM also freezes Python’s garbage collector during capture, which took Qwen3-0.6B’s capture from 35 s to 2 s (PR #21146), so on small models this step matters less than older posts suggest: our Qwen3-8B captured in 2.0 s.

6Optimization levels and --enforce-eager: a faster start, slower serving

-O2 is the default. -O1 compiles less and captures only piecewise graphs; -O0 skips compilation, CUDA graphs and FlashInfer autotuning, and --enforce-eager skips compilation and graphs (vLLM docs). Every token then costs more. In our runs eager mode was not even the fastest start: 47 s, against 33 s with a warm kernel cache that keeps full speed.

7Restart faster on the same node with vLLM Preload

vllm preload starts a daemon per GPU that keeps the processed, sharded weights and the FlashInfer autotune results in GPU memory. An engine started with --load-format ipc_cache maps them instead of loading:

$ vllm preload --model /models/m --tensor-parallel-size 4
$ vllm serve /models/m --tensor-parallel-size 4 --load-format ipc_cache

The docs promise restarts in seconds instead of minutes. The daemon and the engine must share the node, the user and the vLLM version, the daemon holds the GPU memory while it lives, and pipeline parallelism is rejected (vLLM docs). It helps a restart, not a new replica on a new node.

8Share GPUs between models with sleep mode

A server started with --enable-sleep-mode can offload its weights to CPU memory or drop them, free the GPU, and wake up in seconds without restarting the process (vLLM docs). It is for sharing GPUs between models, not for adding replicas.

9vLLM initialized engine snapshots: one GPU, same machine, experimental

vLLM can save an initialized engine with CRIU and restore it, experimentally: one GPU, tensor parallelism 1, the same machine, root. The vLLM roadmap reports Qwen3-32B on an H100 going from 54.70 s to 20.38 s; the docs say it “is not a portable model artifact” (vLLM docs, #48193).

Put together, the stock fixes give these results in public write-ups and in our runs:

What it means. Stock fixes take a small model from minutes to about half a minute, a 3–4× gain, and every team should apply the first four. Then they hit a floor: booting processes and CUDA, profiling, capturing and warming up, which every new replica repeats. On a 180B MoE model that floor is most of the start: caches took Qwen3.8-Flash-Next only from 751 to 668 s.

03 · THE FLOORWhy the new replica stays slow

The replica that matters is the one you add during a traffic spike, on a node that has never run the model. Its page cache is empty, no Preload daemon holds its weights, and vLLM’s snapshots stay on the machine that made them. What it still has to do after every stock fix:

• Boot: 23 s for Qwen3-8B even with everything cached.
• Read the weights from the node’s disk or network at whatever speed they give.
• Profile, capture and warm up, per GPU, every time.
• Tune kernels that no cache covers. On GLM-5.3-Flash, autotuning 967 GEMMs was 17% of the 650 s start, about 110 s on every replica.

Every write-up that goes far enough hits this floor and names it. route179 ended with 98 s of “fixed engine overhead” after a streamer and a compile cache; Tensorfuse with 34 s of imports and engine init; our Qwen3-8B with 23 s of boot. The engines know it too. SGLang’s answer is a weight-cache daemon that keeps the loaded weights resident: on a 1T-parameter model it took weight loading from 495 s to 0.63 s and the whole restart from 8.8 to 0.5 minutes, on the same node (LMSYS). vLLM’s Preload is the same idea. Both make a restart fast; neither helps the replica on the next node.

Why doesn’t vLLM ship a portable snapshot itself? Its roadmap draws the line on purpose: vLLM “owns engine semantics and readiness”, capture systems own process and container capture, and deployment systems own storage, distribution and placement (#48193). The request for CUDA checkpoint/restore inside vLLM has been open since February 2026 with dozens of comments (RFC #34303), and what shipped is the single-GPU, same-machine snapshot above.

You can build the capture yourself. One engineer did, with CRIU and cuda-checkpoint around a 3B model, and took the cold start from 117 s to 5.8 s. The bill: seven extra container capabilities plus seccomp and AppArmor switched off, every thread pool pinned to one thread so that CRIU stops segfaulting on 70-odd threads, a patched CRIU that maps pages instead of copying them, a patched vLLM that reloads weights from a flat file, and the author’s own verdict that CRIU is “remarkably fiddly and temperamental” (j9s). That is the engineering warmctl packages.

Bottom line. Caches remove work that can be saved as files. The rest is the state of a running engine, and the only way to skip building it is to save the engine itself.

04 · WARMCTLwarmctl: restore an engine that is already warm

warmctl is an open-source tool (Apache-2.0) that does the cold start once and keeps the result. It starts vLLM, lets it load, compile, capture, tune and warm up, then freezes the ready engine into a snapshot: the memory of its processes, the GPU memory with the weights, compiled kernels and CUDA graphs, and the server’s state. A new replica restores the snapshot instead of starting vLLM, and answers in seconds.

It builds on three tools:

• CRIU (Checkpoint/Restore In Userspace), the standard Linux tool that saves a tree of running processes to disk and recreates it.
• cuda-checkpoint, NVIDIA’s utility that moves a process’s GPU state into host memory, so it is saved with the rest, and back after a restore.
• cuInterpose, NVIDIA’s library that lets the processes of one engine spread over several GPUs be checkpointed together.

In containers without the privileges CRIU needs, such as RunPod pods, warmctl uses DMTCP, which checkpoints from inside the processes. The snapshot is an OCI artifact, so it travels through any container registry, and you serve it with one command:

$ curl -fsSL https://warmctl.io/install.sh | sh
$ warmctl catalog
$ warmctl up qwen3-8b

The catalog has ready snapshots for 21 models. For your own model, flags and GPUs, warmctl build makes one (tutorial). Measured cold vLLM starts against warmctl restores from local storage:

A restore reads the whole snapshot, so its time depends on where the snapshot lives. From the page cache Qwen3-8B serves in 9.1 s, from our VM’s 0.44 GiB/s disk in 63 s, and from a RunPod network volume in about 21 s (RunPod post). Qwen3-32B on two L40S restores in 13.8 s against 251 s cold and 47 s with every stock cache.

Where it works today: the snapshot restores on machines with the same GPU model and count, NVIDIA driver branch and CPU features as the machine that built it; one node with one to eight GPUs; vLLM only. It is in beta. The status page lists what works and what does not yet.

Two other tools save a warm engine, each tied to a platform. Modal snapshots the CPU and GPU memory of a container before it takes requests, inside Modal’s serverless runtime (Modal). NVIDIA Dynamo Snapshot does CRIU and cuda-checkpoint as a Kubernetes operator with privileged node agents, and keeps its checkpoints in one cluster’s storage. The comparison page sets the three side by side.

Bottom line. Caches make the second start on a machine fast. A snapshot makes every new replica start like that, or faster, on any matching machine.

05 · CHOOSEWhich fix for which cold start

06 · FAQQuestions about vLLM cold starts

Why does vLLM take so long to start?

Before the first token vLLM boots Python and its processes, loads the weights, compiles the model with torch.compile, profiles memory for the KV cache, captures CUDA graphs, tunes kernels and warms up. Qwen3-8B on one L40S takes 134 s cold; large MoE models on eight GPUs take ten minutes or more.

Why does vLLM take 10 minutes to start?

Ten-minute starts come from large mixture-of-experts models on several GPUs. On GLM-5.3-Flash on 8×H100 the weights are 7% of a 650 s start; compiling, capturing CUDA graphs, autotuning GEMMs and warming up are the other 93%.

Does --enforce-eager make vLLM start faster?

It skips compilation and CUDA graphs, but serving gets slower. In our runs eager mode started Qwen3-8B in 47 s, slower than the 33 s of a normal start with a warm kernel cache.

Can I reuse the torch.compile cache on another machine?

Yes: copy the whole ~/.cache/vllm directory (VLLM_CACHE_ROOT), plus the Triton, FlashInfer and CUDA caches. The cache is keyed on the vLLM version, configuration, model code and GPU, so any change triggers a recompile.

Does a faster disk fix the vLLM cold start?

Only the weight loading step. It is half of a small model’s cold start and 7% of GLM-5.3-Flash’s on 8×H100.

vLLM takes forever to start: is it slow or hung?

If the log stops moving for minutes, it is a hang, not a slow start. Start with --load-format dummy to rule out the weights; on several GPUs set NCCL_DEBUG=INFO and check VLLM_HOST_IP, NCCL_SOCKET_IFNAME and that /dev/shm is mounted; with LoRA or unusual flags try --enforce-eager once to isolate CUDA graph capture. vLLM’s troubleshooting page lists the known hangs.

Why is even a small model slow to start?

The fixed costs do not shrink with the model: booting processes, initializing CUDA, compiling and tuning kernels. A 3B model took 117 s cold in one published test, and in our runs 55 of Qwen3-8B’s 134 s were boot alone, before any weights were read.

How is warmctl different from vLLM Preload and sleep mode?

Preload keeps weights in GPU memory on the same node for fast restarts, and sleep mode frees and refills GPUs inside one running server. warmctl saves the whole warm engine as a snapshot that restores on any machine with the same GPUs, driver branch and CPU features, so a new replica on a new node starts in seconds.

Sources

• Our runs: Qwen3-8B and Qwen3-32B on L40S with vLLM 0.30 (KServe post); GLM-5.3-Flash on 8×H100 (Nebius post); Qwen3.8-Flash-Next on RunPod (RunPod post); the model pairs on the homepage.
• vLLM docs: torch.compile, engine arguments, optimization levels, Preload, sleep mode, initialized snapshots, InstantTensor; vLLM blog: one uv flag.
• Write-ups: Tensorfuse, RunPod, route179, j9s, H Company, LMSYS, Modal; paper: Breaking the Ice: Analyzing Cold Start Latency in vLLM.
• vLLM issues: roadmap #48193, RFC #34303, PR #21146; troubleshooting for hangs.