Start Qwen3.8-27B in 11 seconds on an RTX PRO 6000

00 · THE IDEAWarm up once, start warm everywhere

A vLLM engine does a lot of work before it answers its first request: it loads the weights, compiles the model, autotunes kernels and captures CUDA graphs for every batch size it will serve. Every new replica, restart and scale-out does all of it again. warmctl does that work once, takes a snapshot of the warm engine and ships it as an ordinary OCI image. Starting it later is a restore, not a cold start.

This guide walks the whole path on one NVIDIA RTX PRO 6000 Blackwell with 96 GB of memory and Qwen3.8-27B, one of the most downloaded models on Hugging Face this month (7.7 million downloads), in its FP8 release: download it, describe the GPU and the engine in two small YAML files, build the warm image, run it, benchmark it, push it to a registry and start it from there.

01 · INSTALLInstall warmctl

$ curl -fsSL https://warmctl.io/install.sh | sudo sh
warmctl-install: downloading warmctl-v2.0.0-beta.9-linux-amd64.tar.gz
warmctl-install: checksum ok
warmctl-install: installed warmctl v2.0.0-beta.9 in /opt/warmctl
$ sudo sysctl -w kernel.io_uring_disabled=2   # CRIU cannot dump io_uring

The installer picks the newest release, checks its SHA-256 and puts warmctl and NVIDIA’s cuInterpose in /opt/warmctl; it took 2.1 s. Snapshots need io_uring off, because CRIU cannot dump it.

02 · MODELDownload the model from Hugging Face

$ python3 -m venv ~/hf && ~/hf/bin/pip install -U huggingface_hub hf_xet
$ ~/hf/bin/hf download Qwen/Qwen3.8-27B-FP8 --revision 017b9c7af6b5689d5dd426a76e0bc077eb5ca20a \
    --local-dir /models/Qwen3.8-27B-FP8

29 GB in about 100 s. Pin the revision: a snapshot is the warm state of one exact checkpoint, the profile records which, and the engine inside the container runs offline.

03 · HARDWARECheck the host and the GPU

A snapshot restores only on the hardware it was built on: the same GPU model and count, the same driver version, a CPU with the same features. So warmctl keeps a table of the GPUs it supports and matches yours by its nvidia-smi name. warmctl hardware prints the table and marks this host:

$ warmctl hardware
ID                 NVIDIA-SMI NAME                               MEMORY      ARCH    NVLINK  TOPOLOGY  THIS HOST
h100-80gb-sxm      NVIDIA H100 80GB HBM3                         81559 MiB   sm_90   yes     nvswitch
h200-141gb         NVIDIA H200                                   143771 MiB  sm_90   yes     nvswitch
l40s-48gb          NVIDIA L40S                                   46068 MiB   sm_89   no      pcie
b200-180gb         NVIDIA B200                                   183359 MiB  sm_100  yes     nvswitch
rtx-pro-6000-96gb  NVIDIA RTX PRO 6000 Blackwell Server Edition  97887 MiB   sm_120  no      pcie      1 GPU

doctor checks the same row along with everything else a build needs: Docker, the driver, io_uring, the NVIDIA container runtime and the kernel.

$ sudo warmctl doctor
ok   docker           29.8.2 (API 1.56)
ok   gpus             1 x NVIDIA RTX PRO 6000 Blackwell Server Edition, driver 580.173.02
ok   hardware         rtx-pro-6000-96gb (1 x NVIDIA RTX PRO 6000 Blackwell Server Edition)
ok   io_uring         kernel.io_uring_disabled=2 (snapshots need 2)
ok   nvidia-runtime   nvidia-ctk on PATH: true
ok   kernel           6.11.0-1016-nvidia

The profile in the next step names the row, rtx-pro-6000-96gb. If your GPU is not in the table, doctor says warn hardware with the GPU’s name and build would stop. Describe it in a hardware.yaml next to your profile, one row per GPU model, with the same fields as the RTX PRO 6000 row warmctl ships:

# ~/warmctl/hardware.yaml
schema: warmctl.hardware/v1
rows:
  - id: rtx-pro-6000-96gb
    nvidia_smi_name: "NVIDIA RTX PRO 6000 Blackwell Server Edition"
    memory_mib: 97887         # name and memory: nvidia-smi --query-gpu=name,memory.total
    arch: sm_120              # Blackwell, compute capability 12.0
    nvlink: false
    topology_class: pcie      # nvswitch for NVLink systems
    device_config_name: NVIDIA_RTX_PRO_6000_Blackwell_Server_Edition
    driver_min: "580"

$ warmctl hardware --dir ~/warmctl            # your row joins the table
$ sudo warmctl doctor --profile-dir ~/warmctl  # and the host check finds it

build reads the same directory through --profile-dir, as step 05 shows.

04 · PROFILEDescribe the engine in one YAML file

A profile says which model to serve, on which hardware, with which vLLM settings, and what to warm up: whatever the warmup runs ends up compiled in the snapshot. Anything left out comes from warmctl’s defaults. This one is set up for serving, not for a demo. warmctl ships it as qwen38-27b-rtx6000, so --profile qwen38-27b-rtx6000 works without the file; copy it into ~/warmctl when you want to change something, and a file there wins over the shipped one:

# ~/warmctl/qwen38-27b-rtx6000.yaml
schema: warmctl.profile/v1
id: qwen38-27b-rtx6000
model:
  id: Qwen/Qwen3.8-27B-FP8
  revision: 017b9c7af6b5689d5dd426a76e0bc077eb5ca20a   # pin it: a snapshot is this exact checkpoint
  served_names: [qwen38-27b]
parallel: {tp: 1}
hardware: [rtx-pro-6000-96gb]
vllm:                                   # any `vllm serve` flag
  max-model-len: 131072                 # 128K context
  max-num-seqs: 64                      # requests decoded at once
  max-num-batched-tokens: 8192          # prefill chunk
  kv-cache-memory-bytes: 51539607552    # 48 GiB of KV cache, room left for activations
  enable-chunked-prefill: true
  limit-mm-per-prompt: {image: 0, video: 0}   # text only: skip the vision encoder's buffers
  mamba-cache-mode: align               # prefix caching for the hybrid linear-attention layers
graphs:                                 # CUDA graphs captured before the snapshot
  mode: FULL_AND_PIECEWISE
  decode_batches: [1, 2, 4, 8, 16, 24, 32, 48, 64]
warmup:                                 # run once, so their kernels are compiled
  decode_prompt_tokens: 32
  prefill_tokens: [512, 2048, 8192]
resources: {memory: 128GiB}

The numbers follow from the card. The FP8 weights take about 29 GiB of the 96, 48 GiB go to the KV cache and the rest stays free for activations and CUDA graphs. Qwen3.8-27B mixes full-attention and linear-attention layers, and only 16 of its 64 layers keep a per-token cache; the other 48 are linear attention with a fixed-size state per sequence. So vLLM fits 755,547 tokens into the 48 GiB, 5.76 full 128K contexts at once.

$ warmctl profile show ~/warmctl/qwen38-27b-rtx6000.yaml   # merged over the defaults
$ warmctl profile argv ~/warmctl/qwen38-27b-rtx6000.yaml   # the exact command it will run
vllm serve /model --compilation-config {"cudagraph_capture_sizes":[1,2,4,8,16,24,32,48,64],"cudagraph_mode":"FULL_AND_PIECEWISE","mode":3,…} --enable-chunked-prefill --enable-prefix-caching --enable-sleep-mode --kv-cache-memory-bytes 51539607552 --limit-mm-per-prompt {"image":0,"video":0} --mamba-cache-mode=align --max-model-len 131072 --max-num-batched-tokens 8192 --max-num-seqs 64 --served-model-name qwen38-27b --worker-extension-cls=warmctl_vllm.worker.Extension …

05 · BUILDBuild the warm image

$ sudo warmctl build --profile-dir ~/warmctl --profile qwen38-27b-rtx6000 \
    --model-dir /models/Qwen3.8-27B-FP8 --gpus 0 --output /snaps/qwen38-27b
[    0.0s] profile qwen38-27b-rtx6000: 1 ranks, 9 graph sizes, 19 warmup cases
[    1.1s] donor warmctl-qwen38-27b started; loading the model
[  216.1s] engine ready             # the cold start every replica pays without warmctl
[  216.2s] 1 workers: sm_120, vLLM 0.30.1rc1.dev616+g1a001d584.d20261003, torch 2.15.0.dev20260928+cu134
[  216.9s] warmup 1/19 (512 tokens x 1: 0.75s)
[  219.3s] warmup 3/19 (8192 tokens x 1: 1.48s)
[  228.3s] warmup 12/19 (32 tokens x 64: 1.43s)
[  233.3s] warmup 19/19 (48 tokens x 1: 0.72s)
[  234.8s] canary recorded: 512-token prompt, miss 0.746s, hit 0.747s
[  300.0s] weights exported: 27.72 GiB in 1 ranks, slowest rank 65.1s
[  302.0s] CUDA checkpointed: 3 owners in 1.9s
[  308.2s] CRIU dump of 4 processes in 6.2s (criu 4.2.1)
[  336.9s] published /snaps/qwen38-27b: 340 files, 38.15 GiB (weights 27.72 GiB) in 5.5s
[  336.9s] snapshot completed: qwen38-27b: profile qwen38-27b-rtx6000, 1 ranks on rtx-pro-6000-96gb, 38.15 GiB

build starts vLLM in a container exactly as the profile says and pays the cold start once. Then it runs the warmup grid, records a canary (a prompt, its first token and log-probability), exports the weights, checkpoints the GPU with cuda-checkpoint and dumps the processes with CRIU:

06 · RUNStart it warm

$ sudo warmctl up /snaps/qwen38-27b --gpus 0 --listen 127.0.0.1:8000
[    0.0s] snapshot qwen38-27b: profile qwen38-27b-rtx6000, 1 ranks, 38.15 GiB
[    5.6s] trees installed in 0.7s, CRIU restore in 4.3s
[    6.5s] CUDA restored: 3 owners (1 in parallel) in 1.0s
[    9.5s] weights reloaded and fingerprints match: 27.72 GiB, slowest rank 2.9s
[   11.0s] canary matches (policy warn; miss 0.747s, hit 0.744s)
[   11.0s] serving qwen38-27b-rtx6000 on http://127.0.0.1:8000

up checks the host against the snapshot, starts a fresh container, restores the processes and the GPU state at the same addresses, reloads the weights and compares the canary before it opens the port. Nothing is loaded, compiled or captured again. That is the warm path: the snapshot was still in the page cache from the build. With the cache dropped, every byte comes off the disk and the same restore took 106.6 s, because this VM reads its volume at 0.44 GiB/s and 38 GiB take that long to read. The disk sets that number, not the engine, and it is still half of the cold start.

$ curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
    "model": "qwen38-27b", "messages": [{"role": "user", "content": "What is the capital of France? One word."}],
    "chat_template_kwargs": {"enable_thinking": false}}' | jq -r '.choices[0].message.content'
Paris

07 · BENCHMARKHow fast it serves

vllm bench serve against the restored engine, from a sidecar container on the same host: random prompts, --ignore-eos so every request writes all its tokens, about six minutes in all. First 1,024-token prompts with 512-token answers, from 1 to 64 requests at once:

One request decodes at 46 tokens per second. Each token reads nearly all 27.7 GiB of weights, and the card’s 1.8 TB/s of memory bandwidth caps a single stream near 60. Batching shares that read: 64 requests make 1,372 tokens per second, 30 times one, and each still gets 24. Time to first token rises at 32 and 64 because new prompts queue behind 8,192-token prefill chunks; lower max-num-batched-tokens if first-token latency matters more than throughput.

Then one request with a long prompt:

Prefill runs at about 10,700 tokens per second, so a 32K-token prompt is answered in 3.1 s, and decoding barely slows down with context: three of every four layers keep a fixed-size state, so the attention the long prompt adds is a quarter of what a dense model of this size would pay.

08 · SHIPPush it to a registry, start it anywhere

$ sudo warmctl push /snaps/qwen38-27b ghcr.io/you/qwen38-27b
# on any other host with an RTX PRO 6000 and a 580 driver:
$ sudo warmctl up ghcr.io/you/qwen38-27b

The snapshot is one OCI artifact: manifest.json as its config and every file as content-addressed layers, so any registry stores it, GHCR, Harbor, ECR or your own. push tags it by platform and lists it under latest, so one repository can hold an image per GPU and up takes the one that fits the host. We pushed to a registry on the same host (89 s for 38 GiB) and ran up against an empty store: pulled and serving in 102 s. The download itself took 10 s; the rest of the pull was this VM’s disk writing 38 GiB, and the restore after it took 11 s, as before.

09 · NEXTWhere to go from here

• Kubernetes: the same warmctl up NAME runs in the runtime image as a pod (guide).
• Skip the build: this exact image is in the catalog. On an RTX PRO 6000 with driver 580.173.02, sudo warmctl up qwen38-27b pulls it and serves. A snapshot restores only on the driver it was built with, so other drivers need their own build.
• The catalog also holds ready-made images for H100, H200, B200 and L40S.