--- title: "Start Qwen3.8-27B in 11 seconds on an RTX PRO 6000" description: "Cut Qwen3.8-27B startup on an NVIDIA RTX PRO 6000 from a 215-second vLLM cold start to an 11-second restore: a step-by-step warmctl tutorial with profile, build, OCI push and a throughput benchmark." author: Kirill Bespalov date: 2026-10-19 url: https://warmctl.io/blog/qwen38-27b-11-second-start-rtx-pro-6000/ --- # Start Qwen3.8-27B in 11 seconds on an RTX PRO 6000 > A cold vLLM start of Qwen3.8-27B takes 215 seconds: load, compile, capture CUDA graphs. Do it once, ship the warm engine as an OCI image, and every start after is an 11-second restore. Step by step, with the benchmark. **Key results** - On one NVIDIA RTX PRO 6000 Blackwell Server Edition (96 GB, driver 580.173.02), a vLLM cold start of Qwen3.8-27B FP8 takes 215 s to engine ready; warmctl up restores the warm engine in 11.0 s from page cache, 106.6 s from a cold page cache on a 0.44 GiB/s volume. - warmctl build of Qwen3.8-27B on the RTX PRO 6000 takes 337 s and produces a 38.15 GiB snapshot (27.72 GiB of weights, 9.9 GiB of CRIU images); the 48 GiB KV cache holds 755,547 tokens. - vllm bench serve on the restored engine, 1,024-token prompts and 512-token answers: 45 output tok/s at 1 request, 324 at 8, 916 at 32, 1,372 at 64; time to first token 0.76 s at 8K and 3.06 s at 32K prompt tokens. - warmctl hardware lists the GPUs warmctl supports and marks the host; warmctl doctor checks that the host GPU has a hardware row. vLLM cold start**215 s** warmctl up11.0 sQwen3.8-27B on one RTX PRO 6000, from start to the first served request. The cold start loads, compiles and captures; the warm start restores the result. ## Warm up once, start warm everywhere A vLLM engine does a lot of work before it answers its first request: it loads the weights, compiles the model, autotunes kernels and captures CUDA graphs for every batch size it will serve. Every new replica, restart and scale-out does all of it again. warmctl does that work once, takes a snapshot of the warm engine and ships it as an ordinary OCI image. Starting it later is a restore, not a cold start. This guide walks the whole path on one NVIDIA RTX PRO 6000 Blackwell with 96 GB of memory and **Qwen3.8-27B**, one of the most downloaded models on Hugging Face this month (7.7 million downloads), in its FP8 release: download it, describe the GPU and the engine in two small YAML files, build the warm image, run it, benchmark it, push it to a registry and start it from there. | You need | | |---|---| | GPU | NVIDIA RTX PRO 6000 (96 GB), driver 580 or newer | | Host | Linux x86-64, Docker with the NVIDIA container toolkit, root | | Disk | about 100 GB free: the model, the snapshot and the vLLM image | | Time | about 30 minutes, most of it pulling the 26 GB vLLM image and the one cold start | ## Install warmctl ``` $ curl -fsSL https://warmctl.io/install.sh | sudo sh warmctl-install: downloading warmctl-v2.0.0-beta.9-linux-amd64.tar.gz warmctl-install: checksum ok warmctl-install: installed warmctl v2.0.0-beta.9 in /opt/warmctl $ sudo sysctl -w kernel.io_uring_disabled=2 # CRIU cannot dump io_uring ``` The installer picks the newest release, checks its SHA-256 and puts `warmctl` and NVIDIA’s cuInterpose in `/opt/warmctl`; it took 2.1 s. Snapshots need io_uring off, because CRIU cannot dump it. ## Download the model from Hugging Face ``` $ python3 -m venv ~/hf && ~/hf/bin/pip install -U huggingface_hub hf_xet $ ~/hf/bin/hf download Qwen/Qwen3.8-27B-FP8 --revision 017b9c7af6b5689d5dd426a76e0bc077eb5ca20a \ --local-dir /models/Qwen3.8-27B-FP8 ``` 29 GB in about 100 s. Pin the revision: a snapshot is the warm state of one exact checkpoint, the profile records which, and the engine inside the container runs offline. ## Check the host and the GPU A snapshot restores only on the hardware it was built on: the same GPU model and count, the same driver version, a CPU with the same features. So warmctl keeps a table of the GPUs it supports and matches yours by its `nvidia-smi` name. `warmctl hardware` prints the table and marks this host: $ warmctl hardware ID NVIDIA-SMI NAME MEMORY ARCH NVLINK TOPOLOGY THIS HOST h100-80gb-sxm NVIDIA H100 80GB HBM3 81559 MiB sm_90 yes nvswitch h200-141gb NVIDIA H200 143771 MiB sm_90 yes nvswitch l40s-48gb NVIDIA L40S 46068 MiB sm_89 no pcie b200-180gb NVIDIA B200 183359 MiB sm_100 yes nvswitch rtx-pro-6000-96gb NVIDIA RTX PRO 6000 Blackwell Server Edition 97887 MiB sm_120 no pcie 1 GPU `doctor` checks the same row along with everything else a build needs: Docker, the driver, io_uring, the NVIDIA container runtime and the kernel. $ sudo warmctl doctor ok docker 29.8.2 (API 1.56) ok gpus 1 x NVIDIA RTX PRO 6000 Blackwell Server Edition, driver 580.173.02 ok hardware rtx-pro-6000-96gb (1 x NVIDIA RTX PRO 6000 Blackwell Server Edition) ok io_uring kernel.io_uring_disabled=2 (snapshots need 2) ok nvidia-runtime nvidia-ctk on PATH: true ok kernel 6.11.0-1016-nvidia The profile in the next step names the row, `rtx-pro-6000-96gb`. If your GPU is not in the table, `doctor` says `warn hardware` with the GPU’s name and `build` would stop. Describe it in a `hardware.yaml` next to your profile, one row per GPU model, with the same fields as the RTX PRO 6000 row warmctl ships: ``` # ~/warmctl/hardware.yaml schema: warmctl.hardware/v1 rows: - id: rtx-pro-6000-96gb nvidia_smi_name: "NVIDIA RTX PRO 6000 Blackwell Server Edition" memory_mib: 97887 # name and memory: nvidia-smi --query-gpu=name,memory.total arch: sm_120 # Blackwell, compute capability 12.0 nvlink: false topology_class: pcie # nvswitch for NVLink systems device_config_name: NVIDIA_RTX_PRO_6000_Blackwell_Server_Edition driver_min: "580" ``` ``` $ warmctl hardware --dir ~/warmctl # your row joins the table $ sudo warmctl doctor --profile-dir ~/warmctl # and the host check finds it ``` `build` reads the same directory through `--profile-dir`, as step 05 shows. ## Describe the engine in one YAML file A profile says which model to serve, on which hardware, with which vLLM settings, and what to warm up: whatever the warmup runs ends up compiled in the snapshot. Anything left out comes from warmctl’s defaults. This one is set up for serving, not for a demo. warmctl ships it as `qwen38-27b-rtx6000`, so `--profile qwen38-27b-rtx6000` works without the file; copy it into `~/warmctl` when you want to change something, and a file there wins over the shipped one: ``` # ~/warmctl/qwen38-27b-rtx6000.yaml schema: warmctl.profile/v1 id: qwen38-27b-rtx6000 model: id: Qwen/Qwen3.8-27B-FP8 revision: 017b9c7af6b5689d5dd426a76e0bc077eb5ca20a # pin it: a snapshot is this exact checkpoint served_names: [qwen38-27b] parallel: {tp: 1} hardware: [rtx-pro-6000-96gb] vllm: # any `vllm serve` flag max-model-len: 131072 # 128K context max-num-seqs: 64 # requests decoded at once max-num-batched-tokens: 8192 # prefill chunk kv-cache-memory-bytes: 51539607552 # 48 GiB of KV cache, room left for activations enable-chunked-prefill: true limit-mm-per-prompt: {image: 0, video: 0} # text only: skip the vision encoder's buffers mamba-cache-mode: align # prefix caching for the hybrid linear-attention layers graphs: # CUDA graphs captured before the snapshot mode: FULL_AND_PIECEWISE decode_batches: [1, 2, 4, 8, 16, 24, 32, 48, 64] warmup: # run once, so their kernels are compiled decode_prompt_tokens: 32 prefill_tokens: [512, 2048, 8192] resources: {memory: 128GiB} ``` The numbers follow from the card. The FP8 weights take about 29 GiB of the 96, 48 GiB go to the KV cache and the rest stays free for activations and CUDA graphs. Qwen3.8-27B mixes full-attention and linear-attention layers, and only 16 of its 64 layers keep a per-token cache; the other 48 are linear attention with a fixed-size state per sequence. So vLLM fits **755,547 tokens** into the 48 GiB, 5.76 full 128K contexts at once. ``` $ warmctl profile show ~/warmctl/qwen38-27b-rtx6000.yaml # merged over the defaults $ warmctl profile argv ~/warmctl/qwen38-27b-rtx6000.yaml # the exact command it will run vllm serve /model --compilation-config {"cudagraph_capture_sizes":[1,2,4,8,16,24,32,48,64],"cudagraph_mode":"FULL_AND_PIECEWISE","mode":3,…} --enable-chunked-prefill --enable-prefix-caching --enable-sleep-mode --kv-cache-memory-bytes 51539607552 --limit-mm-per-prompt {"image":0,"video":0} --mamba-cache-mode=align --max-model-len 131072 --max-num-batched-tokens 8192 --max-num-seqs 64 --served-model-name qwen38-27b --worker-extension-cls=warmctl_vllm.worker.Extension … ``` ## Build the warm image ``` $ sudo warmctl build --profile-dir ~/warmctl --profile qwen38-27b-rtx6000 \ --model-dir /models/Qwen3.8-27B-FP8 --gpus 0 --output /snaps/qwen38-27b [ 0.0s] profile qwen38-27b-rtx6000: 1 ranks, 9 graph sizes, 19 warmup cases [ 1.1s] donor warmctl-qwen38-27b started; loading the model [ 216.1s] engine ready # the cold start every replica pays without warmctl [ 216.2s] 1 workers: sm_120, vLLM 0.30.1rc1.dev616+g1a001d584.d20261003, torch 2.15.0.dev20260928+cu134 [ 216.9s] warmup 1/19 (512 tokens x 1: 0.75s) [ 219.3s] warmup 3/19 (8192 tokens x 1: 1.48s) [ 228.3s] warmup 12/19 (32 tokens x 64: 1.43s) [ 233.3s] warmup 19/19 (48 tokens x 1: 0.72s) [ 234.8s] canary recorded: 512-token prompt, miss 0.746s, hit 0.747s [ 300.0s] weights exported: 27.72 GiB in 1 ranks, slowest rank 65.1s [ 302.0s] CUDA checkpointed: 3 owners in 1.9s [ 308.2s] CRIU dump of 4 processes in 6.2s (criu 4.2.1) [ 336.9s] published /snaps/qwen38-27b: 340 files, 38.15 GiB (weights 27.72 GiB) in 5.5s [ 336.9s] snapshot completed: qwen38-27b: profile qwen38-27b-rtx6000, 1 ranks on rtx-pro-6000-96gb, 38.15 GiB ``` `build` starts vLLM in a container exactly as the profile says and pays the cold start once. Then it runs the warmup grid, records a canary (a prompt, its first token and log-probability), exports the weights, checkpoints the GPU with cuda-checkpoint and dumps the processes with CRIU: ## Start it warm ``` $ sudo warmctl up /snaps/qwen38-27b --gpus 0 --listen 127.0.0.1:8000 [ 0.0s] snapshot qwen38-27b: profile qwen38-27b-rtx6000, 1 ranks, 38.15 GiB [ 5.6s] trees installed in 0.7s, CRIU restore in 4.3s [ 6.5s] CUDA restored: 3 owners (1 in parallel) in 1.0s [ 9.5s] weights reloaded and fingerprints match: 27.72 GiB, slowest rank 2.9s [ 11.0s] canary matches (policy warn; miss 0.747s, hit 0.744s) [ 11.0s] serving qwen38-27b-rtx6000 on http://127.0.0.1:8000 ``` `up` checks the host against the snapshot, starts a fresh container, restores the processes and the GPU state at the same addresses, reloads the weights and compares the canary before it opens the port. Nothing is loaded, compiled or captured again. That is the warm path: the snapshot was still in the page cache from the build. With the cache dropped, every byte comes off the disk and the same restore took 106.6 s, because this VM reads its volume at 0.44 GiB/s and 38 GiB take that long to read. The disk sets that number, not the engine, and it is still half of the cold start. ``` $ curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{ "model": "qwen38-27b", "messages": [{"role": "user", "content": "What is the capital of France? One word."}], "chat_template_kwargs": {"enable_thinking": false}}' | jq -r '.choices[0].message.content' Paris ``` ## How fast it serves `vllm bench serve` against the restored engine, from a sidecar container on the same host: random prompts, `--ignore-eos` so every request writes all its tokens, about six minutes in all. First 1,024-token prompts with 512-token answers, from 1 to 64 requests at once: One request decodes at 46 tokens per second. Each token reads nearly all 27.7 GiB of weights, and the card’s 1.8 TB/s of memory bandwidth caps a single stream near 60. Batching shares that read: 64 requests make 1,372 tokens per second, 30 times one, and each still gets 24. Time to first token rises at 32 and 64 because new prompts queue behind 8,192-token prefill chunks; lower `max-num-batched-tokens` if first-token latency matters more than throughput. Then one request with a long prompt: | Prompt | First token | Per token | Tokens/s | |---|---|---|---| | 1K | 139 ms | 21.7 ms | 46 | | 8K | 761 ms | 22.0 ms | 45 | | 32K | 3,062 ms | 23.0 ms | 43 | Prefill runs at about 10,700 tokens per second, so a 32K-token prompt is answered in 3.1 s, and decoding barely slows down with context: three of every four layers keep a fixed-size state, so the attention the long prompt adds is a quarter of what a dense model of this size would pay. ## Push it to a registry, start it anywhere ``` $ sudo warmctl push /snaps/qwen38-27b ghcr.io/you/qwen38-27b # on any other host with an RTX PRO 6000 and a 580 driver: $ sudo warmctl up ghcr.io/you/qwen38-27b ``` The snapshot is one OCI artifact: `manifest.json` as its config and every file as content-addressed layers, so any registry stores it, GHCR, Harbor, ECR or your own. `push` tags it by platform and lists it under `latest`, so one repository can hold an image per GPU and `up` takes the one that fits the host. We pushed to a registry on the same host (89 s for 38 GiB) and ran `up` against an empty store: pulled and serving in 102 s. The download itself took 10 s; the rest of the pull was this VM’s disk writing 38 GiB, and the restore after it took 11 s, as before. ## Where to go from here - Kubernetes: the same `warmctl up NAME` runs in the runtime image as a pod ([guide](https://warmctl.io/docs/guides/kubernetes/)). - Skip the build: this exact image is in the [catalog](https://warmctl.io/catalog/). On an RTX PRO 6000 with driver 580.173.02, `sudo warmctl up qwen38-27b` pulls it and serves. A snapshot restores only on the driver it was built with, so other drivers need their own build. - The catalog also holds ready-made images for H100, H200, B200 and L40S.