Over 600K tokens/s: DeepSeek 4.1 Flash on 8×B300, vLLM vs DeepInfer

01 · WHYNew inference engine: DeepInfer

In October DeepSeek open-sourced DeepInfer, “the high-performance inference framework that powers DeepSeek’s large-scale production services”. It ships with kernels DeepSeek also publishes separately (FlashMLA, DeepGEMM, DeepEP, DeepSelect), data-parallel attention with expert-parallel MoE, and its DSpark speculative decoding. vLLM, which DeepInfer started from, has a recipe for the same model on B300 that uses most of the same kernels.

So the obvious question: on one 8×B300 node, the largest unit most teams rent, how far apart are they? We ran DeepSeek-V4.1-Flash-1017 on both and pushed each to the limit of the node, for output tokens and for prefill.

02 · THE ENGINESvLLM and DeepInfer: what’s different

Both engines run the same model with mostly the same kernels, and both scale it the same way: attention replicated per GPU, experts sharded across GPUs. The differences are in scope and packaging. vLLM is a general engine you install and serve from. DeepInfer is the core of DeepSeek’s own service, released without the service around it.

03 · SETUP8×B300, one model, two engines

The vLLM numbers are from an engine restored by warmctl, compared below with the same recipe started natively on the previous day’s node, a B300 SXM6 PC. Random prompts are the worst case for speculative decoding: DSpark accepted 19% of its drafts at 1K and 12% at 8K. Real text accepts more, so every decode number here is a floor.

04 · THE LAYOUTHow DEP8 puts the model on eight GPUs

The model is a mixture of experts: 384 routed experts and one shared expert per layer, 6 routed experts per token. Attention is a sparse variant of MLA: a sliding window of 128 tokens plus an indexer that picks the 512 most relevant earlier tokens, so the cost of a step barely grows with context. The two Engram tables, n-gram embeddings that the model looks up instead of computing, are 40% of the checkpoint.

vLLM’s recipe runs it as DEP8: data-parallel attention, expert-parallel MoE. Every GPU runs its own copy of attention with its own requests and KV cache, and holds 48 of the 384 experts. In every MoE layer the GPUs exchange tokens over NVSwitch so that each token reaches the six experts it was routed to, then exchange the results back.

Per GPU that leaves a lot of room. The weights a rank needs are 46 GiB; the profile reserves a fixed 160 GiB for the KV cache, in DeepSeek’s NVFP4 MLA format, where vLLM counts room for 52 million tokens per rank, 264 requests at the full 196K context. Memory is not what limits this node. The limit is 256 running sequences per rank, 2,048 for the node.

05 · vLLM AT FULL LOAD658K tokens/s in, 65K out

Prefill first, because that is where the big number is. We sent prompts with a single output token, so the node does nothing but read them. With 2,048 concurrent 8K prompts vLLM’s recipe processes 658,259 tokens/s, about 82K per GPU; with 512 concurrent 32K prompts, 589,924. Longer prompts are cheaper per token: 1K prompts at the same concurrency give 462,366.

For decode, 2,048 concurrent requests get 64,990 output tokens/s when prompts are short and 54,985 at 1K → 1K, about 6,900 per GPU, at 29 ms per token. 8K prompts bring output down to 29,500, but then the node is reading 235,996 prompt tokens/s on the side. 2,048 is also the ceiling: 256 sequences per rank. With 4,096 requests the extra ones wait in the queue and output stays at 55,506.

The decode numbers are from the engine warmctl restored; the prefill peaks are from the recipe started natively the day before, on a B300 SXM6 PC. Where we ran the same point on both they agree within 2%: 54,985 against 53,892 at 1K → 1K, 29,500 against 29,580 at 8K → 1K. A restore brings the engine back as it was, CUDA graphs and autotuned kernels included. At half the concurrency the restored engine prefilled at 554K and 557K.

06 · DEEPINFERBuilt for a rack, run on a node

Getting DeepInfer to run took longer than benchmarking it. It builds from source and wants NVCC 13.1 or newer, which we took from NVIDIA’s CUDA 13.2 wheels. It moves Engram rows between GPUs with NCCL’s GPU-initiated networking, which needs an RDMA device even on one node: on an 8×B200 VM without a working one it did not start (DeepEP#789), and on this B300 it ran with driver 580 but not with 610.

The open-source release is offline only: no HTTP server, requests go through LLM.generate. We sent it the same random-token workloads in-process, which spares it the API server, the tokenizer and the network that vLLM’s numbers include. We ran it twice: with its defaults, and tuned the way DeepSeek’s README and example scripts set up V4.1.

Bounded replay is the setting that matters most on paper. V4.1’s layers 21 to 39 reuse the KV cache of four earlier layers and attend through a 128-token window, so a prompt’s early tokens never need to pass through them; with replay on, DeepInfer runs only the last 140 tokens of each prompt through those layers.

On decode vLLM is ahead everywhere: 1.8× with short prompts, 2.0× at 1K → 1K and 4.0× at 8K → 1K, where each request first prefills 8K tokens. DeepInfer’s best 1K → 1K number, 27,906 output tokens/s, came from the default run; the tuned run made 21,161, most likely because EPLB moves experts every 30 steps, which random tokens, already evenly spread over experts, do not reward.

Prefill is the bigger gap. Tuned, DeepInfer reads 71,632 tokens/s of 8K prompts and 71,875 of 32K prompts, about 9K per GPU; vLLM reads 658,259 and 589,924. Bounded replay raised DeepInfer’s 32K number by 11%, from 64,847, not by the factor of two the skipped layers suggest, so the time goes elsewhere.

DeepSeek’s own prefill curve shows 82–89K tokens/s per GPU, about nine times what we measured. It comes from a different setup: 4 GPUs per prefill group, expert load forced to balance (DEEPINFER_FORCE_LOAD_BALANCE=1 in its benchmark script), and 4K-token prompt suffixes on top of a cached context. Our node also lacked MLOPart, which DeepGEMM uses to keep MoE buffers on the right half of a B300 and which driver 580 does not offer. We could not reproduce DeepSeek’s number on one node, and we have not profiled DeepInfer’s prefill.

The README says what DeepInfer is built for: decode on 64 GPUs and prefill on 4, “each within one scale-up domain”, with prefill and decode on separate machines. The parts that hold such a deployment together, a KV cache system, prefill/decode disaggregation and load balancing, are not in the open-source release. On our node DeepInfer had to do everything at once, in the same eight engines, and its scheduler runs a whole step as either prefill or decode, never both. That is the setup vLLM’s recipe is built for, and not the one DeepInfer is.

07 · STARTUP15.5 minutes, or 88 seconds

The engine’s log shows where 930 s of a cold start go, with empty caches on a fresh disk. Loading weights is the small part: 96 s, Engram tables included. The rest is getting the GPUs ready: 84 s of torch.compile and KV setup, 110 s of JIT kernel warmup, 319 s of FlashInfer autotuning over 21 token buckets, and 199 s capturing 67 CUDA graphs. vLLM’s recipe, started natively on the previous day’s node, took 24 minutes to engine ready. DeepInfer loaded in 298 s with a warm JIT cache and 748 s with a cold one.

warmctl does the slow part once: it builds the engine, warms it up, and snapshots it whole: the process memory of all 27 processes (CRIU), the CUDA state of every GPU (cuda-checkpoint), compiled kernels, the autotuning results and CUDA graphs, and the 372 GiB of weights as the engine laid them out, 851 GiB in all. warmctl up restores it in 88.4 s from page cache: 43 s for the processes, 22 s for the GPU state, 12 s to reload the weights, then a canary prompt compared with the original engine before the port opens. The node read the snapshot at 155 GiB/s from page cache. Every answer we checked with greedy decoding after the restore was right.

The snapshot is in the warmctl catalog. On an 8×B300 host with NVIDIA driver 610.57.04:

DeepSeekMIT

DeepSeek-V4.1-Flash-1017

MoE · Engram · DSpark · Oct 17 update · DEP8

• 8×B300·88 s·851 GiB·610

restore88 s cold start930 s

deepseek-v41-flash-1017Run

$ curl -fsSL https://warmctl.io/install.sh | sh
$ warmctl up deepseek-v41-flash-1017

up picks the snapshot for the host’s GPUs and driver and serves an OpenAI-compatible API on port 8000:

$ curl -s localhost:8000/v1/completions -H 'Content-Type: application/json' -d '{
    "model": "deepseek-v41-flash-1017", "prompt": "The capital of France is", "max_tokens": 8, "temperature": 0}' | jq -r '.choices[0].text'
 Paris.

The profile holds the vLLM flags, the graph sizes and the warmup grid. The same model is in the catalog for 8×B200 (DEP8, driver 615), 4×B200 (TP4) and 8×B300 with driver 580 (TP8).

08 · WHICH ENGINE, WHENChoosing between vLLM and DeepInfer

The benchmarks say how fast each engine is on one node. Which one to pick depends on what you are building.

• An API for your users on one or a few nodes: vLLM. On this node it made 2–4× the output tokens and 8× the prefill, and it ships as a server with an OpenAI-compatible API. DeepInfer’s release has no server at all.
• Any model other than DeepSeek-V4.1-Flash: vLLM. DeepInfer supports this one model.
• Reproducible offline generation, such as RL rollouts, evals or synthetic data: try DeepInfer. Batch-invariant inference and deterministic sampling are built in and documented for this model, and an offline batch API is what such jobs use anyway. Measure throughput on your own shapes first; on ours it was a half to a quarter of vLLM’s on decode.
• A DeepSeek service on rack-scale NVLink (GB200 or GB300 NVL72) with a team to run it: look at both. DeepInfer is how DeepSeek serves the model at that scale, but you would build the KV cache system, the disaggregation and the load balancing yourself. vLLM covers that ground with disaggregated prefill through KV connectors.
• Huawei Ascend 950: DeepInfer, which supports it next to NVIDIA SM100.
• Autoscaling, spot capacity or frequent rollouts: whichever engine you run, the cold start decides. This one takes 15 minutes per node before the first token, and vLLM’s recipe took 24 on another node. With vLLM, warmctl up brings the same engine back in 88 seconds.

For most teams with a node or two, the answer is vLLM. DeepInfer is worth following as the reference for how DeepSeek runs its own model, and worth a try for deterministic offline work. On a single node, with the settings DeepSeek recommends, it is not the faster engine.