---
title: "Over 600K tokens/s: DeepSeek 4.1 Flash on 8×B300, vLLM vs DeepInfer"
description: "DeepSeek-V4.1-Flash-1017 on one 8×B300 node: vLLM DEP8 reaches 55K output and 658K prefill tokens/s; DeepInfer, tuned as DeepSeek recommends, 28K and 72K. How each engine is built, and an 88 s warm start."
author: Kirill Bespalov
date: 2026-10-21
url: https://warmctl.io/blog/deepseek-b300-vllm-vs-deepinfer/
---

# Over 600K tokens/s: DeepSeek 4.1 Flash on 8×B300, vLLM vs DeepInfer

> DeepSeek open-sourced DeepInfer, the engine behind its own service. We ran it against vLLM on one 8×B300 node with DeepSeek-V4.1-Flash-1017: how the model sits in memory, how far each engine goes on output tokens and prefill, and why DeepInfer trails on a single node.

**Key results**

- DeepSeek-V4.1-Flash-1017 on one 8×B300 node with vLLM 0.31 (DEP8: data-parallel attention, expert-parallel MoE, DSpark with 5 draft tokens): 54,985 output tok/s at 1K-token prompts and 1K-token answers with 2,048 concurrent requests, 64,990 with 128-token prompts, 29,500 at 8K → 1K.
- Prefill on 8×B300 with vLLM's B300 recipe: 658,259 tok/s with 8K-token prompts, 589,924 with 32K-token prompts.
- DeepInfer (deepseek-ai/DeepInfer) on 8×B300, EP8 × DP8, offline, with bounded replay, Engram in GPU memory and EPLB: 36,381 output tok/s at 128 → 1K, 21,161 at 1K → 1K (27,906 with defaults), 7,297 at 8K → 1K, 71,632 prefill tok/s with 8K prompts and 71,875 with 32K prompts.
- A DEP8 vLLM engine with CUDA graphs only up to 1,536 tokens made 37,239 output tok/s at 1K → 1K; graphs up to 8,192 tokens brought it to 54,985.
- A vLLM cold start of the DEP8 engine on 8×B300 takes 930 s to engine ready (FlashInfer autotune 319 s, CUDA graph capture 199 s); warmctl up restores the 851 GiB snapshot in 88.4 s from page cache.

**Updated October 21.** The first version of this post ran DeepInfer with its defaults. We reran it the same day with the settings DeepSeek recommends for V4.1, bounded replay included; the tables show both runs, and every comparison uses the better of the two.

- 658K prefill tokens/s on one node, vLLM, 8K prompts
- 55.0K output tokens/s at 1K → 1K; 65.0K with short prompts
- 2–4× vLLM over DeepInfer on decode, 8× on prefill
- 88 s warmctl up, against a 930 s cold start

## New inference engine: DeepInfer

In October DeepSeek open-sourced [DeepInfer](https://github.com/deepseek-ai/DeepInfer), “the high-performance inference framework that powers DeepSeek’s large-scale production services”. It ships with kernels DeepSeek also publishes separately (FlashMLA, DeepGEMM, DeepEP, DeepSelect), data-parallel attention with expert-parallel MoE, and its DSpark speculative decoding. vLLM, which DeepInfer started from, has a recipe for the same model on B300 that uses most of the same kernels.

So the obvious question: on one 8×B300 node, the largest unit most teams rent, how far apart are they? We ran **DeepSeek-V4.1-Flash-1017** on both and pushed each to the limit of the node, for output tokens and for prefill.

## vLLM and DeepInfer: what’s different

Both engines run the same model with mostly the same kernels, and both scale it the same way: attention replicated per GPU, experts sharded across GPUs. The differences are in scope and packaging. vLLM is a general engine you install and serve from. DeepInfer is the core of DeepSeek’s own service, released without the service around it.

  vLLMDeepInferWhat it isGeneral-purpose open-source inference engine and serverDeepSeek’s production engine, open-sourced for one modelModelsHundreds of open models, this one includedDeepSeek-V4.1-Flash onlyHardwareNVIDIA, AMD, TPU and moreNVIDIA SM100 (B200, B300, GB200) and Huawei Ascend 950InterfaceOpenAI-compatible HTTP server with tool calls and reasoning parsersPython API, offline batches (`LLM.generate`), no serverInstallDocker image or pip wheelBuild from source: GCC 13, CUDA 13 with NVCC 13.1 or newerStands out withPrefix caching, structured outputs, LoRA, disaggregated prefill through KV connectorsDSpark with confidence-scheduled verification, batch-invariant inference, deterministic samplingNot included—KV cache system, prefill/decode disaggregation, load balancingBuilt forAny scale, from one GPU up4 GPUs per prefill group, 64 per decode group, in one NVLink domain

## 8×B300, one model, two engines

  System setupGPUs8× NVIDIA B300 SXM6 AC, 275 GB of HBM3e each, NVLink 5 through NVSwitchHost192 vCPUs, 2.7 TB RAM, six local NVMe drives in RAID 0 (21 TB)Model`deepseek-ai/DeepSeek-V4.1-Flash-1017`, 476 GB: FP4 experts, FP8 everything else, two Engram tables of 384M rowsvLLM0.31.1rc1, the image of [vLLM’s B300 recipe](https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4.1-Flash), NVIDIA driver 610.57.04; DEP8 with 256 sequences and 8K-token steps per rank, DSpark with 5 draft tokensDeepInfer`deepseek-ai/DeepInfer` at 8b16149 (October 20), built from source with CUDA 13.2; EP8 × DP8, DSpark with 5 draft tokens, offline API; B300 SXM6 AC, driver 580warmctlv2.0.0-beta.10, snapshot `deepseek-v41-flash-1017:8xb300ac-275gb-610.57.04`, 851 GiBLoad`vllm bench serve` against the OpenAI API: random-token prompts, fixed output length (`--ignore-eos`), twice as many prompts as the concurrency; DeepInfer gets the same prompts in-process through `LLM.generate`

The vLLM numbers are from an engine restored by warmctl, compared below with the same recipe started natively on the previous day’s node, a B300 SXM6 PC. Random prompts are the worst case for speculative decoding: DSpark accepted 19% of its drafts at 1K and 12% at 8K. Real text accepts more, so every decode number here is a floor.

## How DEP8 puts the model on eight GPUs

The model is a mixture of experts: 384 routed experts and one shared expert per layer, 6 routed experts per token. Attention is a sparse variant of MLA: a sliding window of 128 tokens plus an indexer that picks the 512 most relevant earlier tokens, so the cost of a step barely grows with context. The two **Engram** tables, n-gram embeddings that the model looks up instead of computing, are 40% of the checkpoint.

vLLM’s recipe runs it as **DEP8**: data-parallel attention, expert-parallel MoE. Every GPU runs its own copy of attention with its own requests and KV cache, and holds 48 of the 384 experts. In every MoE layer the GPUs exchange tokens over NVSwitch so that each token reaches the six experts it was routed to, then exchange the results back.

Per GPU that leaves a lot of room. The weights a rank needs are 46 GiB; the profile reserves a fixed 160 GiB for the KV cache, in DeepSeek’s NVFP4 MLA format, where vLLM counts room for **52 million tokens per rank**, 264 requests at the full 196K context. Memory is not what limits this node. The limit is 256 running sequences per rank, 2,048 for the node.

## 658K tokens/s in, 65K out

Prefill first, because that is where the big number is. We sent prompts with a single output token, so the node does nothing but read them. With 2,048 concurrent 8K prompts vLLM’s recipe processes **658,259 tokens/s**, about 82K per GPU; with 512 concurrent 32K prompts, 589,924. Longer prompts are cheaper per token: 1K prompts at the same concurrency give 462,366.

For decode, 2,048 concurrent requests get **64,990 output tokens/s** when prompts are short and 54,985 at 1K → 1K, about 6,900 per GPU, at 29 ms per token. 8K prompts bring output down to 29,500, but then the node is reading 235,996 prompt tokens/s on the side. 2,048 is also the ceiling: 256 sequences per rank. With 4,096 requests the extra ones wait in the queue and output stays at 55,506.

| In → out | Concurrent | Output tok/s | Total tok/s | Native, out | TPOT |
|---|---|---|---|---|---|
| 128 → 1K | 2,048 | 64,990 | 73,114 | — | 24.8 ms |
| 1K → 1K | 2,048 | 54,985 | 109,969 | 53,892 | 28.8 ms |
| 1K → 1K | 4,096 | — | — | 55,506 | 32.8 ms |
| 8K → 1K | 2,048 | 29,500 | 265,496 | 29,580 | 52.8 ms |
| 2K → 128 | 2,048 | 19,753 | 335,801 | — | 74.4 ms |

The decode numbers are from the engine warmctl restored; the prefill peaks are from the recipe started natively the day before, on a B300 SXM6 PC. Where we ran the same point on both they agree within 2%: 54,985 against 53,892 at 1K → 1K, 29,500 against 29,580 at 8K → 1K. A restore brings the engine back as it was, CUDA graphs and autotuned kernels included. At half the concurrency the restored engine prefilled at 554K and 557K.

  Our first restored engine made only **37.2K** output tokens/s at 1K → 1K. It captured CUDA graphs up to 1,536 tokens, the recipe up to 8,190. Under load most steps mix decode with a prompt chunk, and in data-parallel mode every rank pads to the largest one, so those steps ran without graphs. Graphs up to 8,192 tokens brought it to 55.0K ([the profile change](https://github.com/TensorRails/warmctl/pull/28)).

## Built for a rack, run on a node

Getting DeepInfer to run took longer than benchmarking it. It builds from source and wants NVCC 13.1 or newer, which we took from NVIDIA’s CUDA 13.2 wheels. It moves Engram rows between GPUs with NCCL’s GPU-initiated networking, which needs an RDMA device even on one node: on an 8×B200 VM without a working one it did not start ([DeepEP#789](https://github.com/deepseek-ai/DeepEP/issues/789)), and on this B300 it ran with driver 580 but not with 610.

The open-source release is offline only: no HTTP server, requests go through `LLM.generate`. We sent it the same random-token workloads in-process, which spares it the API server, the tokenizer and the network that vLLM’s numbers include. We ran it twice: with its defaults, and tuned the way DeepSeek’s README and example scripts set up V4.1.

  DeepInfer settingTuned run (October 21)WhyParallelismEP8 × DP8: eight engines, experts across all eight GPUsas DeepSeek runs itBounded replay`YOCO_PREFILL_TRUNCATION_MODE=compact`, 140 tokensREADME: the default for V4.1DSpark5 draft tokens, confidence-scheduled verification (`ENABLE_MTP_VARLEN=1`)READMEExperts32 redundant experts, EPLB every 30 stepsDeepSeek’s example scriptsEngramtables in GPU memory (`ENGRAM_FORCE_GPU_STORAGE=1`)B300 has the room; no 190 GiB of pinned host memoryBatching256 sequences and 32K-token prefill steps per ranksame limits as vLLM; DeepSeek’s benchmark default stepHostdriver 580, CUDA 13.2, unlimited locked memory, RDMA NICsDeepInfer did not start on driver 610

Bounded replay is the setting that matters most on paper. V4.1’s layers 21 to 39 reuse the KV cache of four earlier layers and attend through a 128-token window, so a prompt’s early tokens never need to pass through them; with replay on, DeepInfer runs only the last 140 tokens of each prompt through those layers.

On decode vLLM is ahead everywhere: **1.8×** with short prompts, **2.0×** at 1K → 1K and **4.0×** at 8K → 1K, where each request first prefills 8K tokens. DeepInfer’s best 1K → 1K number, 27,906 output tokens/s, came from the default run; the tuned run made 21,161, most likely because EPLB moves experts every 30 steps, which random tokens, already evenly spread over experts, do not reward.

Prefill is the bigger gap. Tuned, DeepInfer reads **71,632 tokens/s** of 8K prompts and 71,875 of 32K prompts, about 9K per GPU; vLLM reads 658,259 and 589,924. Bounded replay raised DeepInfer’s 32K number by 11%, from 64,847, not by the factor of two the skipped layers suggest, so the time goes elsewhere.

| Workload | Concurrent | vLLM | DeepInfer, tuned | DeepInfer, defaults | vLLM ÷ best |
|---|---|---|---|---|---|
| 128 → 1K | 2,048 | 64,990 | 36,381 | — | 1.8× |
| 1K → 1K | 2,048 | 54,985 | 21,161 | 27,906 | 2.0× |
| 1K → 1K | 4,096 | 55,506 | — | 26,294 | 2.1× |
| 8K → 1K | 2,048 | 29,500 | 7,297 | 6,933 | 4.0× |
| prefill 8K | 2,048 | 658,259 | 71,632 | — | 9.2× |
| prefill 32K | 512 | 589,924 | 71,875 | 64,847 | 8.2× |

DeepSeek’s own prefill curve shows 82–89K tokens/s per GPU, about nine times what we measured. It comes from a different setup: 4 GPUs per prefill group, expert load forced to balance (`DEEPINFER_FORCE_LOAD_BALANCE=1` in its benchmark script), and 4K-token prompt suffixes on top of a cached context. Our node also lacked MLOPart, which DeepGEMM uses to keep MoE buffers on the right half of a B300 and which driver 580 does not offer. We could not reproduce DeepSeek’s number on one node, and we have not profiled DeepInfer’s prefill.

The README says what DeepInfer is built for: decode on 64 GPUs and prefill on 4, “each within one scale-up domain”, with prefill and decode on separate machines. The parts that hold such a deployment together, a KV cache system, prefill/decode disaggregation and load balancing, are not in the open-source release. On our node DeepInfer had to do everything at once, in the same eight engines, and its scheduler runs a whole step as either prefill or decode, never both. That is the setup vLLM’s recipe is built for, and not the one DeepInfer is.

## 15.5 minutes, or 88 seconds

The engine’s log shows where 930 s of a cold start go, with empty caches on a fresh disk. Loading weights is the small part: 96 s, Engram tables included. The rest is getting the GPUs ready: 84 s of torch.compile and KV setup, 110 s of JIT kernel warmup, **319 s of FlashInfer autotuning** over 21 token buckets, and **199 s capturing 67 CUDA graphs**. vLLM’s recipe, started natively on the previous day’s node, took 24 minutes to engine ready. DeepInfer loaded in 298 s with a warm JIT cache and 748 s with a cold one.

warmctl does the slow part once: it builds the engine, warms it up, and snapshots it whole: the process memory of all 27 processes (CRIU), the CUDA state of every GPU (cuda-checkpoint), compiled kernels, the autotuning results and CUDA graphs, and the 372 GiB of weights as the engine laid them out, 851 GiB in all. `warmctl up` restores it in **88.4 s** from page cache: 43 s for the processes, 22 s for the GPU state, 12 s to reload the weights, then a canary prompt compared with the original engine before the port opens. The node read the snapshot at 155 GiB/s from page cache. Every answer we checked with greedy decoding after the restore was right.

The snapshot is in the warmctl catalog. On an 8×B300 host with NVIDIA driver 610.57.04:

  [
      DeepSeekMIT
      DeepSeek-V4.1-Flash-1017

MoE · Engram · DSpark · Oct 17 update · DEP8

      8×B300·**88 s**·851 GiB·**610**

        restore88 s
        cold start**930 s**

      deepseek-v41-flash-1017
Run
    ](https://warmctl.io/catalog/#deepseek-v41-flash-1017)

```
$ curl -fsSL https://warmctl.io/install.sh | sh
$ warmctl up deepseek-v41-flash-1017
```

  `up` picks the snapshot for the host’s GPUs and driver and serves an OpenAI-compatible API on port 8000:

```
$ curl -s localhost:8000/v1/completions -H 'Content-Type: application/json' -d '{
    "model": "deepseek-v41-flash-1017", "prompt": "The capital of France is", "max_tokens": 8, "temperature": 0}' | jq -r '.choices[0].text'
 Paris.
```

The [profile](https://github.com/TensorRails/warmctl/blob/main/profiles/deepseek-v41-flash-1017-b300-dep8.yaml) holds the vLLM flags, the graph sizes and the warmup grid. The same model is in the catalog for 8×B200 (DEP8, driver 615), 4×B200 (TP4) and 8×B300 with driver 580 (TP8).

## Choosing between vLLM and DeepInfer

The benchmarks say how fast each engine is on one node. Which one to pick depends on what you are building.

    - **An API for your users on one or a few nodes: vLLM.** On this node it made 2–4× the output tokens and 8× the prefill, and it ships as a server with an OpenAI-compatible API. DeepInfer’s release has no server at all.

    - **Any model other than DeepSeek-V4.1-Flash: vLLM.** DeepInfer supports this one model.

    - **Reproducible offline generation, such as RL rollouts, evals or synthetic data: try DeepInfer.** Batch-invariant inference and deterministic sampling are built in and documented for this model, and an offline batch API is what such jobs use anyway. Measure throughput on your own shapes first; on ours it was a half to a quarter of vLLM’s on decode.

    - **A DeepSeek service on rack-scale NVLink (GB200 or GB300 NVL72) with a team to run it: look at both.** DeepInfer is how DeepSeek serves the model at that scale, but you would build the KV cache system, the disaggregation and the load balancing yourself. vLLM covers that ground with disaggregated prefill through KV connectors.

    - **Huawei Ascend 950: DeepInfer**, which supports it next to NVIDIA SM100.

    - **Autoscaling, spot capacity or frequent rollouts: whichever engine you run, the cold start decides.** This one takes 15 minutes per node before the first token, and vLLM’s recipe took 24 on another node. With vLLM, `warmctl up` brings the same engine back in 88 seconds.

For most teams with a node or two, the answer is vLLM. DeepInfer is worth following as the reference for how DeepSeek runs its own model, and worth a try for deterministic offline work. On a single node, with the settings DeepSeek recommends, it is not the faster engine.

### Run it on your GPUs

DeepSeek-V4.1-Flash-1017 is in the warmctl catalog with GLM-5.3-Flash, Ling-3.1-flash, Nemotron 3.5 Super, Qwen and gpt-oss, each a warm vLLM snapshot that serves in seconds.

    [Open in the catalog](https://warmctl.io/catalog/#deepseek-v41-flash-1017)[Install warmctl](https://warmctl.io/docs/install/)[GitHub](https://github.com/TensorRails/warmctl)