---
title: "Benchmarking Ling-3.1-flash on 8×B200"
description: "Ling-3.1-flash, a 560B MoE with linear attention, on 8×B200 with vLLM: decode barely slows from 1K to 120K tokens, 5.2M tokens of KV cache, MTP on real text, recall at 100K and a 47 s warm start."
author: Kirill Bespalov
date: 2026-10-17
url: https://warmctl.io/blog/benchmarking-ling-3-1-flash/
---

# Benchmarking Ling-3.1-flash on 8×B200

> Ling-3.1-flash keeps a growing KV cache in only 9 of its 72 layers. We measured what that changes on one 8×B200 node: decode from 1K to 120K tokens, cache capacity, prefix reuse, MTP drafts and recall, and where a 14-minute vLLM cold start goes.

**Key results**

- Ling-3.1-flash (560B MoE, 25B active, FP8; 63 KDA linear-attention and 9 gated MLA layers) on 8×B200 with vLLM, TP8 with expert parallelism and MTP: one request decodes at 10.0 ms per token with a 1K-token context and 13.1 ms with 120K; from 32K to 120K the time per token barely moves (13.0 to 13.1 ms).
- Time to first token for one request: 0.47 s at 1K tokens, 1.0 s at 32K, 1.5 s at 64K and 2.3 s at 120K, about 51,000 tokens/s of prefill.
- With 64 GiB of KV cache per GPU, vLLM fits 5,247,384 tokens of Ling-3.1-flash: 40 requests at the full 131,072-token context.
- A 60K-token prefix shared across requests brings time to first token down to 0.66 s, against 1.47 s for an unseen 64K-token prompt.
- 64 concurrent requests with 1K-token prompts produce 1,276 output tok/s; 64 concurrent 2K-token generations produce 2,496 output tok/s.
- MTP with 3 draft tokens: 50% of drafts accepted on prose (184 tok/s for one request) and 68% on code (225 tok/s).
- A secret code hidden at 10%, 50% and 90% depth of 34K- and 116K-token documents was recalled in 6 of 6 tries.
- A vLLM cold start takes 854 s to engine ready on 8×B200; warmctl up restores the 647 GiB snapshot to serving in 47 s from page cache.

[
      inclusionAIMIT
      Ling-3.1-flash

560B · 25B active · hybrid MoE · FP8 · MTP

      8×B200·**47 s**·647 GiB·**580**

        restore47 s
        cold start**854 s**

      ling31-flash
Run
    ](https://warmctl.io/catalog/#ling31-flash)

- 13.1 ms per token at 120K context, 10.0 ms at 1K
- 5.2M tokens of KV cache: 40 requests at 128K
- 6/6 hidden codes found, up to 116K
- 47 s warmctl up, against an 854 s cold start

## Seven linear layers, then one with a cache

  **Ling-3.1-flash** is inclusionAI’s newest open-weight model, under the MIT license: a mixture of experts with **560B parameters, 25B active per token**. Each MoE layer has 512 routed experts and one shared expert, and the router sends every token to 8 of them. The checkpoint is FP8, 572 GB, and includes one multi-token-prediction (MTP) layer for speculative decoding.

The interesting part is attention. The 72 layers repeat one block of eight: seven **Kimi Delta Attention** (KDA) layers, then one **gated Multi-head Latent Attention** (MLA) layer.

**KDA** is linear attention with a delta rule. Each head carries a fixed state matrix, 128×128 at Ling’s head size. A new token overwrites what the state currently associates with its key, a rank-one update, and per-channel gates let old content decay. Reading and updating that state costs the same at token 100 and at token 100,000, and the state never grows.

**MLA** is full attention over a compressed cache. Per token it stores a 512-wide latent and a 64-wide RoPE key, 576 values shared by all heads. Every earlier token stays addressable, so lookups are exact, but this cache grows with the context.

The part of memory that scales with context therefore lives in 9 layers out of 72. That leads to four predictions, and this post tests each one: decode speed should barely depend on context length; prefill should scale about linearly; a large number of long requests should fit in memory at once; and a single fact deep in the context should still be retrievable, because the MLA layers can look it up.

## One 8×B200 node

All runs used one node and vLLM, on an engine restored from a warmctl snapshot of the warmed-up model; section 10 explains why.

| Setup |  |
|---|---|
| GPUs | 8× NVIDIA B200, 179 GiB each, NVLink between all eight (NV18) |
| Host | 2× Intel Xeon Platinum 8580 (160 vCPU), 1.7 TiB RAM, network SSD |
| OS and driver | Ubuntu 24.04.5, kernel 6.11.0-1016-nvidia, NVIDIA driver 580.173.02 |
| Engine | vLLM 0.32.0rc1 nightly (3f1bcfd8), with the BailingMoeV3 tensor-parallel fix (vllm#61642) |
| Model | inclusionAI/Ling-3.1-flash at a4ccd6b4, FP8 weights, BF16 KV cache |
| Serving | TP8 with expert parallelism; FP8 MoE on DeepGEMM (FlashInfer’s TRT-LLM router takes at most 32 experts per group, Ling has 64); MTP with 3 draft tokens; 128K context; 64 GiB of KV cache per GPU; up to 64 sequences; 8,192-token prefill chunks |
| warmctl | v2.0.0-beta.6, snapshot ling31-flash:8xb200-180gb-580.173.02, 647 GiB |
| Load | vllm bench serve: random-token prompts, fixed output length (--ignore-eos), medians per run |

Random prompts are the worst case for speculative decoding. Section 07 measures MTP on real text separately.

## Flat past 32K

One request, 256 output tokens. Time per output token is **10.0 ms** at 1K tokens of context and **13.1 ms** at 120K. The increase happens below 32K: from 32K to 120K, a context 3.7× longer, it moves from 13.0 to 13.1 ms, and the inter-token latency stays between 13.0 and 14.0 ms across the whole range.

With full attention in every layer, each decoded token reads the entire KV cache, so the time per token rises with the context. Here only nine layers read a cache, and an MLA layer holds 1,152 bytes per token; the rest of the step is MoE and KDA work, which does not depend on context. We have no clean explanation for the step between 8K and 32K yet; 64K and 120K land on the same value.

## Faster per token as prompts grow

Time to first token is **0.47 s** at 1K and **2.35 s** at 120K. Divided by prompt length, that is 2,200 tokens/s at 1K and **51,164 tokens/s** at 120K: short prompts are dominated by fixed per-request cost, long ones by compute, and the cost per token keeps falling up to the 128K limit. With quadratic attention the curve would bend the other way, with prefill tokens per second dropping as prompts get longer.

One catch. The first 32K prompt took **5.4 s**, the second 1.0 s. The warmup grid of the profile prefills 512, 2,048 and 8,192 tokens; the first longer prompt JIT-compiled kernels for its shape on the spot. A long prefill in the warmup grid moves that compile into the snapshot, so the first long request after a restore does not pay for it. The tables use the second request.

| Prompt | TTFT | Prefill | TPOT | ITL | tok/s per user |
|---|---|---|---|---|---|
| 1K | 0.47 s | 2,200 tok/s | 10.0 ms | 13.0 ms | 100 |
| 8K | 0.52 s | 15,783 tok/s | 11.0 ms | 13.1 ms | 91 |
| 32K | 1.03 s | 31,845 tok/s | 13.0 ms | 13.3 ms | 77 |
| 64K | 1.47 s | 44,680 tok/s | 13.1 ms | 13.6 ms | 76 |
| 120K | 2.35 s | 51,164 tok/s | 13.1 ms | 14.0 ms | 76 |

## 5.2 million tokens of cache

The profile gives vLLM 64 GiB of KV cache per GPU. vLLM reports room for **5,247,384 tokens**, which it translates into **40 concurrent requests** at the full 131,072-token context.

The arithmetic: with a BF16 cache, an MLA layer stores 576 values × 2 bytes = 1,152 bytes per token, and the latent is replicated on every tensor-parallel rank. Nine such layers make **10.1 KiB per token per GPU**. vLLM’s figure works out to 12.8 KiB: the hybrid allocator pads layers so that the KDA states and the MLA pages share one page size, and it warns that this can cost up to 11%. Were all 72 layers MLA, the same 64 GiB would hold about 829K tokens, 6× fewer.

The profile admits up to 64 sequences. At the full 128K context the cache caps that at 40; with 32K prompts all 64 fit with room to spare, 40% of the cache.

## Prefix caching with a recurrent state

Coding agents resend a long, growing history with a few new tokens each turn. With full attention, reusing the cache of a shared prefix is routine. A hybrid model also needs the KDA state at the exact end of the prefix, which vLLM keeps when `mamba-cache-mode` is `align`. We sent six requests that share a 60K-token prefix, each with 1K new tokens. The median time to first token is **0.66 s**, against 1.47 s for a 64K prompt the engine has not seen; the first of the six computes the whole prefix and is slower.

## MTP on real text

Ling has one MTP layer; the profile runs it three times per step to draft three tokens, and the main model verifies them in a single forward pass. How much that buys depends on how predictable the text is. We asked for long answers in prose and in code, one request at a time, and read the acceptance from vLLM’s counters.

| Text | Drafts accepted | Tokens per step | Decode tok/s |
|---|---|---|---|
| prose | 50% | 2.51 | 184 |
| code | 68% | 3.05 | 225 |

Prose accepts 50% of the drafts, 2.5 tokens per step, and decodes at **184 tokens/s**; code accepts 68% at **225 tokens/s**, time to first token included. The random-prompt runs in this post decode at 114 tokens/s per request. Benchmarks with random prompts, ours included, understate what an MTP model does on real text.

## Does the state forget?

A fixed-size state cannot hold everything, so recall is the first question about linear attention. We hid a vault code at 10%, 50% and 90% depth of synthetic documents of 34K and 116K tokens, thousands of numbered filler sentences, and asked for the code with thinking off. Ling returned it **6 of 6 times**; at 116K the answer takes about 4 s, nearly all of it prefill.

| Document | Code at depth | Answer | Time |
|---|---|---|---|
| 34K | 10% | found | 4.4 s |
| 34K | 50% | found | 1.6 s |
| 34K | 90% | found | 1.5 s |
| 116K | 10% | found | 4.0 s |
| 116K | 50% | found | 4.0 s |
| 116K | 90% | found | 4.0 s |

This is what the nine MLA layers are for: exact access to any earlier token, for about 10 KiB of cache per token. One planted fact per document is the easy end of long-context retrieval; multi-hop questions and long reasoning chains are a separate test.

## Throughput for reference

With 1K prompts and 512-token answers, output throughput rises from 114 tokens/s for one request to **1,276** at 64, the profile’s limit on concurrent sequences, with time to first token under 1 s. Sixty-four parallel 2K-token generations reach **2,496 output tokens/s**. With 32K prompts the node processes **47,189 tokens/s** at 64 concurrent requests, prefill and decode together, while each request still streams a token every 40 ms between prefill chunks.

| Tokens in → out | Concurrent | Output tok/s | Total tok/s | TTFT p50 | ITL p50 |
|---|---|---|---|---|---|
| 1k → 512 | 1 | 95 | 285 | 0.50 s | 12.9 ms |
| 1k → 512 | 8 | 373 | 1,119 | 0.56 s | 18.9 ms |
| 1k → 512 | 32 | 799 | 2,398 | 0.54 s | 25.0 ms |
| 1k → 512 | 64 | 1,276 | 3,829 | 0.98 s | 28.9 ms |
| 1k → 2k | 64 | 2,496 | 3,744 | 1.02 s | 30.8 ms |
| 32k → 256 | 8 | 128 | 16,552 | 3.56 s | 20.0 ms |
| 32k → 256 | 32 | 308 | 39,743 | 3.88 s | 29.6 ms |
| 32k → 256 | 64 | 366 | 47,189 | 4.52 s | 39.9 ms |

## Where a 14-minute cold start goes

vLLM’s own log of the build shows where 854 s go before the first token. Loading 572 GB of weights takes 77 s. Compilation takes most of the rest: torch.compile 122 s, a Triton JIT warmup of 156 kernels 57 s, FlashInfer autotuning 99 s, and the largest item, **397 s of DeepGEMM compiling 2,883 FP8 MoE kernels** for every shape the engine will run. Every restart, scale-out and rollout of every node pays it again.

warmctl runs that once, then snapshots the warm engine: the process memory of all eight ranks (CRIU), the CUDA state of every GPU (cuda-checkpoint), the compiled kernels and CUDA graphs, and the weights as the engine laid them out, 647 GiB in all. `warmctl up` restores it in **47.2 s** from page cache: 16 s to bring the processes back, 11 s for GPU state, 18 s to reload 546 GiB of weights, then a canary token compared bit for bit with the original engine before the port opens. With a cold page cache on this VM’s network disk the same restore took 382 s, bound by reading 643 GiB at 2 GB/s; local NVMe or a fast shared filesystem gets close to the warm number, as our [GLM-5.3-Flash benchmark](https://warmctl.io/blog/boosting-dedicated-inference/) showed with a share read at 7 GiB/s per node.

The snapshot is in the warmctl catalog. On an 8×B200 host with an NVIDIA 580 driver:

```
$ curl -fsSL https://warmctl.io/install.sh | sh
$ warmctl up ling31-flash
```

`up` pulls the snapshot for the host’s GPUs and driver and serves an OpenAI-compatible API on port 8000, with reasoning and tool calls enabled:

```
$ curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
    "model": "ling31-flash", "temperature": 0, "chat_template_kwargs": {"enable_thinking": false},
    "messages": [{"role": "user", "content": "What is the capital of France? One word."}]}' | jq -r '.choices[0].message.content'
Paris
```

The [profile](https://github.com/TensorRails/warmctl/blob/main/profiles/ling31-flash-b200-tp8.yaml) in the repository holds the vLLM flags, the warmup grid and the canary policy.

### Run it on your GPUs

Ling-3.1-flash is in the warmctl catalog with GLM-5.3-Flash, DeepSeek-V4.1-Flash, Nemotron 3.5 Super, Qwen and gpt-oss, each a warm vLLM snapshot that serves in seconds.

    [Open in the catalog](https://warmctl.io/catalog/#ling31-flash)[Install warmctl](https://warmctl.io/docs/install/)[GitHub](https://github.com/TensorRails/warmctl)