Benchmarking Ling-3.1-flash on 8×B200

01 · THE MODELSeven linear layers, then one with a cache

Ling-3.1-flash is inclusionAI’s newest open-weight model, under the MIT license: a mixture of experts with 560B parameters, 25B active per token. Each MoE layer has 512 routed experts and one shared expert, and the router sends every token to 8 of them. The checkpoint is FP8, 572 GB, and includes one multi-token-prediction (MTP) layer for speculative decoding.

The interesting part is attention. The 72 layers repeat one block of eight: seven Kimi Delta Attention (KDA) layers, then one gated Multi-head Latent Attention (MLA) layer.

KDA is linear attention with a delta rule. Each head carries a fixed state matrix, 128×128 at Ling’s head size. A new token overwrites what the state currently associates with its key, a rank-one update, and per-channel gates let old content decay. Reading and updating that state costs the same at token 100 and at token 100,000, and the state never grows.

MLA is full attention over a compressed cache. Per token it stores a 512-wide latent and a 64-wide RoPE key, 576 values shared by all heads. Every earlier token stays addressable, so lookups are exact, but this cache grows with the context.

The part of memory that scales with context therefore lives in 9 layers out of 72. That leads to four predictions, and this post tests each one: decode speed should barely depend on context length; prefill should scale about linearly; a large number of long requests should fit in memory at once; and a single fact deep in the context should still be retrievable, because the MLA layers can look it up.

02 · SETUPOne 8×B200 node

All runs used one node and vLLM, on an engine restored from a warmctl snapshot of the warmed-up model; section 10 explains why.

Random prompts are the worst case for speculative decoding. Section 07 measures MTP on real text separately.

03 · DECODEFlat past 32K

One request, 256 output tokens. Time per output token is 10.0 ms at 1K tokens of context and 13.1 ms at 120K. The increase happens below 32K: from 32K to 120K, a context 3.7× longer, it moves from 13.0 to 13.1 ms, and the inter-token latency stays between 13.0 and 14.0 ms across the whole range.

With full attention in every layer, each decoded token reads the entire KV cache, so the time per token rises with the context. Here only nine layers read a cache, and an MLA layer holds 1,152 bytes per token; the rest of the step is MoE and KDA work, which does not depend on context. We have no clean explanation for the step between 8K and 32K yet; 64K and 120K land on the same value.

04 · PREFILLFaster per token as prompts grow

Time to first token is 0.47 s at 1K and 2.35 s at 120K. Divided by prompt length, that is 2,200 tokens/s at 1K and 51,164 tokens/s at 120K: short prompts are dominated by fixed per-request cost, long ones by compute, and the cost per token keeps falling up to the 128K limit. With quadratic attention the curve would bend the other way, with prefill tokens per second dropping as prompts get longer.

One catch. The first 32K prompt took 5.4 s, the second 1.0 s. The warmup grid of the profile prefills 512, 2,048 and 8,192 tokens; the first longer prompt JIT-compiled kernels for its shape on the spot. A long prefill in the warmup grid moves that compile into the snapshot, so the first long request after a restore does not pay for it. The tables use the second request.

05 · MEMORY5.2 million tokens of cache

The profile gives vLLM 64 GiB of KV cache per GPU. vLLM reports room for 5,247,384 tokens, which it translates into 40 concurrent requests at the full 131,072-token context.

The arithmetic: with a BF16 cache, an MLA layer stores 576 values × 2 bytes = 1,152 bytes per token, and the latent is replicated on every tensor-parallel rank. Nine such layers make 10.1 KiB per token per GPU. vLLM’s figure works out to 12.8 KiB: the hybrid allocator pads layers so that the KDA states and the MLA pages share one page size, and it warns that this can cost up to 11%. Were all 72 layers MLA, the same 64 GiB would hold about 829K tokens, 6× fewer.

The profile admits up to 64 sequences. At the full 128K context the cache caps that at 40; with 32K prompts all 64 fit with room to spare, 40% of the cache.

06 · AGENT TURNSPrefix caching with a recurrent state

Coding agents resend a long, growing history with a few new tokens each turn. With full attention, reusing the cache of a shared prefix is routine. A hybrid model also needs the KDA state at the exact end of the prefix, which vLLM keeps when mamba-cache-mode is align. We sent six requests that share a 60K-token prefix, each with 1K new tokens. The median time to first token is 0.66 s, against 1.47 s for a 64K prompt the engine has not seen; the first of the six computes the whole prefix and is slower.

07 · SPECULATIVE DECODINGMTP on real text

Ling has one MTP layer; the profile runs it three times per step to draft three tokens, and the main model verifies them in a single forward pass. How much that buys depends on how predictable the text is. We asked for long answers in prose and in code, one request at a time, and read the acceptance from vLLM’s counters.

Prose accepts 50% of the drafts, 2.5 tokens per step, and decodes at 184 tokens/s; code accepts 68% at 225 tokens/s, time to first token included. The random-prompt runs in this post decode at 114 tokens/s per request. Benchmarks with random prompts, ours included, understate what an MTP model does on real text.

08 · RECALLDoes the state forget?

A fixed-size state cannot hold everything, so recall is the first question about linear attention. We hid a vault code at 10%, 50% and 90% depth of synthetic documents of 34K and 116K tokens, thousands of numbered filler sentences, and asked for the code with thinking off. Ling returned it 6 of 6 times; at 116K the answer takes about 4 s, nearly all of it prefill.

This is what the nine MLA layers are for: exact access to any earlier token, for about 10 KiB of cache per token. One planted fact per document is the easy end of long-context retrieval; multi-hop questions and long reasoning chains are a separate test.

09 · UNDER LOADThroughput for reference

With 1K prompts and 512-token answers, output throughput rises from 114 tokens/s for one request to 1,276 at 64, the profile’s limit on concurrent sequences, with time to first token under 1 s. Sixty-four parallel 2K-token generations reach 2,496 output tokens/s. With 32K prompts the node processes 47,189 tokens/s at 64 concurrent requests, prefill and decode together, while each request still streams a token every 40 ms between prefill chunks.

10 · STARTUPWhere a 14-minute cold start goes

vLLM’s own log of the build shows where 854 s go before the first token. Loading 572 GB of weights takes 77 s. Compilation takes most of the rest: torch.compile 122 s, a Triton JIT warmup of 156 kernels 57 s, FlashInfer autotuning 99 s, and the largest item, 397 s of DeepGEMM compiling 2,883 FP8 MoE kernels for every shape the engine will run. Every restart, scale-out and rollout of every node pays it again.

warmctl runs that once, then snapshots the warm engine: the process memory of all eight ranks (CRIU), the CUDA state of every GPU (cuda-checkpoint), the compiled kernels and CUDA graphs, and the weights as the engine laid them out, 647 GiB in all. warmctl up restores it in 47.2 s from page cache: 16 s to bring the processes back, 11 s for GPU state, 18 s to reload 546 GiB of weights, then a canary token compared bit for bit with the original engine before the port opens. With a cold page cache on this VM’s network disk the same restore took 382 s, bound by reading 643 GiB at 2 GB/s; local NVMe or a fast shared filesystem gets close to the warm number, as our GLM-5.3-Flash benchmark showed with a share read at 7 GiB/s per node.

The snapshot is in the warmctl catalog. On an 8×B200 host with an NVIDIA 580 driver:

$ curl -fsSL https://warmctl.io/install.sh | sh
$ warmctl up ling31-flash

up pulls the snapshot for the host’s GPUs and driver and serves an OpenAI-compatible API on port 8000, with reasoning and tool calls enabled:

$ curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
    "model": "ling31-flash", "temperature": 0, "chat_template_kwargs": {"enable_thinking": false},
    "messages": [{"role": "user", "content": "What is the capital of France? One word."}]}' | jq -r '.choices[0].message.content'
Paris

The profile in the repository holds the vLLM flags, the warmup grid and the canary policy.