Should you split prefill and decode? The math of inference cost

01 · THE QUESTIONA claim first, then the question: is it cheaper?

A language model’s work on a request splits into two phases. First the model reads the whole prompt (prefill), then it writes the answer one token at a time (decode). A common piece of advice is to give the phases different GPUs, so that one user’s long prompt does not slow down everyone else’s generation. Latency gets smoother that way. This is what is called disaggregation, or P/D.

For latency the advice works. But a provider sells tokens and pays for GPU-hours. It tries to reuse as much of the KV cache as it can: whatever has already been computed is not computed again. Agent sessions with a long context and short additions push the reuse share to 90–99%. Then there is little prefill work left, and a separate prefill pool risks sitting idle. In a shared pool it is enough to scale one resource: the one that is short.

In this article we build a model that answers the question quantitatively. It is the same problem as classic cluster planning, where servers are packed by CPU and memory. Jobs have different resource profiles, machines have a capacity for each resource, and the idle part of the hardware costs money. The terms we use (prefill, decode, KV cache, cache hit, HBM, SLO) are collected in a glossary at the end of the article.

02 · TWO PHASESWhat each phase consumes

Why think about this: everything that follows rests on the phases eating different things. Prefill is large matrix multiplications, where every byte of weights read from memory is used for thousands of tokens at once. Decode reads the same weights at every step for the sake of one token in each sequence of the batch. That is why the same GPU processes an order of magnitude more tokens per second in prefill than it produces in decode.

These two measurements give the main constant of the article. Call it κ (kappa): how many times faster a GPU does prefill than decode. For us κ ≈ 82K / 6.9K ≈ 12. In other words, one output token costs about as much GPU time as 12 uncached input tokens.

What it means. The phases do not compete for the same resource. Prefill loads the arithmetic and leaves memory bandwidth idle. Decode loads memory bandwidth and capacity and leaves the arithmetic idle. Such workloads are worth packing together.

03 · THE BOTTLENECKResources as vectors: the bottleneck law

Why a model is needed: without one, the “together or apart” argument comes down to comparing charts from different papers. A model shows where a gain comes from and lets you compute it on your own numbers.

Describe each class of requests (RAG, reasoning, agent) by a resource vector: how many seconds of one GPU’s work it takes on each resource. There are four resources:

• Compute: prefill of the uncached tokens; their count divided by one GPU’s prefill speed.
• HBM bandwidth: decode steps; the number of output tokens divided by one GPU’s decode speed.
• HBM capacity × time: the KV cache that sits in memory while the request lives, including an agent’s pauses for tool calls. This is Little’s law: the average occupied capacity equals request rate × KV size × lifetime.
• Network: KV bytes transferred between pools. For modern models with a compact KV this is almost zero (section 11).

Every GPU provides exactly one “GPU-second” of each resource per second. Then the number of GPUs the stream needs, under ideal packing (no queues, no bursts), is:

Here j is a request class, λ_j is how many such requests arrive per second, a_j,r is how much of resource r one request needs, and C_r is how much of resource r one GPU provides. The maximum is taken over resources: the fleet is set by the resource that runs short. The other resources are bought but partly idle.

This is the same law by which servers are packed in a data center. If some jobs need a lot of CPU and little memory and others the reverse, it pays to put them on the same machines. Separate “CPU servers” and “memory servers” would leave half the hardware unused. Schedulers like Borg and the whole theory of multidimensional packing are built on this observation.

What it means. “Should the phases be split” is a special case of “should jobs with different resource profiles be packed together”. That question has a known answer, and the next section derives it.

04 · THE THEOREMWhy splitting is not cheaper on one GPU type

Write the prefill phase’s total demand for each resource as P_r and the decode phase’s as D_r. A monolith covers the sum of the demands; separate pools on the same GPUs cover each phase on its own. The maximum of a sum is at most the sum of the maxima, so:

Prefill’s bottleneck is compute, decode’s is memory. So on identical hardware separate pools always leave more resources idle. In the best case the shared pool costs half as much: that happens when both phases load the GPU equally.

The “under ideal packing” caveat matters. The theorem assumes that prefill’s arithmetic and decode’s memory reads run on the GPU at the same time without getting in each other’s way. In reality they do, and that is the next section. It also shows that with strong enough interference the inequality flips.

What it means. Disaggregation by itself does not save GPUs. Where it does save, the source of the saving is something else, and it has to be named explicitly.

05 · INTERFERENCEθ: how much the phases really get in each other’s way

Why another letter: it turns the word “interference” into a number that can be measured. When the engine puts both decode and a chunk of someone else’s prefill into one step (this is called chunked prefill), the step takes longer than pure decode. Describe the length of such a step as:

T_comp is how long the step would take if it were bound by arithmetic alone, T_mem by memory reads alone. The coefficient θ (theta) is non-negative. At θ = 0 one phase’s work hides entirely in the other’s idle time: prefill rides along with decode for free. At θ = 1 the phases run strictly in turn, as if the GPU split its time between them. θ can also exceed one: that happens when the engine synchronizes many GPUs at every step, and a chunk of prefill on one of them holds up all the others (section 07). Then a shared step takes longer than the two phases in turn.

Now normalize the load. U is how many GPUs would be needed if they did nothing but the prefill of the whole stream. V is the same for decode. Then:

What is θ in practice? The best data point we know is the paper “Not All Prefills Are Equal” (PPD, 2026): Llama-3.1-8B on one H100, 1,024-token prefill. At a batch of 200, a full prefill of a new prompt increased the decode step time (TPOT) by 48%, while a short addition to an already cached context increased it by only 2%. So θ depends on the kind of prefill: for an agent’s short additions it is close to zero, for long cold prompts it is large.

What it means. Long cold prompts really do interfere with decode, and they are worth moving away. Short additions to a warm context are better left on the decode GPU next to their cache. That is how PPD’s hybrid approach and Together AI’s cold/warm prefill split are built.

06 · SPECIALIZATIONs: the first reason to pay for a split

At s = 1 the formulas above always say “together”. Separate pools can win only when each of them works more efficiently than it would in the shared pool. Call that gain s: how many times faster a specialized pool is on its phase. Where it comes from:

• A different parallelism layout. Prefill runs better in small groups of GPUs, decode of a large MoE in large ones (section 07).
• An unmixed batch. A decode pool can hold a larger batch and capture its steps into CUDA graphs without adapting to prefill.
• Other hardware. Decode can go to GPUs with more memory per dollar, prefill to GPUs with more arithmetic. That is exactly where Splitwise’s cost gain comes from.

The last letter is ρ (rho), the phase balance: the smaller of U and V divided by the larger. ρ = 1 means the phases need equal numbers of GPUs, ρ → 0 means one phase has almost disappeared. Comparing U/s + V/s with the monolith’s formula gives the condition (for simplicity we assume both pools gain equally; in general the split scheme costs U/s_P + V/s_D, and s_P may turn out to be below one):

What the map shows:

• A balanced stream and a small θ: co-location wins big. At ρ = 1 and θ = 0 the split needs a specialization gain above 2×, which is rare.
• An extreme mix: the architecture barely matters. As ρ → 0 all the curves converge to s = 1. Cost is set by the dominant phase, and its settings decide it, not the pool layout.
• A large θ: the split is almost free. If the phases run in turn anyway, the shared pool gains nothing from packing.

What it means. The “keep them apart so they don’t interfere” argument is about latency. For cost the question is different: does specialization give the pool a gain s that exceeds the loss from worse packing. The map shows exactly how much has to be gained.

07 · LARGE MoEWhy DeepSeek-class models get split anyway

Why a separate section: large model providers like DeepSeek run their models precisely with disaggregation, and the model has to explain that. Two terms first.

MoE (mixture of experts) is a model in which every layer consists of hundreds of “experts”, and every token passes through only a few of them. Expert parallelism (EP) is a layout in which the experts are spread across GPUs, and at every layer tokens are sent to wherever their experts live. Wide-EP is the same across dozens of GPUs: each holds fewer experts, so at a decode step it reads fewer weights.

• Wide-EP gives decode a large gain s. In vLLM’s October 23 post on DeepSeek-V4.1-Flash on GB200, the MoE part of a decode step fell from 13.4 to 6.0 ms when going from 4 to 32 GPUs per group. The rest of the step is about 10 ms, so the whole step got about 1.46× faster at the same batch per GPU.
• Prefill wants small groups. In the same post, prefill in a group of 4 GPUs delivered per GPU only 83% of a group of 2. DeepSeek’s own engine, DeepInfer, in its published measurements keeps prefill on groups of 4 GPUs and decode on groups of 64. A shared pool has to pick one group size and loses on one of the phases.
• Synchronization pushes θ above one. The same post says outright that wide-EP decode synchronizes all GPUs at every layer, and one slow GPU stalls the whole instance. A chunk of prefill on one GPU holds up all 32. It also shows that the loss from uneven load grows with group size: 5.9% with 2 GPUs versus 13.6% with 4.

So for the DeepSeek class on a rack, θ_eff > 1 and s is well above one. Both reasons push toward splitting. But that is a gain from specialization and synchronization, not a consequence of the slogan “the phases interfere with each other”.

On a single node of 8 GPUs the choice is different, although the interference is there too. Our measurement on 8×B300 showed that vLLM in one shared DEP8 configuration delivers 658K tok/s of prefill and 55K tok/s of decode, and on an 8K → 1K mix the same node produces 29.5K tok/s of decode while spending 36% of its prefill capacity. If decode at 8K context is no slower than at 1K, the formula of section 05 turns these numbers into θ ≈ 1.3: the eight DEP8 ranks synchronize just like thirty-two do. But splitting such a node into 2 GPUs for prefill and 6 for decode would mean losing wide-EP on both sides, that is, getting s < 1 on decode. Here the split loses not because there is no interference, but because specialization is impossible on 8 GPUs.

What it means. Disaggregation is a tool of scale. It pays off where a decode instance spans dozens of GPUs. For a single node, a dense model or a small MoE, the default choice is a shared pool with chunked prefill.

08 · EXAMPLEThree workload types on the same numbers

Why an example: to see how the formulas behave at realistic request sizes. The GPU speeds are taken from our measurement (82K tok/s prefill and 6.9K tok/s decode per GPU, KV 3,309 bytes per token, 160 GiB for KV per GPU). The workloads themselves are an illustration:

• RAG: 100 requests per second, 12K uncached input tokens, 300 output.
• Reasoning: 10 requests per second, 1K input, 8K of reasoning and answer on output.
• Agent: 100 turns per second, a 40K context, each turn adds 4K new tokens (hit 90%) and writes 400. Between turns there is a 30 s pause for a tool call, and the KV sits in HBM all that time.

• An agent at hit 90% is almost balanced (ρ ≈ 0.84). Every turn is 4K uncached tokens per 400 output, that is 10:1, and κ ≈ 12. Prefill is no trifle here, and co-location saves up to a third of the GPUs.
• At hit 99% prefill disappears (ρ ≈ 0.08), and with it the difference between the schemes. What remains is decode, bound by HBM bandwidth and by the capacity taken by KV during the pauses.
• Reasoning is pure decode. The pool layout does not matter; batch and step speed do.
• The mix is the best case for co-location. RAG brings compute, reasoning and agents bring memory load, and together they fill the GPU on both resources.

The balance formula is simple:

Compute it on your own numbers

The calculator uses the formulas above with our GPU speeds (82K and 6.9K tok/s, a 29 ms decode step, as in the 1K → 1K measurement) and ideal packing: no queues, no headroom, no minimum instance size. KV options: DeepSeek-V4.1-Flash (FP8), GLM-5.3-Flash (BF16), a hypothetical midpoint, and a dense 70B model with GQA (BF16). The prefill and decode speeds do not change with them; that is a deliberate simplification, so that the effect of KV size alone is visible.

What it means. Look not at the cache hit rate but at the ratio of uncached input to output. Compare it with κ on your hardware, and you will know which corner of the section 06 map your stream is in.

09 · PRICESShadow prices: what a token of each phase really costs

Why: a provider charges different prices for input, cached input and output. The model shows what those tokens cost the provider itself.

Choosing a fleet is a linear programming problem. There is a set of possible configurations (hardware × role × parallelism × batching policy), each with its own hourly price and its own prefill and decode throughput. Demand has to be covered at the minimum total. Such a problem has two useful properties:

• The optimum has no more roles than demand has dimensions. This is the standard fact about basic solutions of linear programs; it holds exactly as long as the problem has no other constraints, such as a per-class SLO or hardware quotas, each of which can add a role. Two phases: no more than two roles. Cold prefill, warm prefill and decode: no more than three, as at Together AI. A zoo of five pool types is almost certainly suboptimal.
• The dual variables are the unit cost of demand. They show what one more prefill token or one more decode token costs. The key observation: if the shared pool is bound by decode, the marginal cost of a prefill token equals θ divided by the prefill speed. At a small θ it is close to zero. In the split scheme the same token costs the full price of GPU time.

Compare with the price list. DeepSeek’s public prices for V4.1-Flash: $0.30 per 1M uncached input tokens, $0.006 for cached ones, $1.20 for output. Taking a notional $9.50 per GPU-hour and our speeds at full load, compute costs about $0.03 per 1M prefill tokens and about $0.38 per 1M decode tokens. An illustration: no headroom for peaks, SLOs or idle time.

• On the price list output is 4 times dearer than input; in GPU time about 12 times. So the provider’s margin cushion sits mostly in input tokens. The more precisely the fleet is bound by decode, the cheaper input is for it.
• A cached token at $0.006 per 1M cannot be recomputed; it has to be stored. The price is about 5 times below the cost of recomputation. But it is enough for about 2 minutes of holding that token in HBM at 3.3 KB per token.

What it means. The price of cached input is a price of storage, not of compute. The economics of agent inference is set by where and how long KV sits, not by whether the pools are split.

10 · HEADROOM AND DRIFTWhat real, uneven traffic takes away

Why: so far we have assumed ideal packing under a steady stream. A real stream fluctuates, and the fluctuations need headroom. Here the shared pool has two more advantages.

• Headroom grows like a square root, and splitting it is a loss. By the classic square-root staffing rule, a pool with an average load of R GPUs needs about R + β√R GPUs, where β sets the acceptable probability of queueing. Two pools pay β(√R_P + √R_D) instead of β√(R_P+R_D). An example (illustration): 9 and 36 GPUs at β = 2 need an extra ≈ 4.6 GPUs, that is 10% of the fleet.
• The cache hit rate drifts, and the phase balance with it. By day, agents with a warm cache dominate; in the evening, batch processing of cold documents arrives. A rigid split has to keep each pool at its own peak size, that is, pay the sum of the maxima. A flexible fleet pays the maximum of the sum.
• Granularity. If a decode instance takes 32–64 GPUs and a prefill instance 2–4, a small provider simply cannot split the phases without enormous rounding up.

Hence the practical option: flexible roles, nodes with one model that switch between prefill and decode when the mix changes. The price of a switch is the time it takes to restart the engine with a different profile. For DeepSeek-V4.1-Flash on 8×B300 a vLLM cold start took us 930 s, and a restore from a warm warmctl snapshot 88 s. That is not an argument for or against P/D, but the condition under which a flexible fleet works at all.

What it means. Even where a split pays off on average, a fixed P:D ratio is a bad idea. A fleet that moves the boundary between the pools as the cache hit rate moves is the better deal.

11 · KVKeep the cache or recompute it

Why: at a high cache hit rate the main expense is storing KV between turns. A rule is needed for where and how long to keep it. Two simple thresholds.

Load or recompute. Loading KV over the network is faster than recomputing it if the link is faster than B* = (KV bytes per token) × (prefill speed). For DeepSeek-V4.1-Flash that is ≈ 0.25 GB/s per GPU, for GLM-5.3-Flash ≈ 1 GB/s. For a dense 70B model with GQA at the same prefill speed it would be ≈ 27 GB/s; at the 30–50K tok/s per GPU realistic for it, 10–16 GB/s. For models with a compact KV any server network beats recomputation, and moving KV between pools costs next to nothing: an 8K-token prompt is 27 MB.

How long to keep it. A session’s KV is worth keeping in memory tier i for a pause of length T if the expected saving on recomputation exceeds the cost of storage:

Here p_return is the probability that the session comes back, C_recompute is the cost of a repeat prefill, and c_i is the price of storing a byte for a second in tier i (HBM, host DRAM, local NVMe, remote storage). For HBM the formula simplifies: T* equals the time in which the GPU, prefilling, would fill its whole KV budget. For DeepSeek-V4.1-Flash that is about 10 minutes at p = 0.9. For a dense model with a hundred times larger KV it is seconds, which is why DRAM and NVMe are needed there.

A subtlety: c_i is a shadow price. If HBM is not full, storing in it is free, and the cache is worth keeping to the limit. The price appears only when HBM capacity becomes the bottleneck. For agents with long pauses that happens first.

What it means. New models with a KV of a few kilobytes per token shift the economics from recomputation to storage. Memory tiers and routing that brings a session back to its cache give more than the choice between a monolith and P/D.

12 · LITERATUREWhat the papers actually showed

Why: most of the headline multipliers were obtained against a specific, often old, baseline. They cannot be added or multiplied; each paper has its own model, hardware, lengths and metric.

• DistServe (OSDI 2024): up to 7.4× in sustainable load, or up to 12.6× in SLO tightness. Compared with 2023 vLLM on OPT up to 175B, lengths up to 2048 tokens. The metric is goodput under an SLO, not cost.
• Splitwise (ISCA 2024): 1.4× throughput at −20% cost, or 2.35× at the same budget. This is a simulation on GPU profiles, and the cost gain comes largely from different hardware for the phases. In our language that is s > 1 through hardware.
• Sarathi-Serve (OSDI 2024): 2.6–5.6× in served load for a co-located scheme with chunked prefill. A strong argument for co-location, but again against 2023 vLLM.
• Mooncake (FAST 2025): +115% on A800 and +107% on H800 in Kimi’s production. This is a KV-centric architecture: cache in DRAM and on SSD, a cache-aware scheduler and early rejection of requests. The gain cannot be attributed to disaggregation alone.
• NVIDIA “Beyond the Buzz” (MLSys 2026): the benefit of P/D depends on model size, prefill share, SLO and the capacity ratio of the stages. The authors recommend an adaptive role ratio. Most of the results come from a closed simulator.
• PPD “Not All Prefills Are Equal” (2026): +48% to TPOT from a full prefill and +2% from an addition to a cached context. A direct measurement of two kinds of θ, though on one H100 and 8–30B models.
• Together AI, cache-aware disaggregation (2026): +35–40% in sustainable QPS from separating cold prefill, warm prefill and decode. The baseline is an already disaggregated 2P1D scheme, not a shared pool, and this is an engineering post without a full cost model.

What it means. The literature gives no universal winner. It gives mechanisms: specialization, heterogeneous hardware, cache-aware routing and the cold/warm split. Each is a separate term in the model, and they have to be computed on your own workload.

13 · DECISIONWhen to split

• Split a large MoE if the decode instance spans 16–64 GPUs with wide-EP. Specialization and synchronization give s > 1 and θ_eff > 1.
• Split if there is markedly cheaper hardware for one of the phases, for example decode on GPUs with more memory per dollar. That is s through hardware.
• Move long cold prompts away if ITL is under a tight SLO and θ for them is large. Leave short additions to a warm context on the decode GPU.
• Do not split on a single node, on dense and small MoE models, and with a balanced workload mix. A shared pool with chunked prefill, prefix cache and cache-aware routing is cheaper.
• Do not expect miracles at a high cache hit rate. If uncached input is less than κ/5 of output, the pool layout barely affects cost. Invest in KV tiers, KV lifetime and routing instead.
• Do not fix P:D. If the mix drifts, you need a fleet with flexible roles and fast switching.

14 · NEXTWhat we want to measure

The model’s main unknown is θ for a modern MoE on a real engine. It can be estimated on one node without any disaggregation. Our published 8K → 1K run on 8×B300 produced 29.5K output tokens per second. Meanwhile the prefill of the incoming 8K prompts took about 36% of the node’s prefill capacity. If pure decode at the same context is measured separately (with prefix cache, no new prefill), the formula of section 05 yields θ. For example, pure decode at 36K tok/s would give θ ≈ 0.5, and 46K would give θ ≈ 1. And if decode at 8K context is as fast as the 55K measured at 1K context, θ ≈ 1.3: on DEP8 the phases interfere with each other more than if they ran in turn. That is the first thing worth measuring.

• θ for cold prefill and for additions to a warm context: repeat the PPD experiment on a DeepSeek-class model.
• κ at different context lengths: how much a decode step slows down at 32K–200K.
• The actual KV volume transferred per request against the B* threshold.

It is a small experiment; budget and timing to be decided later. When the numbers arrive, they go into the calculator above in place of the illustrations.

Glossary

Token

A piece of text, usually part of a word. The model reads and writes in tokens, and providers price a million tokens separately for input and output.

Prefill

The first phase: the model processes the whole prompt in one pass. Thousands of tokens go in parallel, so the GPU is busy with arithmetic.

Decode

The second phase: the model emits the answer one token per step. At every step the GPU re-reads the model weights and the accumulated context from memory. Little arithmetic per step, a lot of reading.

KV cache

The intermediate results of prefill for every token of the context (the attention keys and values, hence K and V). Decode reads them at every step, so they stay in GPU memory for as long as the request lives. Its size is bytes per token, and it differs a hundredfold between models.

Cache hit

The share of input tokens whose KV cache already exists: the same system prompt, say, or an agent’s previous turns. Those tokens need not be recomputed, and APIs sell them cheaper. The rest we call uncached.

HBM

The memory on the GPU itself. It is two different resources: capacity (how many gigabytes of weights and KV fit) and bandwidth (how many terabytes per second can be read).

Compute-bound / memory-bound

What runs out first: arithmetic (compute-bound, that is prefill) or reading from memory (memory-bound, that is decode).

TTFT, ITL

Time to first token: how long you wait for the first token of the answer. Inter-token latency: the pause between the following tokens. Prefill sets TTFT, decode sets ITL.

SLO

The promised service level, for example “ITL p95 ≤ 50 ms”. Requests that meet the SLO are called goodput.

Pool

A group of GPUs with one role. In a monolith one shared pool does both prefill and decode. With disaggregation there are two pools, and the KV cache travels from the prefill pool to the decode pool over the network.

GPU-hour, TCO

Renting one GPU for an hour is a provider’s main cost line. TCO (total cost of ownership) is the full cost: GPU-hours plus network, storage and operations.

Sources

• Our measurements: DeepSeek-V4.1-Flash on 8×B300: vLLM vs DeepInfer (October 21, 2026).
• vLLM: DeepSeek-V4.1-Flash, P/D and wide-EP (October 23, 2026).
• DistServe · Splitwise · Sarathi-Serve · Mooncake · Beyond the Buzz · Not All Prefills Are Equal · Together AI, cache-aware disaggregation
• ThunderServe · Llumnix · Preble · SGLang · MegaScale-Infer · CacheBlend · Cake
• API prices: DeepSeek. The $9.50 GPU-hour is one cloud’s public on-demand B300 price, October 2026.