---
title: "Confidential inference for the Enterprise: attested, measured, and where it stands"
description: "Private, confidential LLM inference for the enterprise: AMD SEV-SNP and NVIDIA GPU CC on RTX PRO 6000 and 8×B300. Attestation, measured cost, a silent vLLM bug."
author: Kirill Bespalov
date: 2026-10-26
url: https://warmctl.io/blog/confidential-inference/
---

# Confidential inference for the Enterprise: attested, measured, and where it stands

> October 2026 was a month of breaches at cloud and AI platforms. Everyone encrypts data in transit (TLS/HTTPS) and at rest (KMS); data in use (RAM, vRAM), the weights and prompts inside a running model, stays readable by whoever runs your neocloud provider’s compute node, and a zero-day in the hypervisor hands that to an attacker too: every VM on the host, read out or cloned whole. For a bank under DORA and PCI DSS, or a healthcare or pharma company under HIPAA and GDPR, serving a private model on it is the whole risk. Confidential computing closes that gap. We ran open-weight models on an RTX PRO 6000 and on eight B300s in that mode, verified the proof, measured the cost, and hit a silent vLLM bug, hard performance limits and missing features.

**Key results**

- On an AMD SEV-SNP guest (EPYC Turin) with one NVIDIA RTX PRO 6000 Blackwell Server Edition in confidential computing mode (driver 595.91.07, CC state ON, environment PRODUCTION, DevTools off, all 97 GB of VRAM protected), a fresh SEV-SNP attestation report with a random nonce verified against the AMD KDS VCEK chain for Turin (ECDSA P-384), and the GPU attestation report verified with NVIDIA’s local GPU verifier against the driver and VBIOS RIMs.
- Under GPU confidential computing, host-to-device and device-to-host copies over a PCIe 5.0 x16 link reach 17–21 GB/s for 16 MiB and larger, a 4 KiB copy costs about 15 µs, a non-blocking copy blocks the calling thread for its whole duration, and two streams do not overlap; device-to-device copies run at about 730 GB/s and a BF16 GEMM at 451 TFLOPS, unaffected.
- Stock vLLM 0.32.0 and the 2026-10-26 nightly return garbage under GPU confidential computing with every attention backend, with CUDA graphs on or off, while the same weights in Transformers answer correctly: the V2 model runner reads input ids through a UVA view of pinned host memory that the CC driver reports as managed memory, so the GPU sees a stale copy; VLLM_USE_V2_MODEL_RUNNER=0 fixes it (vllm-project/vllm issue 57224). SGLang 0.5.22 is unaffected.
- Serving under GPU confidential computing with vLLM 0.32.0 (V1 runner, CUDA graphs) on one RTX PRO 6000, 1,024-token prompts and 512-token answers: Qwen3-8B decodes one stream at 12.5 ms per token and produces 2,565 output tok/s with 64 concurrent requests; Qwen3-14B 21.6 ms and 1,691 tok/s; Qwen3-0.6B 2.6 ms and 6,316 tok/s; Phi-4-mini 8.0 ms and 3,504 tok/s; DeepSeek-R1-Distill-Llama-8B 12.2 ms and 2,732 tok/s; gpt-oss-20b 5.0 ms and 4,155 tok/s. SGLang 0.5.22 matches vLLM on Qwen3-8B within 1% at 8, 32 and 64 requests.
- A GPU process that filled 48 GiB of VRAM with a marker and exited left nothing behind for the next process in CC mode: 0 marker words, 100% zeros.
- cuda-checkpoint cannot lock a CUDA process in this confidential VM (CUDA_ERROR_NOT_SUPPORTED), and cudaHostRegister is not supported, so GPU snapshot and restore of a serving engine is not possible inside a CC guest with driver 595.

Can we prove it?YesCPU and GPU sign reports with our own nonce; we checked both against AMD and NVIDIA, not the provider.
    Does it slow the GPU?NoVRAM is isolated, not encrypted, so compute runs at full speed. Batched serving loses a few percent.
    What does it cost?The busHost ↔ GPU copies run at 17–21 GB/s, a third of the link, and every start is cold: no GPU snapshots.
    Works out of the box?Not yetStock vLLM serves silent garbage. SGLang works; vLLM needs one environment variable.
  1× RTX PRO 6000 and 8× B300 · AMD SEV-SNP · NVIDIA confidential computing, encrypted NVLink on B300

The breaches behind that first sentence: [Railway](https://thehackernews.com/2026/10/railway-says-intruder-may-have-accessed.html) asked every customer to rotate every secret, [LiteLLM’s](https://thehackernews.com/2026/10/leaked-litellm-master-key-let-attackers.html) release pipeline was hijacked to steal inference tokens, and [OpenCode, Zeabur and PingCAP](https://thehackernews.com/2026/10/threatsday-ai-secrets-leak-pingcap.html) reported stolen credentials. All of them stopped at credentials; an attacker who reaches a compute node does not need any.

##
Secure sensitive data in transit, at rest and in use

  Data lives in three places, and each needs its own protection:

    - **In transit**, on the network: the prompt going out, the tokens coming back. TLS (HTTPS) encrypts it on the wire.

    - **At rest**, on disk: model files, logs, backups. Disk or bucket encryption with keys in a KMS covers it.

    - **In use**, in memory: weights in GPU memory, prompts and the KV cache in RAM, tensors on PCIe. The machine must read it to compute on it, so until confidential computing nothing protected it.

  one inference request · three states of datain transit · encryptedcomputer**your app**prompt, answerTLS
in transitcompute node · GPUin useRAM**—**PCIe**—**VRAM**—**KMSat restdisk**model files**volumes, logsTLS and KMS cover two of the threein use: readable by the host
  The middle belongs to the host. A hypervisor can read any page of any guest it runs, and a VM is a file: pause it, dump it, clone it, and the clone answers questions with your model. GPU memory is reachable from the host through the card’s PCIe windows; the traffic between CPU and GPU crosses the bus unencrypted. None of this needs a malicious provider, only one compromised operator account, and the tenant never learns it happened. On a shared host every tenant is in the same position.

  one host · three tenants · root on the hypervisorplain VMsbank datahealth datayour modelhypervisor**root**dumpmemory copyattacker’s copyemptysame host, same root, same dump0 of 3 VMs copied

Confidential computing is the hardware answer: the CPU encrypts the VM’s memory with a key the hypervisor never holds, the GPU fences its memory and encrypts its bus, and both sign a report of what they are running, so the tenant can check before sending a single prompt. An attacker with full root on the host sees ciphertext in RAM, ciphertext on the bus, a GPU that refuses the host’s reads, and a proof they cannot forge. That is the promise; the rest of this post is what it costs and where it currently breaks.

##
What confidential computing changes, link by link

  Four mechanisms have to hold at once, and each covers what the others cannot:

  chain of trust · four linkscheckingevery link must holdCPU TEEguest RAM encryptedGPU CCVRAM isolated, links encryptedattestationsigned CPU + GPU reportskey releasemodel key to the enclaveeach link closes what the others cannot0 of 4 links

The CPU and the GPU protect data in use differently, and the difference explains most of this post. The CPU encrypts memory itself; the GPU encrypts only what leaves the chip:

| Where the data is | Under CC | How |
|---|---|---|
| Guest RAM | encrypted | per-VM key, in the CPU’s memory controller |
| CPU ↔ GPU over PCIe | encrypted | SPDM keys, AES-GCM bounce buffers |
| GPU ↔ GPU over NVLink | encrypted | NVLE on Blackwell; not on Hopper |
| GPU memory (VRAM) | isolated, not encrypted | only your VM reaches it; wiped on exit |

**Why VRAM is not encrypted.** Pushing HBM’s 8 TB/s through AES would cost the GPU the bandwidth it is bought for. So the GPU guards its memory instead: it sits on the same package as the chip, the host cannot reach it while CC is on, and it is wiped between processes. NVIDIA documents this design for Hopper. That is why compute runs at full speed while every copy over PCIe pays (section 05).

We checked one of the claims directly. A process filled 48 GiB of the GPU’s memory with a marker word and exited without clearing it; a second process allocated the same 48 GiB and scanned it: 0 marker words, 100% zeros. Since VRAM is isolated rather than encrypted, this wipe is part of the guarantee, not a courtesy. On GPUs in the clear, a decade of papers has found the previous tenant’s data in freshly allocated memory; this one scrubbed it. One run, one card, no paired run without CC, but it is the right answer.

One thing confidential computing does not do, and people often expect it to: it protects the VM from the host, not a process from root inside the same VM.

  who can read a process’s memory · one confidential VMprocess writes a secretoutside the VM**root on the host**—by design · not measuredhost
readconfidential VM · SEV-SNPprocess**./secret**—guestreadinside the VM**root in the guest**—measuredsame memory, two readersthe boundary is the VM
  The left half is the design: the CPU encrypts guest memory with a key the hypervisor never holds, so a host-side dump of those pages is ciphertext. Showing it takes hypervisor access, which a rented confidential VM does not give, so we did not measure it. The right half we did measure: as root inside the guest we read the string straight out of the process’s memory through `/proc/71157/mem`. Everything inside the VM, Vault included, trusts the guest kernel and whoever is root in it; what the enclave buys you is that nobody outside gets in.

##
The machine

|  |  |
|---|---|
| Provider | Verda, Finland (FIN-03): one VM with an RTX PRO 6000, one with 8×B300 |
| Guest | Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic, KVM, no nested virtualization |
| CPU | AMD EPYC Turin, 30 vCPU, 87 GiB RAM, SEV-SNP at VMPL 0 |
| GPU | RTX PRO 6000 Blackwell Server Edition, 96 GB, PCIe 5.0 x16 |
| Driver | 595.91.07 open kernel module, CUDA 13.4 |
| CC | ON, PRODUCTION, DevTools OFF, all VRAM protected |
| Engines | vLLM 0.32.0 and the 2026-10-26 nightly, SGLang 0.5.22, Transformers |

  We ran everything on Verda. Nebius support told us that none of its platforms supports confidential computing today.

```
$ nvidia-smi conf-compute -q
    CC State                   : ON
    Multi-GPU Mode             : None
    CPU CC Capabilities        : AMD SEV-SNP
    GPU CC Capabilities        : CC Capable
    CC GPUs Ready State        : Ready
$ nvidia-smi conf-compute -e; nvidia-smi conf-compute -d; nvidia-smi conf-compute -gm
CC Environment: PRODUCTION
DevTools Mode: OFF
Protected memory size:   99463296 KiB
Unprotected memory size: 0 KiB
$ dmesg | grep -E "SEV|confidential"
Memory Encryption Features active: AMD SEV SEV-ES SEV-SNP
SEV: SNP running at VMPL0.
software IO TLB: Memory encryption is active and system is using DMA bounce buffers
systemd[1]: Detected confidential virtualization sev-snp.
```

Two checks before any benchmark. **Production, not DevTools:** DevTools mode relaxes the protections so profilers work; ours ran production. **On is not ready:** CUDA fails with `system not yet initialized` until the guest runs `nvidia-smi conf-compute -srs 1`, which should come only after GPU attestation.

##
Prove it from inside

  Everything above is what the vendors say the hardware does. Attestation is how you check it without taking the provider’s word: you send a fresh random nonce, the CPU and the GPU each sign a report that contains it and what they are running, and you verify both signatures against AMD’s and NVIDIA’s roots, keys the provider does not hold.

  how an enclave proves itself · remote attestationyou send a fresh noncetenant**you**—fresh random noncenoncechallengeconfidential VMCPU · AMD SEV-SNP**signs a report**—signed · chip key (VCEK)GPU · CC mode**signs a report**—signed · device key
model keyreportsevidenceyou verify**against vendor roots**AMD chain · ARK → VCEK**·**NVIDIA chain · RIMs**·**nonce is yours**·**debug off**·**firmware golden**·**no provider key involvedtrust AMD and NVIDIA, not the provider0 of 5 checks
  We did both from inside the VM, about 100 lines of Python (scripts in the post’s data folder):

|  | CPU · SEV-SNP | GPU · CC mode |
|---|---|---|
| Report | 1,184 bytes via configfs-tsm | SPDM, via NVML |
| Signed by | VCEK, a chip-unique key | the GPU’s device key |
| Checked against | AMD KDS: ARK → ASK → VCEK | NVIDIA cert chain and RIM service |
| Result | valid; nonce ours, debug off | valid; driver and VBIOS golden |

    - **Use your own fresh nonce.** A report requested with a fixed string is replayable; an agent we had run earlier that morning did exactly that.

    - **Turin needs the 8-byte chip id** for the KDS lookup; the 64-byte field from the report returns 404.

    - **A valid signature is not an identity.** It proves this VM has that launch measurement; to know what the number means you need the provider’s firmware, kernel and initrd to recompute it, which we did not have.

The last link is the key. We ran it with Vault as the broker: a script checks both reports, then pulls the key and decrypts the model; with a wrong expected measurement it refuses. KServe 0.20’s confidential model serving does the same with a Key Broker Service such as Trustee, but it needs Confidential Containers, and those need nested KVM, which this guest does not have.

##
What confidential computing costs

  In CC mode the GPU cannot read guest memory and the CPU cannot touch GPU memory, so every copy between them takes a detour through a shared bounce buffer: the CPU encrypts the data into it, the GPU pulls the ciphertext over PCIe, and its copy engine decrypts it into VRAM. On a single card this is the only thing the GPU side encrypts; VRAM itself stays unencrypted behind the isolation. That is the cost, and the reason it lands on copies, not on compute.

  one copy, host → GPU · CC modeyour prompt in guest RAMconfidential VM · guest RAM**your prompt**············host sees: encrypted RAM1 · encryptCPU · AES-GCMbounce buffer · shared memory**the only door**············host sees: ciphertext only2 · DMAPCIeGPU · VRAM**3 · copy engine decrypts**············host sees: no accessGPU → host runs in reverse: the GPU encrypts, the CPU decryptsstep 0 of 3

We measured it with synchronous `cudaMemcpy` from pinned memory, medians over a batch, while the link was at full Gen5 x16:

| Copy size | H2D | H2D | D2H | D2D |
|---|---|---|---|---|
| 4 KiB | 15 µs | 0.3 GB/s | 0.3 GB/s | 1 GB/s |
| 64 KiB | 19 µs | 3.4 GB/s | 3.7 GB/s | 9 GB/s |
| 1 MiB | 85 µs | 12.3 GB/s | 13.7 GB/s | 126 GB/s |
| 16 MiB | 878 µs | 19.1 GB/s | 20.9 GB/s | 1,929 GB/s |
| 256 MiB | 15,395 µs | 17.4 GB/s | 18.3 GB/s | 731 GB/s |
| 1 GiB | 60,896 µs | 17.6 GB/s | 18.2 GB/s | 728 GB/s |

  host → GPU copies · GPU CC onpinned · medianPCIe 5.0 x16 in the clear ≈ 504 KiBRTX PRO 6000**0.3 GB/s**B300**0.1 GB/s**1 MiBRTX PRO 6000**12.3 GB/s**B300**10.8 GB/s**16 MiBRTX PRO 6000**19.1 GB/s**B300**19.8 GB/s**1 GiBRTX PRO 6000**17.6 GB/s**B300**17.7 GB/s**RTX PRO 6000 on PCIe 5.0 · B300 on PCIe 6.0same 17–21 GB/s ceiling

    - **Bulk copies reach 17–21 GB/s**, against about 50 GB/s on the same link in the clear. That is still twice an H100 NVL in a TDX guest (8.4 GB/s), so Blackwell made the path better, not free.

    - **Small copies cost about 15 µs each**, from 256 bytes to 4 KiB. Every decode step makes a few: input ids in, tokens out.

    - **Async is not async.** A 256 MiB `cudaMemcpyAsync` blocked the caller for the whole 15.2 ms transfer, so CPU work stops overlapping GPU work.

    - **Streams do not overlap.** Two such copies on two streams took 29.2 ms, back to back.

    - **Compute is untouched.** Device-to-device copies ran at 728 GB/s and a BF16 GEMM at 451 TFLOPS.

Plain PyTorch shows the same. A 4 KiB `non_blocking` copy returns in 15 µs on an idle stream, but after 85.5 ms behind 85 ms of matmuls, because it waits for the queue to drain. A decode loop that sends every token through the host pays 0.14 ms per token on Qwen3-0.6B (1%) and 0.52 ms on Qwen3-8B (2%).

**What it means.** CC does not slow the GPU, it slows the door into it, because the door is the only part that is encrypted. Inference that keeps its work on the GPU and batches requests pays a few percent. Anything that crosses PCIe often or in bulk pays in full: host-side token loops, weight loading and KV-cache offload to CPU memory run at 17–21 GB/s with no overlap.

##
“!!!!” instead of “Paris”

  Then we started vLLM. It loaded Qwen3-8B, served 2,255 tokens per second at 32 requests, and every answer was garbage, with not one warning in the log.

```
$ curl -s localhost:8000/v1/completions -d '{"model":"Qwen/Qwen3-8B","prompt":"The capital of France is","max_tokens":16,"temperature":0}'
" ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !"
$ docker logs vllm | grep -ciE "error|warn.*cc|confidential"
0
```

Every vLLM configuration we tried was wrong, each in its own way. Transformers and SGLang, with the same weights on the same GPU, were right:

| Stack | “The capital of France is” |
|---|---|
| Transformers, eager, same weights | Paris. The capital of France is a… |
| vLLM nightly, TRITON_ATTN | 今天早上起来发现没有找到对应的业务需求不2. 3 3 3 3 |
| vLLM nightly, FLASHINFER | \n\n2222222222222222222222 |
| vLLM 0.32.0 stable, CUDA graphs | !!!!!!!!!!!!!!!!!!!!!!!* |
| vLLM 0.32.0, VLLM_USE_V2_MODEL_RUNNER=0 | Paris. The capital of France is a… |
| SGLang 0.5.22, defaults | Paris. The capital of France is a… |

Weights, kernels and plain copies all checked out byte for byte. The first layer gave it away: the GPU was not reading the prompt we sent.

prompt we sent       [785, 6722, 315, 9625, 374]      # "The capital of France is"
what the GPU read    [53461, 94817, 112247, 62908, 0]
  **The cause.** vLLM’s default V2 model runner does not copy the prompt to the GPU. It maps a CPU buffer once and lets the GPU read it in place. In CC mode the driver does not treat that buffer as shared, so the GPU gets a one-time snapshot and keeps reading the first, stale contents forever. This is vLLM issue [#57224](https://github.com/vllm-project/vllm/issues/57224), first seen on an H100 in a TDX guest. Until PRs [#57415](https://github.com/vllm-project/vllm/pull/57415) or [#57404](https://github.com/vllm-project/vllm/pull/57404) land, one variable switches back to the older runner with explicit copies:

$ docker run --gpus all -e VLLM_USE_V2_MODEL_RUNNER=0 -p 8000:8000 vllm/vllm-openai:v0.32.0 Qwen/Qwen3-8B
$ curl -s localhost:8000/v1/completions -d '{"model":"Qwen/Qwen3-8B","prompt":"The capital of France is","max_tokens":16,"temperature":0}'
" Paris. The capital of France is also the capital of the French Republic."
  **Not every inference engine is ready for CC out of the box.** vLLM’s default path served wrong answers at full speed with a clean log; SGLang and vLLM’s older runner were fine. Attestation cannot catch this: it proves the hardware is genuine, not that the software computes the right thing. Run a canary, a prompt with a known answer, after every start and before the port opens.

## Six models in CC mode

  With the runner fixed, how fast does a confidential GPU serve? vLLM 0.32.0 with the V1 runner and CUDA graphs, and SGLang 0.5.22 for comparison, each with 85% of the memory and a 4K context; `vllm bench serve` from a sidecar container with random 1,024-token prompts, 512-token answers and `--ignore-eos`, from 1 to 64 requests at once. Six models that people actually run on a 96 GB card: Qwen3 at 0.6B, 8B and 14B, Phi-4-mini, DeepSeek-R1-Distill-Llama-8B and gpt-oss-20b.

  RTX PRO 6000 · GPU CC on64 requests · tok/sQwen3-0.6BvLLM · 2.6 ms/token**6,316**Phi-4-minivLLM · 8.0 ms/token**3,504**gpt-oss-20bvLLM · 4.9 ms/token**4,155**Qwen3-8BvLLM · 12.5 ms/token**2,565**Qwen3-8BSGLang · 12.3 ms/token**2,565**R1-Distill-Llama-8BvLLM · 12.2 ms/token**2,732**Qwen3-14BvLLM · 21.5 ms/token**1,691**1K prompt, 512-token answer, V1 runnervLLM = SGLang within 1%

| Model | Engine | ms / token | 1 request | 8 | 32 | 64 |
|---|---|---|---|---|---|---|
| Qwen3-0.6B | vLLM | 2.6 | 380† | 2,190 | 4,871 | 6,316 |
| Qwen3-8B | vLLM | 12.5 | 80† | 569 | 1,639 | 2,565 |
| Qwen3-8B | SGLang | 12.3 | 81† | 567 | 1,635 | 2,565 |
| Qwen3-14B | vLLM | 21.5 | 46† | 336 | 1,025 | 1,691 |
| Phi-4-mini 3.8B | vLLM | 8.0 | 125† | 856 | 2,348 | 3,504 |
| DeepSeek-R1-Distill-Llama-8B | vLLM | 12.2 | 82† | 590 | 1,719 | 2,732 |
| gpt-oss-20b | vLLM | 4.9 | 202† | 885 | 2,508 | 4,155 |
| Qwen3-8B, --enforce-eager | vLLM | 13.9 | 72† | 503 | — | — |
| Qwen3-8B, --no-async-scheduling | vLLM | 12.7 | 79† | 575 | — | — |
| Qwen3-0.6B | SGLang | 2.5 | 405† | 2,291 | — | — |

Output tokens per second at 1, 8, 32 and 64 concurrent requests. † Single-request decode rate, measured after the one-time first-request pause.

  All runs · TTFT, TPOT, ITL p99
| Model | Engine | Requests | Output tok/s | TTFT median, ms | TPOT median, ms | ITL p99, ms |
|---|---|---|---|---|---|---|
| Qwen3-0.6B | vLLM | 1 | 380† | 22 | 2.6 | 2.7 |
| Qwen3-0.6B | vLLM | 8 | 2,190 | 54 | 3.6 | 3.8 |
| Qwen3-0.6B | vLLM | 32 | 4,871 | 97 | 6.4 | 7.9 |
| Qwen3-0.6B | vLLM | 64 | 6,316 | 64 | 10.0 | 21.7 |
| Qwen3-8B | vLLM | 1 | 80† | 75 | 12.5 | 12.7 |
| Qwen3-8B | vLLM | 8 | 569 | 365 | 13.4 | 13.9 |
| Qwen3-8B | vLLM | 32 | 1,639 | 776 | 18.3 | 18.4 |
| Qwen3-8B | vLLM | 64 | 2,565 | 252 | 23.7 | 27.0 |
| Qwen3-8B | SGLang | 1 | 81† | 98 | 12.3 | 12.6 |
| Qwen3-8B | SGLang | 8 | 567 | 361 | 13.5 | 14.1 |
| Qwen3-8B | SGLang | 32 | 1,635 | 1051 | 17.5 | 18.4 |
| Qwen3-8B | SGLang | 64 | 2,565 | 308 | 22.2 | 24.2 |
| Qwen3-14B | vLLM | 1 | 46† | 115 | 21.5 | 21.8 |
| Qwen3-14B | vLLM | 8 | 336 | 662 | 22.7 | 23.3 |
| Qwen3-14B | vLLM | 32 | 1,025 | 1314 | 28.8 | 28.8 |
| Qwen3-14B | vLLM | 64 | 1,691 | 362 | 35.3 | 37.4 |
| Phi-4-mini 3.8B | vLLM | 1 | 125† | 44 | 8.0 | 8.3 |
| Phi-4-mini 3.8B | vLLM | 8 | 856 | 161 | 9.1 | 9.4 |
| Phi-4-mini 3.8B | vLLM | 32 | 2,348 | 302 | 13.2 | 13.7 |
| Phi-4-mini 3.8B | vLLM | 64 | 3,504 | 237 | 17.8 | 77.9 |
| DeepSeek-R1-Distill-Llama-8B | vLLM | 1 | 82† | 68 | 12.2 | 12.7 |
| DeepSeek-R1-Distill-Llama-8B | vLLM | 8 | 590 | 357 | 12.9 | 13.4 |
| DeepSeek-R1-Distill-Llama-8B | vLLM | 32 | 1,719 | 735 | 17.4 | 17.6 |
| DeepSeek-R1-Distill-Llama-8B | vLLM | 64 | 2,732 | 244 | 22.1 | 24.8 |
| gpt-oss-20b | vLLM | 1 | 202† | 50 | 4.9 | 5.1 |
| gpt-oss-20b | vLLM | 8 | 885 | 227 | 8.6 | 9.8 |
| gpt-oss-20b | vLLM | 32 | 2,508 | 446 | 12.1 | 12.7 |
| gpt-oss-20b | vLLM | 64 | 4,155 | 240 | 14.8 | 43.5 |
| Qwen3-8B, --enforce-eager | vLLM | 1 | 72† | 74 | 13.9 | 14.1 |
| Qwen3-8B, --enforce-eager | vLLM | 8 | 503 | 368 | 15.3 | 15.6 |
| Qwen3-8B, --no-async-scheduling | vLLM | 1 | 79† | 60 | 12.7 | 12.8 |
| Qwen3-8B, --no-async-scheduling | vLLM | 8 | 575 | 342 | 13.3 | 13.6 |
| Qwen3-0.6B | SGLang | 1 | 405† | 50 | 2.5 | 2.6 |
| Qwen3-0.6B | SGLang | 8 | 2,291 | 62 | 3.3 | 3.6 |

    - **The engine does not matter.** vLLM and SGLang match within 1% on Qwen3-8B at every load. Both are bound by the GPU, so the bridge tax is already hidden in their per-token times.

    - **One user barely feels CC.** Qwen3-8B decodes a token in 12.5 ms, and about 10 of those are simply reading the weights from GPU memory.

    - **Tuning works as usual.** CUDA graphs and async scheduling help as much as in the clear, and the queue, not the bridge, sets time to first token.

    - **The first vLLM request is slow.** It took 23 s on four of six models, then 0.1 s. We could not tie it to CC; warm the engine before opening the port.

Startup is the other story: one to three minutes to ready, as in the clear, but with nothing to skip it.

| Model | Engine | Ready, s | Weights, s | First request, s |
|---|---|---|---|---|
| Qwen3-0.6B | vLLM | 151 | 0.1 | 22.4 |
| Qwen3-8B | vLLM | 155 | 1.1 | 23.1 |
| Qwen3-8B | SGLang | 75 | 1.2 | 0.2 |
| Qwen3-14B | vLLM | 191 | 21.6 | 23.1 |
| Phi-4-mini 3.8B | vLLM | 120 | 5.5 | 0.1 |
| DeepSeek-R1-Distill-Llama-8B | vLLM | 160 | 11.6 | 23.8 |
| gpt-oss-20b | vLLM | 120 | 4.5 | 0.1 |
| Qwen3-0.6B | SGLang | 70 | 0.3 | 0.2 |

Weights are not the bottleneck: from the page cache 15 GiB reached the GPU in about a second. Compile and graph capture take the rest, exactly as without CC. The difference is that a CC guest cannot restore a snapshot to skip them (section 09).

**Bottom line.** Throughput is not the main price of CC: a batched engine runs close to what the GPU allows. Published comparisons put stock vLLM at about 10% slower on this card and 26% on a B300, and NVIDIA’s patch ([#52226](https://github.com/vllm-project/vllm/pull/52226)) closes most of that gap. The bigger price is paid at start-up.

##
The same chain on eight B300s

  The same provider then gave us an hour on eight NVIDIA B300s with encrypted NVLink (NVIDIA calls it NVLE), the same driver and an AMD Turin SEV-SNP guest. Two questions one card could not answer: does a faster bus help, and what does encryption do to NVLink? Attestation came first, exactly as designed: CUDA refuses to start until the GPUs are verified and set ready.

```
$ nvidia-smi conf-compute -grs
Confidential Compute GPUs Ready state: not-ready
$ ./probe                     # can CUDA start?
cudaGetDeviceCount: system not yet initialized (n=-1)
$ python3 -m verifier.cc_admin --user_mode   # attest all 8 GPUs first
GPU Attestation is Successful.
$ nvidia-smi conf-compute -srs 1   # only now: let CUDA in
$ ./probe
cudaGetDeviceCount: no error (n=8)
```

|  | RTX PRO 6000 · PCIe 5 | 8×B300 · PCIe 6 |
|---|---|---|
| Host → GPU, 1 GiB | 17.6 GB/s | 17.7 GB/s |
| One 4 KiB copy | 15 µs | 33 µs |
| GPU memory copy | 728 GB/s | 3.2 TB/s |
| GPU ↔ GPU, encrypted NVLink | — | 645–659 GB/s |
| 8-GPU all-reduce, 256 MiB | — | 494 GiB/s |

    - **A faster bus does not help.** PCIe 6 doubles the link, yet the encrypted bridge stays at about 18 GB/s. The limit is encryption on the CPU side, not the wire.

    - **Encrypted NVLink is fast.** GPUs talk to each other at hundreds of GB/s, so tensor parallelism stays inside the enclave at full speed. NVLE encrypts the link; HBM stays unencrypted and isolated, as on the RTX card. Only traffic to the CPU crawls.

    - **Same bugs, same fixes.** vLLM’s default runner returned garbage again; the V1 runner, SGLang and an 8-GPU tensor-parallel vLLM all answered “Paris”. `cuda-checkpoint` refused again.

    - **Memory is wiped between tenants.** A process filled 260 GiB of one GPU with a marker and exited; the next process found none of it. With VRAM unencrypted, this wipe is what keeps one tenant’s weights from the next.

  8×B300 · GPU CC on · NVLE64 requests · tok/sQwen3-8BvLLM · TP8 · 3.3 ms**12,128**Qwen3-8BSGLang · TP1 · 4.1 ms**8,280**Qwen3-8BvLLM · TP1 · 4.6 ms**7,842**Qwen3-235B-A22B-FP8vLLM · TP8 · 9.6 ms**3,285**1K prompt, 512-token answerTP8 over encrypted NVLink

| Model | Engine | ms / token | 1 request | 8 | 32 | 64 |
|---|---|---|---|---|---|---|
| Qwen3-8B | vLLM · TP1 | 4.6 | 216† | 1,417 | 4,837 | 7,842 |
| Qwen3-8B | SGLang · TP1 | 4.1 | 242† | 1,635 | 4,996 | 8,280 |
| Qwen3-8B | vLLM · TP8 | 3.3 | 304† | 1,789 | 7,170 | 12,128 |
| Qwen3-235B-A22B-FP8 | vLLM · TP8 | 9.6 | 105† | 554 | 1,922 | 3,285 |

Output tokens per second at 1, 8, 32 and 64 concurrent requests, 1K prompt and 512-token answer. † Single-request decode rate.

  All runs · TTFT, TPOT, ITL p99
| Model | Engine | Requests | Output tok/s | TTFT median, ms | TPOT median, ms | ITL p99, ms |
|---|---|---|---|---|---|---|
| Qwen3-8B | vLLM, TP1 | 1 | 216† | 28 | 4.6 | 4.8 |
| Qwen3-8B | vLLM, TP1 | 8 | 1,417 | 107 | 5.0 | 5.2 |
| Qwen3-8B | vLLM, TP1 | 32 | 4,837 | 169 | 6.3 | 8.9 |
| Qwen3-8B | vLLM, TP1 | 64 | 7,842 | 147 | 7.9 | 42.3 |
| Qwen3-8B | SGLang, TP1 | 1 | 242† | 26 | 4.1 | 4.3 |
| Qwen3-8B | SGLang, TP1 | 8 | 1,635 | 128 | 4.5 | 4.8 |
| Qwen3-8B | SGLang, TP1 | 32 | 4,996 | 367 | 5.5 | 5.9 |
| Qwen3-8B | SGLang, TP1 | 64 | 8,280 | 193 | 6.9 | 7.9 |
| Qwen3-235B-A22B-FP8 | vLLM, TP8 | 1 | 105† | 160 | 9.6 | 9.7 |
| Qwen3-235B-A22B-FP8 | vLLM, TP8 | 8 | 554 | 500 | 12.6 | 13.0 |
| Qwen3-235B-A22B-FP8 | vLLM, TP8 | 32 | 1,922 | 638 | 15.5 | 15.8 |
| Qwen3-235B-A22B-FP8 | vLLM, TP8 | 64 | 3,285 | 279 | 18.1 | 20.6 |
| Qwen3-8B | vLLM, TP8 | 1 | 304† | 32 | 3.3 | 3.4 |
| Qwen3-8B | vLLM, TP8 | 8 | 1,789 | 67 | 3.8 | 4.0 |
| Qwen3-8B | vLLM, TP8 | 32 | 7,170 | 118 | 4.2 | 5.7 |
| Qwen3-8B | vLLM, TP8 | 64 | 12,128 | 88 | 5.1 | 27.9 |

    - **A faster GPU makes the loop visible.** One B300 decodes Qwen3-8B 2.7× faster than the RTX card. Here SGLang leads vLLM by 5–13%: when the GPU step is this short, CPU-side work and the bridge start to show.

    - **Big models work.** Qwen3-235B in FP8 across eight GPUs decodes at 9.6 ms per token, reaches 3,519 tok/s at 128 requests, and starts answering a 32K-token prompt in under a second.

    - **Start-up is slow, but not because of CC.** The weight loader, not the bridge, sets the pace: the 235B model was ready in 593 s from disk and 379 s with its weights already in memory.

| Model | Engine | Ready, s | Weights, s |
|---|---|---|---|
| Qwen3-8B | vLLM, TP1 | 166 | 1.7 |
| Qwen3-8B | SGLang, TP1 | 151 | 11.9 |
| Qwen3-235B-A22B-FP8 | vLLM, TP8 | 593 | 195.8 |
| Qwen3-235B-A22B-FP8, weights in memory | vLLM, TP8 | 379 | 14.0 |
| Qwen3-8B | vLLM, TP8 | 261 | 0.4 |

**What B300 adds.** Scale does not change the picture. GPUs and NVLink run at full speed inside the enclave, the host bridge stays at about 18 GB/s whatever the bus, and the engines need the same fixes. The faster the GPU, the more the CPU-side loop costs, so CC-aware engines matter more on B300 than on a workstation card.

##
What CC mode will not do

| Feature | Under CC | What it means |
|---|---|---|
| GPU snapshots (cuda-checkpoint) | not supported | every start is cold |
| KV-cache offload to host memory | not supported | no KV offload in vLLM |
| GPU reading host memory in place | stale data | breaks vLLM’s V2 runner |
| Zero-copy and managed memory | works | explicit sharing is fine |
| NVLink between GPUs | encrypted | tensor parallelism at full speed |
| Nested virtualization | none | no Confidential Containers inside |

```
$ cuda-checkpoint --action lock --pid $(pgrep -f EngineCore)
CUDA_ERROR_NOT_SUPPORTED: operation not supported
$ cuda-checkpoint --action lock --pid $(pgrep victim)      # a plain 1 GiB CUDA process
CUDA_ERROR_NOT_SUPPORTED: operation not supported
```

  **What it means for warm starts.** warmctl restores an engine in seconds from a snapshot of its CPU and GPU state. In a CC guest the GPU checkpoint API refuses every CUDA process, so a confidential replica pays the full cold start each time. Until NVIDIA supports checkpoints in CC mode, the only levers are kernel caches and fast weight loading.

##
What total confidentiality costs

  Confidential computing closes the last gap, data in use, and on Blackwell the hardware delivers. This is the bill we measured.

  the bill · measured on RTX PRO 6000 and 8×B300what confidentiality costsGPU compute451 TFLOPS BF16; VRAM isolated, not encryptedfreeBatched servingvLLM and SGLang within 1%; bridge 1–2% of a decode stepa few %Host ↔ GPU copies17–21 GB/s on PCIe 5 and 6, synchronous, no overlap⅓ of the linkCold startsno GPU snapshots, no KV offload to host memoryevery startEngine readinessvLLM default serves silent garbage; one flag and a canarycheck yoursyou getRAM encrypted · PCIe and NVLink encrypted · VRAM isolated and wiped · proof signed by AMD and NVIDIA · keys only for a verified VMcompute stays free, the doorway does not5 of 5 items

**Worth it, with eyes open.** Know what you are buying: the CPU encrypts RAM, while the GPU encrypts only its links and guards VRAM by isolation and wiping, which is exactly why it keeps full speed. For a bank or a healthcare company that must keep prompts and weights away from the provider, the price is low where it matters most: the GPU runs at full speed and a batched engine loses a few percent. The real costs are operational: slow, always-cold starts and engines that are not yet CC-aware by default.

### Before you ship

    - **Verify both reports yourself**, with your own nonce, against AMD and NVIDIA; reject DevTools mode and a debug policy.

    - **Ask the provider for reference images** (firmware, kernel, initrd), or the launch measurement is a number you cannot interpret.

    - **Pin the engine:** SGLang as is, or vLLM with `VLLM_USE_V2_MODEL_RUNNER=0` until #57415 lands.

    - **Run a canary** with a known answer after every start, before the port opens.

Every script and raw output from these runs is in the post’s data folder.

### Sources

    - October 2026 incidents: [Railway](https://thehackernews.com/2026/10/railway-says-intruder-may-have-accessed.html), [LiteLLM](https://thehackernews.com/2026/10/leaked-litellm-master-key-let-attackers.html), [OpenCode, Zeabur, PingCAP (ThreatsDay)](https://thehackernews.com/2026/10/threatsday-ai-secrets-leak-pingcap.html).

    - vllm-project/vllm issue [#57224](https://github.com/vllm-project/vllm/issues/57224): silent garbage output in CC mode, V2 runner UVA views; PRs [#57415](https://github.com/vllm-project/vllm/pull/57415) and [#57404](https://github.com/vllm-project/vllm/pull/57404); issue [#55428](https://github.com/vllm-project/vllm/issues/55428) on cudaHostRegister.

    - NVIDIA, [Decode optimizations for NVIDIA confidential computing](https://github.com/vllm-project/vllm/pull/52226) (vLLM PR #52226) and the [CPU↔GPU bridge microbenchmark](https://github.com/vllm-project/vllm/pull/60338) (PR #60338).

    - [The Serialized Bridge: Understanding and Recovering LLM Serving Performance under Blackwell GPU Confidential Computing](https://arxiv.org/abs/2606.23969) (arXiv 2606.23969).

    - NVIDIA, [Confidential Compute on NVIDIA Hopper H100](https://images.nvidia.com/aem-dam/en-zz/Solutions/data-center/HCC-Whitepaper-v1.0.pdf) (whitepaper) and [Hardware-rooted AI security that won’t slow you down](https://developer.nvidia.com/blog/hardware-rooted-ai-security-that-wont-slow-you-down/) (Blackwell, RTX PRO 6000).

    - [NVIDIA nvtrust](https://github.com/NVIDIA/nvtrust) (local GPU verifier) and the [attestation SDK](https://github.com/NVIDIA/attestation-sdk); AMD [SEV-SNP](https://www.amd.com/en/developer/sev.html) and the KDS at kdsintf.amd.com.

    - KServe 0.20, [confidential model serving](https://kserve.github.io/website/blog/kserve-0.20-release); [Blueprint, Bootstrap, and Bridge: a security look at NVIDIA GPU confidential computing](https://arxiv.org/abs/2507.02770).

(()=>{const R=matchMedia('(prefers-reduced-motion: reduce)').matches;const W=t=>new Promise(r=>setTimeout(r,t));
const RUN={
path:async f=>{const st=f.querySelector('.cx-state'),rows=[...f.querySelectorAll('.cx-s b')],set=(t,c)=>{st.textContent=t;st.className='cx-state '+(c||'');};
 for(;;){f.classList.remove('lt','lr','lu');rows.forEach(b=>b.textContent='—');
 f.dataset.s='t';set('in transit · sending');await W(1250);f.classList.add('lt');set('in transit · encrypted','ok');await W(1500);
 f.dataset.s='u';set('in use · plaintext','bad');f.classList.add('lu');rows.forEach(b=>b.textContent='plaintext');await W(2600);
 f.dataset.s='r';set('at rest · writing');await W(1250);f.classList.add('lr');set('at rest · encrypted','ok');await W(2400);}},
attest:async f=>{const st=f.querySelector('.cx-state'),ct=f.querySelector('.cx-count'),n=f.querySelector('.at-n'),o=[...f.querySelectorAll('.at-o')],ck=[...f.querySelectorAll('.at-ck')],set=(e,t,c)=>{e.textContent=t;e.className=e.className.split(' ')[0]+' '+(c||'');};
 const type=async(el,txt,ms)=>{for(let i=1;ie.textContent='—');ck.forEach(c=>{c.classList.remove('on');c.querySelector('b').textContent='·';});set(ct,'0 of 5 checks');
 f.dataset.s='n';set(st,'you send a fresh nonce');await type(n,'a4 68 d7 55 76 c5',30);f.classList.add('an');await W(900);
 f.dataset.s='c';set(st,'the CPU signs a report');await type(o[0],'nonce a4 68 · meas 1237…',25);f.classList.add('ac');await W(1000);
 f.dataset.s='g';set(st,'the GPU signs a report');await type(o[1],'nonce a4 68 · fw 595.91',25);f.classList.add('ag');await W(1000);
 f.dataset.s='v';set(st,'checking against AMD and NVIDIA');f.classList.add('av');await W(700);
 for(let i=0;i{const st=f.querySelector('.cx-state'),ct=f.querySelector('.cx-count'),b=f.querySelector('.bl'),rows=[...f.querySelectorAll('.bl-row')];
 b.classList.add('run');st.textContent='itemizing';for(let i=0;i{const st=f.querySelector('.cx-state'),ct=f.querySelector('.cx-count'),q=c=>[...f.querySelectorAll(c+' .bb-c i')],V=q('.bb-vm'),B=q('.bb-buf'),G=q('.bb-gpu');
 const P=['Th','e ','ca','pi','ta','l '],C=['a4','68','d7','55','76','c5'],put=(a,i,t,c)=>{a[i].textContent=t;a[i].className=c||'';},set=(t,c)=>{st.textContent=t;st.className='cx-state '+(c||'');},cnt=(t,c)=>{ct.textContent=t;ct.className='cx-count '+(c||'');};
 for(;;){f.classList.remove('la','lb');for(let i=0;i{const st=f.querySelector('.cx-state'),ct=f.querySelector('.cx-count'),hp=f.querySelector('.heap'),og=f.querySelector('.og'),oh=f.querySelector('.oh');
 const S='THIS-IS-A-CONFIDENTIAL-STRING-4711',X='a4 68 d7 55 76 c5 0b a8 22 cc',set=(e,t,c)=>{e.textContent=t;e.className=e.className.split(' ')[0]+' '+(c||'');};
 const type=async(el,txt,ms)=>{for(let i=1;i{const st=f.querySelector('.cx-state'),ct=f.querySelector('.cx-count'),em=f.querySelector('.dump em');for(;;){for(const cc of [false,true]){
 f.classList.toggle('cc',cc);f.classList.remove('go');st.textContent=cc?'confidential VMs':'plain VMs';st.className='cx-state '+(cc?'ok':'');em.textContent='empty';ct.textContent=(cc?'0 of 3 readable':'0 of 3 VMs copied');ct.className='cx-count';
 await W(900);f.classList.add('go');em.textContent='copying';await W(2200);em.textContent=cc?'ciphertext':'readable';
 ct.textContent=cc?'0 of 3 readable':'3 of 3 VMs copied';ct.className='cx-count '+(cc?'ok':'bad');await W(2600);}}},
chain:async f=>{const st=f.querySelector('.cx-state'),ct=f.querySelector('.cx-count'),T=[...f.querySelectorAll('.cx-t')],ax=f.querySelector('.cx-axis i'),N=['CPU TEE','GPU CC','attestation','key release'];
 for(;;){T.forEach(t=>t.classList.remove('on','cur'));ax.style.width='0';st.textContent='checking';st.className='cx-state';ct.textContent='0 of 4 links';ct.className='cx-count';await W(900);
 for(let i=0;it.classList.remove('cur'));T[i].classList.add('on','cur');ax.style.width=(12.5+25*i)+'%';st.textContent='checking '+N[i];ct.textContent=(i+1)+' of 4 links';await W(1300);}
 st.textContent='model key released';st.className='cx-state ok';ct.className='cx-count ok';await W(3000);}}};
document.querySelectorAll('figure.cx[data-cx]').forEach(f=>{const go=RUN[f.dataset.cx];if(!go)return;
 if(R){if(f.dataset.cx==='path'){f.classList.add('lt','lr','lu');f.dataset.s='u';f.querySelector('.cx-state').textContent='in use · plaintext';f.querySelector('.cx-state').className='cx-state bad';f.querySelectorAll('.cx-s b').forEach(b=>b.textContent='plaintext');}
  if(f.dataset.cx==='hosts'){f.classList.add('go');}if(f.dataset.cx==='attest'){f.classList.add('an','ac','ag','av','ak');f.querySelector('.at-n').textContent='a4 68 d7 55 76 c5';const o=f.querySelectorAll('.at-o');o[0].textContent='nonce a4 68 · meas 1237…';o[1].textContent='nonce a4 68 · fw 595.91';f.querySelectorAll('.at-ck').forEach(c=>{c.classList.add('on');c.querySelector('b').textContent='✓';});f.querySelector('.cx-state').textContent='model key released';f.querySelector('.cx-state').className='cx-state ok';f.querySelector('.cx-count').textContent='5 of 5 checks';f.querySelector('.cx-count').className='cx-count ok';}if(f.dataset.cx==='bounce'){const P=['Th','e ','ca','pi','ta','l '],C=['a4','68','d7','55','76','c5'];f.classList.add('la','lb');['.bb-vm','.bb-buf','.bb-gpu'].forEach((c,k)=>f.querySelectorAll(c+' .bb-c i').forEach((e,i)=>{e.textContent=k==1?C[i]:P[i];e.className=k==1?'x':'p';}));f.querySelector('.cx-state').textContent='in VRAM: readable to kernels, isolated from the host';f.querySelector('.cx-state').className='cx-state ok';f.querySelector('.cx-count').textContent='step 3 of 3';f.querySelector('.cx-count').className='cx-count ok';}if(f.dataset.cx==='secret'){f.classList.add('lh');f.dataset.s='h';f.querySelector('.cx-state').textContent='host sees ciphertext';f.querySelector('.cx-state').className='cx-state ok';f.querySelector('.heap').textContent=f.querySelector('.og').textContent='THIS-IS-A-CONFIDENTIAL-STRING-4711';f.querySelector('.og').classList.add('bad');f.querySelector('.oh').textContent='a4 68 d7 55 76 c5 0b a8 22 cc';f.querySelector('.oh').classList.add('ok');}if(f.dataset.cx==='chain'){f.querySelectorAll('.cx-t').forEach(t=>t.classList.add('on'));f.querySelector('.cx-axis i').style.width='87.5%';}return;}
 const io=new IntersectionObserver(e=>{if(e[0].isIntersecting){io.disconnect();go(f);}},{threshold:.35});io.observe(f);});})();