Research · confidential computing · RTX PRO 6000 · 8×B300

Confidential inference for the Enterprise: attested, measured, and where it stands

October 2026 was a month of breaches at cloud and AI platforms. Everyone encrypts data in transit (TLS/HTTPS) and at rest (KMS); data in use (RAM, vRAM), the weights and prompts inside a running model, stays readable by whoever runs your neocloud provider’s compute node. One zero-day CVE in the hypervisor and an attacker who lands on the host can read every VM on it, and clone them whole. For a bank under DORA and PCI DSS, or a healthcare or pharma company under HIPAA and GDPR, serving a private model on it is the whole risk.

Confidential computing closes that gap. We ran open-weight models on an RTX PRO 6000 and on eight B300s in that mode, verified the proof from inside, measured the price on the bus and per token, and hit a set of problems in the serving stack: a silent correctness bug in vLLM, performance limits no engine can route around, and features that do not exist in CC mode yet.

Kirill BespalovOctober 202614 min readRTX PRO 6000 · 8×B300 · SEV-SNP · vLLM, SGLang

The breaches behind that first sentence: Railway asked every customer to rotate every secret, LiteLLM’s release pipeline was hijacked to steal inference tokens, and OpenCode, Zeabur and PingCAP reported stolen credentials. All of them stopped at credentials; an attacker who reaches a compute node does not need any. We had four questions for the hardware, and here are the short answers:

01 · WHYSecure sensitive data in transit, at rest and in use

Data lives in three places, and each needs its own protection:

• In transit, on the network: the prompt going out, the tokens coming back. TLS (HTTPS) encrypts it on the wire.
• At rest, on disk: model files, logs, backups. Disk or bucket encryption with keys in a KMS covers it.
• In use, in memory: weights in GPU memory, prompts and the KV cache in RAM, tensors on PCIe. The machine must read it to compute on it, so until confidential computing nothing protected it.

The middle belongs to the host. A hypervisor can read any page of any guest it runs, and a VM is a file: pause it, dump it, clone it, and the clone answers questions with your model. GPU memory is reachable from the host through the card’s PCIe windows; the traffic between CPU and GPU crosses the bus unencrypted. None of this needs a malicious provider, only one compromised operator account, and the tenant never learns it happened. On a shared host every tenant is in the same position.

Confidential computing is the hardware answer: the CPU encrypts the VM’s memory with a key the hypervisor never holds, the GPU fences its memory and encrypts its bus, and both sign a report of what they are running, so the tenant can check before sending a single prompt. An attacker with full root on the host sees ciphertext in RAM, ciphertext on the bus, a GPU that refuses the host’s reads, and a proof they cannot forge. That is the promise; the rest of this post is what it costs and where it currently breaks.

02 · HOWWhat confidential computing changes, link by link

Four mechanisms have to hold at once, and each covers what the others cannot:

The CPU and the GPU protect data in use differently, and the difference explains most of this post. The CPU encrypts memory itself; the GPU encrypts only what leaves the chip:

Why VRAM is not encrypted. Pushing HBM’s 8 TB/s through AES would cost the GPU the bandwidth it is bought for. So the GPU guards its memory instead: it sits on the same package as the chip, the host cannot reach it while CC is on, and it is wiped between processes. NVIDIA documents this design for Hopper. That is why compute runs at full speed while every copy over PCIe pays (section 05).

We checked one of the claims directly. A process filled 48 GiB of the GPU’s memory with a marker word and exited without clearing it; a second process allocated the same 48 GiB and scanned it: 0 marker words, 100% zeros. Since VRAM is isolated rather than encrypted, this wipe is part of the guarantee, not a courtesy. On GPUs in the clear, a decade of papers has found the previous tenant’s data in freshly allocated memory; this one scrubbed it. One run, one card, no paired run without CC, but it is the right answer.

One thing confidential computing does not do, and people often expect it to: it protects the VM from the host, not a process from root inside the same VM.

The left half is the design: the CPU encrypts guest memory with a key the hypervisor never holds, so a host-side dump of those pages is ciphertext. Showing it takes hypervisor access, which a rented confidential VM does not give, so we did not measure it. The right half we did measure: as root inside the guest we read the string straight out of the process’s memory through /proc/71157/mem. Everything inside the VM, Vault included, trusts the guest kernel and whoever is root in it; what the enclave buys you is that nobody outside gets in.

03 · THE BENCHThe machine

We ran everything on Verda. Nebius support told us that none of its platforms supports confidential computing today.

$ nvidia-smi conf-compute -q
    CC State                   : ON
    Multi-GPU Mode             : None
    CPU CC Capabilities        : AMD SEV-SNP
    GPU CC Capabilities        : CC Capable
    CC GPUs Ready State        : Ready
$ nvidia-smi conf-compute -e; nvidia-smi conf-compute -d; nvidia-smi conf-compute -gm
CC Environment: PRODUCTION
DevTools Mode: OFF
Protected memory size:   99463296 KiB
Unprotected memory size: 0 KiB
$ dmesg | grep -E "SEV|confidential"
Memory Encryption Features active: AMD SEV SEV-ES SEV-SNP
SEV: SNP running at VMPL0.
software IO TLB: Memory encryption is active and system is using DMA bounce buffers
systemd[1]: Detected confidential virtualization sev-snp.

Two checks before any benchmark. Production, not DevTools: DevTools mode relaxes the protections so profilers work; ours ran production. On is not ready: CUDA fails with system not yet initialized until the guest runs nvidia-smi conf-compute -srs 1, which should come only after GPU attestation.

04 · ATTESTATIONProve it from inside

Everything above is what the vendors say the hardware does. Attestation is how you check it without taking the provider’s word: you send a fresh random nonce, the CPU and the GPU each sign a report that contains it and what they are running, and you verify both signatures against AMD’s and NVIDIA’s roots, keys the provider does not hold.

We did both from inside the VM, about 100 lines of Python (scripts in the post’s data folder):

• Use your own fresh nonce. A report requested with a fixed string is replayable; an agent we had run earlier that morning did exactly that.
• Turin needs the 8-byte chip id for the KDS lookup; the 64-byte field from the report returns 404.
• A valid signature is not an identity. It proves this VM has that launch measurement; to know what the number means you need the provider’s firmware, kernel and initrd to recompute it, which we did not have.

The last link is the key. We ran it with Vault as the broker: a script checks both reports, then pulls the key and decrypts the model; with a wrong expected measurement it refuses. KServe 0.20’s confidential model serving does the same with a Key Broker Service such as Trustee, but it needs Confidential Containers, and those need nested KVM, which this guest does not have.

05 · THE BRIDGEWhat confidential computing costs

In CC mode the GPU cannot read guest memory and the CPU cannot touch GPU memory, so every copy between them is staged through a shared, encrypted bounce buffer: encrypt on one side, DMA, decrypt on the other. On a single card this is the only thing the GPU side encrypts; VRAM stays unencrypted behind the isolation. That is the cost, and the reason it lands on copies, not on compute. We measured it with synchronous cudaMemcpy from pinned memory, medians over a batch, while the link was at full Gen5 x16:

• Bulk copies reach 17–21 GB/s, against about 50 GB/s on the same link in the clear. That is still twice an H100 NVL in a TDX guest (8.4 GB/s), so Blackwell made the path better, not free.
• Small copies cost about 15 µs each, from 256 bytes to 4 KiB. Every decode step makes a few: input ids in, tokens out.
• Async is not async. A 256 MiB cudaMemcpyAsync blocked the caller for the whole 15.2 ms transfer, so CPU work stops overlapping GPU work.
• Streams do not overlap. Two such copies on two streams took 29.2 ms, back to back.
• Compute is untouched. Device-to-device copies ran at 728 GB/s and a BF16 GEMM at 451 TFLOPS.

Plain PyTorch shows the same. A 4 KiB non_blocking copy returns in 15 µs on an idle stream, but after 85.5 ms behind 85 ms of matmuls, because it waits for the queue to drain. A decode loop that sends every token through the host pays 0.14 ms per token on Qwen3-0.6B (1%) and 0.52 ms on Qwen3-8B (2%).

What it means. CC does not slow the GPU, it slows the door into it, because the door is the only part that is encrypted. Inference that keeps its work on the GPU and batches requests pays a few percent. Anything that crosses PCIe often or in bulk pays in full: host-side token loops, weight loading and KV-cache offload to CPU memory run at 17–21 GB/s with no overlap.

06 · THE BUG“!!!!” instead of “Paris”

Then we started vLLM. It loaded Qwen3-8B, served 2,255 tokens per second at 32 requests, and every answer was garbage, with not one warning in the log.

$ curl -s localhost:8000/v1/completions -d '{"model":"Qwen/Qwen3-8B","prompt":"The capital of France is","max_tokens":16,"temperature":0}'
" ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !"
$ docker logs vllm | grep -ciE "error|warn.*cc|confidential"
0

Every vLLM configuration we tried was wrong, each in its own way. Transformers and SGLang, with the same weights on the same GPU, were right:

Weights, kernels and plain copies all checked out byte for byte. The first layer gave it away: the GPU was not reading the prompt we sent.

prompt we sent       [785, 6722, 315, 9625, 374]      # "The capital of France is"
what the GPU read    [53461, 94817, 112247, 62908, 0]

The cause. vLLM’s default V2 model runner does not copy the prompt to the GPU. It maps a CPU buffer once and lets the GPU read it in place. In CC mode the driver does not treat that buffer as shared, so the GPU gets a one-time snapshot and keeps reading the first, stale contents forever. This is vLLM issue #57224, first seen on an H100 in a TDX guest. Until PRs #57415 or #57404 land, one variable switches back to the older runner with explicit copies:

$ docker run --gpus all -e VLLM_USE_V2_MODEL_RUNNER=0 -p 8000:8000 vllm/vllm-openai:v0.32.0 Qwen/Qwen3-8B
$ curl -s localhost:8000/v1/completions -d '{"model":"Qwen/Qwen3-8B","prompt":"The capital of France is","max_tokens":16,"temperature":0}'
" Paris. The capital of France is also the capital of the French Republic."

07 · SERVINGSix models in CC mode

With the runner fixed, how fast does a confidential GPU serve? vLLM 0.32.0 with the V1 runner and CUDA graphs, and SGLang 0.5.22 for comparison, each with 85% of the memory and a 4K context; vllm bench serve from a sidecar container with random 1,024-token prompts, 512-token answers and --ignore-eos, from 1 to 64 requests at once. Six models that people actually run on a 96 GB card: Qwen3 at 0.6B, 8B and 14B, Phi-4-mini, DeepSeek-R1-Distill-Llama-8B and gpt-oss-20b.

Output tokens per second at 1, 8, 32 and 64 concurrent requests. † Single-request decode rate, measured after the one-time first-request pause.

• The engine does not matter. vLLM and SGLang match within 1% on Qwen3-8B at every load. Both are bound by the GPU, so the bridge tax is already hidden in their per-token times.
• One user barely feels CC. Qwen3-8B decodes a token in 12.5 ms, and about 10 of those are simply reading the weights from GPU memory.
• Tuning works as usual. CUDA graphs and async scheduling help as much as in the clear, and the queue, not the bridge, sets time to first token.
• The first vLLM request is slow. It took 23 s on four of six models, then 0.1 s. We could not tie it to CC; warm the engine before opening the port.

Startup is the other story: one to three minutes to ready, as in the clear, but with nothing to skip it.

Weights are not the bottleneck: from the page cache 15 GiB reached the GPU in about a second. Compile and graph capture take the rest, exactly as without CC. The difference is that a CC guest cannot restore a snapshot to skip them (section 09).

Bottom line. Throughput is not the main price of CC: a batched engine runs close to what the GPU allows. Published comparisons put stock vLLM at about 10% slower on this card and 26% on a B300, and NVIDIA’s patch (#52226) closes most of that gap. The bigger price is paid at start-up.

08 · B300The same chain on eight B300s

The same provider then gave us an hour on eight NVIDIA B300s with encrypted NVLink (NVIDIA calls it NVLE), the same driver and an AMD Turin SEV-SNP guest. Two questions one card could not answer: does a faster bus help, and what does encryption do to NVLink? Attestation came first, exactly as designed: CUDA refuses to start until the GPUs are verified and set ready.

$ nvidia-smi conf-compute -grs
Confidential Compute GPUs Ready state: not-ready
$ ./probe                     # can CUDA start?
cudaGetDeviceCount: system not yet initialized (n=-1)
$ python3 -m verifier.cc_admin --user_mode   # attest all 8 GPUs first
GPU Attestation is Successful.
$ nvidia-smi conf-compute -srs 1   # only now: let CUDA in
$ ./probe
cudaGetDeviceCount: no error (n=8)

• A faster bus does not help. PCIe 6 doubles the link, yet the encrypted bridge stays at about 18 GB/s. The limit is encryption on the CPU side, not the wire.
• Encrypted NVLink is fast. GPUs talk to each other at hundreds of GB/s, so tensor parallelism stays inside the enclave at full speed. NVLE encrypts the link; HBM stays unencrypted and isolated, as on the RTX card. Only traffic to the CPU crawls.
• Same bugs, same fixes. vLLM’s default runner returned garbage again; the V1 runner, SGLang and an 8-GPU tensor-parallel vLLM all answered “Paris”. cuda-checkpoint refused again.
• Memory is wiped between tenants. A process filled 260 GiB of one GPU with a marker and exited; the next process found none of it. With VRAM unencrypted, this wipe is what keeps one tenant’s weights from the next.

Output tokens per second at 1, 8, 32 and 64 concurrent requests, 1K prompt and 512-token answer. † Single-request decode rate.

• A faster GPU makes the loop visible. One B300 decodes Qwen3-8B 2.7× faster than the RTX card. Here SGLang leads vLLM by 5–13%: when the GPU step is this short, CPU-side work and the bridge start to show.
• Big models work. Qwen3-235B in FP8 across eight GPUs decodes at 9.6 ms per token, reaches 3,519 tok/s at 128 requests, and starts answering a 32K-token prompt in under a second.
• Start-up is slow, but not because of CC. The weight loader, not the bridge, sets the pace: the 235B model was ready in 593 s from disk and 379 s with its weights already in memory.

What B300 adds. Scale does not change the picture. GPUs and NVLink run at full speed inside the enclave, the host bridge stays at about 18 GB/s whatever the bus, and the engines need the same fixes. The faster the GPU, the more the CPU-side loop costs, so CC-aware engines matter more on B300 than on a workstation card.

09 · LIMITSWhat CC mode will not do

$ cuda-checkpoint --action lock --pid $(pgrep -f EngineCore)
CUDA_ERROR_NOT_SUPPORTED: operation not supported
$ cuda-checkpoint --action lock --pid $(pgrep victim)      # a plain 1 GiB CUDA process
CUDA_ERROR_NOT_SUPPORTED: operation not supported

What it means for warm starts. warmctl restores an engine in seconds from a snapshot of its CPU and GPU state. In a CC guest the GPU checkpoint API refuses every CUDA process, so a confidential replica pays the full cold start each time. Until NVIDIA supports checkpoints in CC mode, the only levers are kernel caches and fast weight loading.

10 · TAKEAWAYSWhat total confidentiality costs

Confidential computing closes the last gap, data in use, and on Blackwell the hardware delivers. This is the bill we measured.

Worth it, with eyes open. Know what you are buying: the CPU encrypts RAM, while the GPU encrypts only its links and guards VRAM by isolation and wiping, which is exactly why it keeps full speed. For a bank or a healthcare company that must keep prompts and weights away from the provider, the price is low where it matters most: the GPU runs at full speed and a batched engine loses a few percent. The real costs are operational: slow, always-cold starts and engines that are not yet CC-aware by default.

Before you ship

• Verify both reports yourself, with your own nonce, against AMD and NVIDIA; reject DevTools mode and a debug policy.
• Ask the provider for reference images (firmware, kernel, initrd), or the launch measurement is a number you cannot interpret.
• Pin the engine: SGLang as is, or vLLM with VLLM_USE_V2_MODEL_RUNNER=0 until #57415 lands.
• Run a canary with a known answer after every start, before the port opens.

Every script and raw output from these runs is in the post’s data folder.

Sources

• October 2026 incidents: Railway, LiteLLM, OpenCode, Zeabur, PingCAP (ThreatsDay).
• vllm-project/vllm issue #57224: silent garbage output in CC mode, V2 runner UVA views; PRs #57415 and #57404; issue #55428 on cudaHostRegister.
• NVIDIA, Decode optimizations for NVIDIA confidential computing (vLLM PR #52226) and the CPU↔GPU bridge microbenchmark (PR #60338).
• The Serialized Bridge: Understanding and Recovering LLM Serving Performance under Blackwell GPU Confidential Computing (arXiv 2606.23969).
• NVIDIA, Confidential Compute on NVIDIA Hopper H100 (whitepaper) and Hardware-rooted AI security that won’t slow you down (Blackwell, RTX PRO 6000).
• NVIDIA nvtrust (local GPU verifier) and the attestation SDK; AMD SEV-SNP and the KDS at kdsintf.amd.com.
• KServe 0.20, confidential model serving; Blueprint, Bootstrap, and Bridge: a security look at NVIDIA GPU confidential computing.