Phase 17 - Lesson 07

TensorRT-LLM on Blackwell with FP8 and NVFP4

TensorRT-LLM is NVIDIA-only but it wins on Blackwell. On GB200 NVL72 with Dynamo orchestration, SemiAnalysis InferenceX measured $0.012 per million tokens on a 120B model in Q1-Q2 2026, against $0.09/M on H100 + vLLM — a 7x economic gap. The stack is three floating-point regimes compounded: FP8 stays critical for KV cache and attention kernels because it has the dynamic range they need; NVFP4 (4-bit microscaling) handles weights and activations; multi-token prediction (MTP) and disaggregated prefill/decode add another 2-3x on top. Day-0 model support loads FP4 weights directly without post-training conversion. The catch for 2026 engineering teams: TRT-LLM is a closed NVIDIA stack, so adopting it trades portability for throughput. Run the math on your mix of models and hardware before committing.

Type: Learn Languages: Python (stdlib, toy FP8/NVFP4 memory and cost calculator) Prerequisites: Phase 17 · 04 (vLLM Serving Internals), Phase 10 · 13 (Quantization) Time: ~75 minutes

Learning Objectives

• Explain why FP8 stays critical for KV cache and attention even when weights are in NVFP4.
• Compute the HBM footprint of a frontier model under BF16, FP8, and NVFP4 and reason about where the savings come from.
• Name the Blackwell-specific features TRT-LLM exploits (day-0 FP4, MTP, disaggregated serving, all-to-all primitives).
• Decide when TRT-LLM's NVIDIA-lock is worth the 7x cost gap vs vLLM on Hopper.

The Problem

The frontier of inference economics in 2026 is "how many tokens per dollar". The answer depends on four stacked choices: hardware generation (Hopper H100/H200 vs Blackwell B200/GB200), precision (BF16 → FP8 → NVFP4), serving engine (vLLM vs SGLang vs TRT-LLM), and orchestration (plain vs disaggregated vs Dynamo).

On Hopper with vLLM, a 120B MoE runs at ~$0.09 per million tokens. On Blackwell with TRT-LLM + Dynamo, the same model runs at ~$0.012 — 7x cheaper. Some of that gap is hardware (Blackwell is 11-15x per-GPU LLM throughput vs Hopper). Some is the stack: FP4 weights, MTP draft, disaggregated prefill/decode, and NVLink 5 all-to-all for MoE expert communication.

You cannot replicate this outside NVIDIA's stack. That is the tradeoff — portability for economics. Understanding which stack choices give which share of the gap is the point of this lesson.

The Concept

Why FP8 is still the floor for KV cache

A common mistake in 2026: assuming NVFP4 applies everywhere. It does not. KV cache needs FP8 (8-bit floating point) because it stores attention keys and values that span a wide dynamic range. Quantizing KV to FP4 causes catastrophic accuracy loss — the tail of the distribution drops off and attention scores collapse. FP8's exponent bits give KV cache the range it needs.

NVFP4 (2025-2026) applies to weights and activations. Microscaling: each block of weights has its own scale factor so small blocks can span different dynamic ranges without per-tensor scale loss. For activations, FP4 holds up because activations are small-range within a layer.

The typical Blackwell config:

• Weights: NVFP4 (4-bit microscaling).
• Activations: NVFP4.
• KV cache: FP8.
• Attention accumulator: FP32 (softmax stability).

The Blackwell-specific primitives TRT-LLM uses

• Day-0 FP4 weights: model providers ship FP4 weights directly; TRT-LLM loads without post-training conversion. No AWQ / GPTQ step for FP4.
• Multi-token prediction (MTP): same idea as EAGLE (Phase 17 · 05) but integrated into the TRT-LLM build.
• Disaggregated serving: prefill and decode on separate GPU pools, KV cache transferred over NVLink or InfiniBand. Same idea as Dynamo (Phase 17 · 20).
• All-to-all communication primitives: NVLink 5 cut MoE expert communication latency by 3x vs Hopper. TRT-LLM's MoE kernels are tuned for this.
• NVFP4 + MXFP8 microscaling: hardware-accelerated scale-factor handling on Blackwell Tensor Cores.

The numbers you should memorize

• HGX B200 at $0.02/M tokens on GPT-OSS-120B via TRT-LLM.
• GB200 NVL72 at $0.012/M tokens via Dynamo (orchestrating TRT-LLM).
• H100 + vLLM ≈ $0.09/M tokens on comparable workload.
• 2.8x throughput gain in three months of TRT-LLM updates (2026).
• 11-15x per-GPU LLM throughput, Blackwell vs Hopper.
• MLPerf Inference v6.0 (April 2026): Blackwell dominates every submitted task.

What FP4 actually costs in quality

NVFP4 is aggressive. On reasoning-heavy workloads (chain-of-thought, math, code-gen with long context), FP4 weights degrade visibly. Per-block calibration mitigates but does not eliminate. Teams shipping reasoning models often use FP8 weights + FP4 activations as a compromise, or stick to H200 with FP8 throughout.

The rule: always validate task quality on your eval set before committing to NVFP4 weights.

Why this is an NVIDIA-lock decision

TRT-LLM is C++ + CUDA + closed-source kernels. Models need to be compiled for a specific GPU SKU. No AMD, no Intel, no ARM. If your infra strategy is multi-vendor, TRT-LLM is a non-starter for the TRT-LLM-served tier — you can still serve from vLLM on mixed hardware. If you are NVIDIA-only, the 7x gap pays for the lock.

2026 practical recipe

For a

Key Terms

Further Reading

• NVIDIA — Blackwell Ultra MLPerf Inference v6.0 — April 2026 MLPerf results.
• NVIDIA — MoE Inference on Blackwell — NVLink 5 all-to-all and MoE kernels.
• TensorRT-LLM Overview — official engine documentation.
• NVIDIA — Introducing Dynamo — disaggregated orchestration above TRT-LLM.
• MLPerf Inference — the benchmark suite that publishes Blackwell numbers.