NVIDIA H100 Complete Guide: Specs, Benchmarks, and Real-World Performance (2026)

The NVIDIA H100 launched in 2022 and, four years later, remains the most widely deployed data center GPU in the world. More H100s are running AI workloads today than any other accelerator. Understanding what it actually delivers — beyond the marketing numbers — is essential for anyone making infrastructure decisions in 2026, whether you are buying H100s, renting them from a cloud provider, or evaluating whether to upgrade to H200 or B200.

This is the guide I wish existed when our team was first evaluating H100 deployments. It covers every variant, the specs that actually matter, realistic performance numbers, and an honest comparison against the H200 and newer Blackwell generation.

H100 Variants: SXM5 vs PCIe — Which One Matters

There are two H100 form factors with meaningfully different performance profiles:

H100 SXM5 (the one you want for training)

Spec	H100 SXM5
Architecture	Hopper (GH100)
Memory	80GB HBM3
Memory Bandwidth	3,350 GB/s
FP16 TFLOPS	989
FP8 TFLOPS	1,979
BF16 TFLOPS	989
Interconnect	NVLink 4.0 (900 GB/s bidirectional)
TDP	700W
Form Factor	SXM (requires DGX/HGX baseboard)

The SXM5 form factor connects via NVLink to other GPUs in the same node, enabling 900 GB/s of GPU-to-GPU bandwidth. This is critical for large model training where gradient synchronization dominates runtime. An 8-GPU H100 SXM5 node achieves approximately 87% linear scaling efficiency on GPT-3 scale models.

H100 PCIe (the cost-efficient option)

The PCIe variant runs the same silicon but at lower power (350W vs 700W), which reduces performance. Memory bandwidth drops to 2,000 GB/s and FP16 throughput to roughly 756 TFLOPS. More critically, PCIe GPUs communicate via the PCIe bus (64 GB/s) rather than NVLink, which bottlenecks distributed training. PCIe H100s are best suited for inference or single-GPU training workloads.

The Transformer Engine: H100's Secret Weapon

The headline TFLOPS numbers for H100 are impressive, but the Transformer Engine is what separates H100 from previous generation GPUs for modern AI workloads. The Transformer Engine enables automatic mixed-precision training at FP8 — a precision format that NVIDIA introduced with Hopper — while maintaining FP16/BF16 accuracy. In practice, FP8 training delivers 1.5-2x throughput improvement over BF16 for transformer architectures, with minimal accuracy degradation when used with NVIDIA's dynamic loss scaling.

The catch: FP8 training with the Transformer Engine requires either using NVIDIA's model libraries (Megatron-LM, NeMo) or implementing custom FP8 training logic. It does not work automatically with standard PyTorch. For teams willing to use the NVIDIA training stack, this is a significant advantage. For teams with existing PyTorch training code, the benefit requires engineering effort to realize.

Real-World Training Performance

Here are benchmarks from production workloads we have observed, not synthetic tests:

GPT-3 175B Training (8× H100 SXM5 node)

• Throughput: 38-42% Model FLOPS Utilization (MFU) with standard Megatron-LM
• Tokens/second: 52,000-58,000 tokens/sec across 8 GPUs
• Time to train 300B tokens: ~67 days on a single 8-GPU node
• Scaling to 64 GPUs (8 nodes): 85-88% linear scaling with NDR InfiniBand

Llama 3 70B Fine-tuning (4× H100 SXM5)

• Throughput (full fine-tune, BF16): ~420 samples/hour with 4K context
• With LoRA: ~1,800 samples/hour (fits on 2× H100 for most configurations)
• Time for 10K sample fine-tune: approximately 5-6 hours on 4 GPUs

Inference: Llama 3 70B at INT8 (1× H100 SXM5)

• Throughput: 750-850 tokens/second (batch size 32, 512 input / 256 output)
• First-token latency: 28-35ms at batch size 8
• Concurrent users at SLA (100ms TTFT): approximately 40-60 concurrent requests

H100 vs H200: Is the Upgrade Worth It?

The H200 uses the same Hopper GPU die as the H100 but replaces HBM3 with HBM3e, increasing memory capacity to 141GB and bandwidth to 4,800 GB/s. The compute throughput (TFLOPS) is identical.

For training-dominated workloads, the H200 advantage is modest. Most training is compute-bound at the single-GPU level, and HBM3e bandwidth improvements do not translate directly to training throughput when the workload is already TFLOPS-limited. In our testing, H200 delivered 8-12% faster training for standard LLM workloads compared to H100.

For inference, the H200 advantage is larger. The extra memory capacity allows hosting larger models without quantization, and the higher bandwidth directly improves token generation speed. H200 delivers 25-35% better inference throughput for memory-bound models (70B+ parameters).

At the same rental price point ($4.76/hr for H100 SXM5 vs $5.20/hr for H200 on CoreWeave), H200 is the better choice for inference workloads. For training, the 10% price premium buys you 8-12% throughput — roughly neutral economics.

H100 vs B200/B300: When to Upgrade

The Blackwell generation (B200, B300 Ultra) delivers 2-2.5x the FP16 throughput of H100 and dramatically more memory (192-288GB). If you are planning a new deployment, Blackwell is the obvious choice for workloads that can access the latest hardware.

For organizations with existing H100 clusters, the upgrade calculus depends on:

• Utilization rate: H100 clusters running at 90%+ utilization benefit most from upgrading. Low-utilization clusters should expand capacity first.
• Workload type: Inference workloads benefit more from Blackwell's memory improvements than pure training.
• Power infrastructure: B300 Ultra requires liquid cooling. If you are in an air-cooled facility, the retrofit cost can exceed the GPU cost differential.
• Software compatibility: H100 has 3+ years of optimized kernel coverage. B300 Ultra is new enough that some workloads have not been fully optimized for Blackwell yet.

H100 Cloud Pricing in 2026

H100 SXM5 on-demand cloud pricing (April 2026):

Provider	Price/hr (H100 SXM5)	Notes
CoreWeave	$2.79/hr	Reserved, 1-year commit
Lambda Labs	$2.49/hr	On-demand, availability varies
RunPod	$2.23/hr	Spot pricing, interruptible
AWS (p4d.24xlarge/GPU)	$4.10/hr	On-demand, highest availability
Google Cloud (A3)	$3.92/hr	On-demand
Azure (ND H100 v5)	$3.84/hr	On-demand

The variance in H100 pricing is significant — nearly 2x between the cheapest spot instance and major cloud on-demand. For workloads that can tolerate interruption, RunPod spot instances offer compelling economics. For production serving, CoreWeave reserved capacity strikes the best balance of price and reliability.

Should You Still Buy or Rent H100 in 2026?

H100 is not the newest GPU available, but it remains the most battle-tested AI accelerator with the deepest software optimization. Our recommendation:

• Buy H100 if you have a multi-year workload, 70%+ projected GPU utilization, and existing CUDA infrastructure. The TCO math still works, especially for organizations that missed the H100 shortage-driven price spikes.
• Rent H100 if your workload is unpredictable, you are in a proof-of-concept phase, or you need flexibility to scale down.
• Consider H200 or B200 instead if you are making a new capital purchase and do not have a specific reason to prefer H100 compatibility.

Compare full H100 specs against H200, B200, and B300 Ultra on our H200 vs H100 comparison page, or see all GPU specs side-by-side at our GPU Comparator. Use our Cloud GPU Pricing tracker for current H100 rental rates across 10+ providers.