AI Hardware & Infrastructure

1. The GPU Revolution

Why GPUs Dominate AI Training

Graphics Processing Units (GPUs) were originally designed for rendering pixels in video games, but their architecture turned out to be perfectly suited for the matrix multiplications that underpin deep learning. While a CPU excels at sequential, complex tasks with a handful of powerful cores, a GPU distributes thousands of simpler computations across thousands of cores simultaneously.

Modern deep learning involves multiplying enormous matrices — a forward pass through a transformer model with billions of parameters requires trillions of floating-point operations. GPUs handle these through massive parallelism, high memory bandwidth, and specialized tensor cores designed specifically for mixed-precision matrix math.

NVIDIA's Dominance in AI

NVIDIA has established an unassailable lead in AI compute, not just through hardware but through its CUDA software ecosystem, which has become the de facto standard for AI development. Virtually every major deep learning framework — PyTorch, TensorFlow, JAX — is optimized for CUDA, creating a powerful moat.

NVIDIA A100 (Ampere, 2020)

The workhorse that trained most of the foundation models we use today. The A100 introduced third-generation Tensor Cores with TF32 precision and multi-instance GPU (MIG) technology, allowing a single GPU to be partitioned into up to seven independent instances. Available in 40 GB and 80 GB HBM2e variants, the A100 was the first GPU purpose-built for the AI data center era.

NVIDIA H100 (Hopper, 2023)

A generational leap that roughly tripled AI training performance over the A100. The H100 introduced the Transformer Engine with FP8 precision, dramatically accelerating transformer-based workloads. Its fourth-generation NVLink provides 900 GB/s of GPU-to-GPU bandwidth, and it became the most sought-after chip in computing history, with wait times stretching to months.

NVIDIA H200 (Hopper, 2024)

An enhanced H100 with 141 GB of HBM3e memory (up from 80 GB HBM3), providing 4.8 TB/s of memory bandwidth. The H200 is particularly impactful for large language model inference, where memory capacity determines the maximum model size that can fit on a single GPU.

NVIDIA B100/B200 (Blackwell, 2024-2025)

NVIDIA's Blackwell architecture represents another major leap. The B200 features a dual-die design with 192 GB of HBM3e, delivering up to 2.5x the training performance and 5x the inference performance of the H100. The GB200 "superchip" pairs two B200 GPUs with a Grace CPU via NVLink-C2C, forming the building block of next-generation AI supercomputers.

AMD MI300X: The Challenger

AMD's Instinct MI300X is the most credible alternative to NVIDIA's data center GPUs. With 192 GB of HBM3 memory — significantly more than the H100's 80 GB — the MI300X is particularly attractive for large model inference where memory capacity is the bottleneck. AMD's ROCm software stack has matured considerably, with PyTorch support improving, though it still trails CUDA in ecosystem breadth.

GPU Specifications Comparison

2. AI Accelerators & Custom Silicon

While GPUs are general-purpose parallel processors adapted for AI, a growing number of companies are designing application-specific integrated circuits (ASICs) purpose-built for machine learning workloads. These custom chips sacrifice flexibility for efficiency, often delivering better performance-per-watt for specific AI tasks.

3. Training Infrastructure

Training a frontier AI model is one of the most demanding engineering challenges in computing. It requires orchestrating thousands of GPUs across a data center, moving petabytes of data, and sustaining computation for weeks or months without failure.

Data Center Design for AI

AI data centers differ fundamentally from traditional cloud data centers in three critical dimensions:

• Power density: A single rack of 8 NVIDIA B200 GPUs can draw over 10 kW, compared to 5-7 kW for a typical cloud server rack. AI clusters of 10,000+ GPUs demand 50-100+ MW of power — equivalent to a small city.
• Cooling: At these power densities, traditional air cooling is insufficient. Modern AI data centers increasingly use direct liquid cooling (cold plates on each GPU) or immersion cooling (submerging entire servers in dielectric fluid).
• Networking: AI training requires constant communication between GPUs. The network fabric must support ultra-low latency, high-bandwidth all-to-all communication patterns that differ from typical web service traffic.

GPU Clusters & Interconnects

Modern training clusters are built in a hierarchy of interconnects, each optimized for different communication distances:

• NVLink: NVIDIA's proprietary GPU-to-GPU interconnect. NVLink 4 (Hopper) provides 900 GB/s bidirectional bandwidth. NVLink 5 (Blackwell) doubles this to 1,800 GB/s. Critical for tensor parallelism where individual layers are split across GPUs.
• NVSwitch: Enables all-to-all communication within a node (typically 8 GPUs) at full NVLink bandwidth, creating a unified memory space.
• InfiniBand (NDR/XDR): The dominant node-to-node fabric for AI clusters, providing 400 Gb/s (NDR) or 800 Gb/s (XDR) per port with extremely low latency. Essential for data parallelism and pipeline parallelism across nodes.
• Ethernet alternatives: Ultra Ethernet Consortium (UEC) is developing AI-optimized Ethernet to challenge InfiniBand's dominance, with support from AMD, Intel, Google, Meta, and Microsoft.

Distributed Training Strategies

No single GPU can train a frontier model alone. Distributed training splits the workload using several complementary strategies:

• Data parallelism: Each GPU holds a complete copy of the model but processes different batches of data. Gradients are synchronized across all GPUs after each step. Simple to implement but requires the model to fit in a single GPU's memory.
• Tensor parallelism: Individual layers are split across multiple GPUs within a node, using NVLink for the frequent communication required. Best for very wide layers in large transformers.
• Pipeline parallelism: Different layers of the model are placed on different GPUs. Data flows sequentially through the pipeline. Reduces memory requirements per GPU but introduces "pipeline bubbles" that reduce efficiency.
• Expert parallelism: Used for Mixture-of-Experts (MoE) models, where different experts reside on different GPUs and only a subset are activated for each input token.
• ZeRO (Zero Redundancy Optimizer): Partitions optimizer states, gradients, and parameters across GPUs to eliminate memory redundancy while maintaining data parallelism semantics. Developed by Microsoft DeepSpeed.

The Cost of Training Frontier Models

4. Inference Infrastructure

While training captures the headlines, inference — running trained models to generate predictions — accounts for 80-90% of total AI compute spending in production. Optimizing inference is where hardware meets real-world economics.

Latency vs. Throughput Tradeoffs

Inference workloads face a fundamental tension:

• Latency-sensitive: Chatbots, real-time translation, and interactive AI assistants need fast time-to-first-token (TTFT) and high tokens-per-second. Users notice delays beyond 200ms for the first token.
• Throughput-sensitive: Batch processing, document analysis, and offline embedding generation prioritize processing the most requests per dollar, even if individual requests take longer.

Hardware and software choices differ dramatically depending on which dimension you optimize for. Groq's LPU, for instance, sacrifices cost-efficiency for extraordinary latency, while batched GPU inference maximizes throughput at the expense of per-request latency.

Quantization for Deployment

Quantization reduces model weights from high-precision floating-point numbers to lower-precision integers, dramatically reducing memory usage and compute requirements:

Edge AI vs. Cloud Inference

Inference Optimization Techniques

Modern inference engines employ multiple strategies to maximize performance:

• Continuous batching: Instead of waiting for a fixed batch to fill, the server dynamically adds and removes requests from the batch as they arrive and complete. Implemented in vLLM, TensorRT-LLM, and TGI.
• KV cache management: During autoregressive generation, the key-value pairs from previous tokens are cached to avoid redundant computation. PagedAttention (vLLM) manages KV cache like virtual memory, reducing waste from 60-80% to near zero.
• Speculative decoding: A smaller "draft" model generates candidate tokens quickly, then the larger model verifies them in parallel. If most drafts are accepted, you get the quality of the large model at speeds closer to the small model. Can achieve 2-3x speedups.
• Flash Attention: A memory-efficient attention algorithm that reduces GPU memory usage from O(n^2) to O(n) and accelerates attention computation by minimizing reads/writes to slow GPU HBM. Now standard in all major frameworks.
• Model distillation: Training a smaller "student" model to mimic a larger "teacher" model's outputs, creating a more efficient model that retains most of the original's capability.

5. Cloud AI Platforms

The three major cloud providers each offer comprehensive AI/ML platforms, though they differ in strengths, GPU availability, and ecosystem integration.

6. Future of AI Hardware

As AI models continue to scale, the semiconductor industry faces fundamental physical and economic challenges. Several emerging technologies aim to break through current limitations.

Optical Computing

Photonic processors use light instead of electrons for matrix multiplications, potentially achieving orders-of-magnitude improvements in energy efficiency. Companies like Lightmatter and Luminous Computing are developing photonic interconnects and compute elements. Optical computing is particularly promising for inference workloads where the same matrix operations are repeated billions of times.

Neuromorphic Chips

Inspired by biological neural networks, neuromorphic processors like Intel Loihi 2 and IBM NorthPole use event-driven, spike-based computation rather than clock-driven arithmetic. They excel at tasks involving temporal patterns and sparse data, consuming a fraction of the power of conventional chips. While not yet competitive for transformer-based LLMs, they show promise for edge AI, robotics, and sensor processing.

Quantum Machine Learning

Quantum computers could theoretically accelerate specific ML algorithms exponentially, particularly for optimization problems, molecular simulation, and certain types of feature mapping. However, current noisy intermediate-scale quantum (NISQ) devices remain far from practical utility for mainstream AI workloads. The timeline for "quantum advantage" in AI remains uncertain — likely a decade or more for meaningful impact.

The Energy Efficiency Challenge

AI's energy consumption is the industry's most pressing sustainability concern:

• Training a single large model can emit as much CO2 as five cars over their lifetimes.
• AI data centers are projected to consume 3-4% of global electricity by 2030, up from roughly 1-2% today.
• The demand is driving investment in nuclear power (SMRs), renewable energy PPAs, and next-generation cooling technology.
• Algorithmic efficiency gains (better architectures, sparse models, efficient training recipes) are equally important as hardware improvements.