Decoder-Only Models: The GPT Family and Autoregressive Generation

If you have used ChatGPT, Claude, or any modern AI chatbot, you have interacted with a decoder-only Transformer. This architecture -- which generates text one token at a time, each conditioned on all previous tokens -- has become the dominant paradigm for large language models. From GPT-1's modest 117 million parameters to the trillion-parameter behemoths of today, decoder-only models have driven the most dramatic advances in AI.

How Autoregressive Generation Works

Decoder-only models are fundamentally autoregressive: they predict one token at a time, feeding each prediction back as input for the next. The process is simple but powerful:

1. The model receives an input sequence (the prompt).
2. It predicts the probability distribution over possible next tokens.
3. A token is sampled from this distribution.
4. The sampled token is appended to the input, and the process repeats from step 2.

The key architectural feature enabling this is causal attention (also called masked self-attention). Each token can only attend to tokens at earlier positions, preventing the model from "seeing the future." This masking is implemented by applying a triangular mask to the attention scores, setting future positions to negative infinity before the softmax operation.

"Autoregressive generation is deceptively simple: predict one word at a time. Yet from this simple loop emerges the ability to write poetry, solve math problems, and explain quantum physics."

The GPT Evolution

GPT-1 (2018): The Proof of Concept

OpenAI's GPT-1 had just 117 million parameters and 12 Transformer layers. Its key contribution was showing that unsupervised pre-training on a large text corpus followed by supervised fine-tuning could achieve state-of-the-art results on multiple NLP tasks. While less celebrated than BERT at the time, GPT-1 established the decoder-only paradigm.

GPT-2 (2019): Scale Reveals Capabilities

GPT-2 scaled up to 1.5 billion parameters and demonstrated something unexpected: emergent abilities. Without any fine-tuning, GPT-2 could generate coherent articles, translate between languages, and answer questions through zero-shot and few-shot prompting. The model was initially withheld due to concerns about misuse -- a harbinger of the safety debates to come.

GPT-3 (2020): The Few-Shot Revolution

At 175 billion parameters, GPT-3 proved that scaling could unlock remarkable capabilities. Its defining feature was in-context learning: by providing a few examples in the prompt, GPT-3 could perform new tasks without any parameter updates. This eliminated the need for task-specific fine-tuning in many cases and demonstrated the power of large-scale language models.

GPT-4 (2023): The Frontier

GPT-4 represented a massive leap in capability, with strong performance on professional exams, complex reasoning tasks, and multimodal understanding. While its architecture details remain proprietary, it demonstrated that the decoder-only paradigm could scale to produce systems approaching human-level performance on many cognitive tasks.

Why Decoder-Only Won

Despite BERT's early dominance for understanding tasks, decoder-only models have become the standard for general-purpose AI. Several factors explain this:

• Generality: Decoder models can perform any text task by framing it as text generation. Classification becomes generating a class label, extraction becomes generating the extracted text, and translation becomes generating the target language. This universality eliminates the need for task-specific architectures.
• Scaling behavior: Decoder-only models exhibit more predictable and favorable scaling laws than encoder models. Their performance improves smoothly with increased scale.
• Training simplicity: The next-token prediction objective is simpler than masked language modeling, making training more stable at large scales.
• In-context learning: Decoder models naturally support few-shot prompting, allowing them to perform new tasks without fine-tuning.
• Generation capability: By definition, decoder models generate text, which is the most versatile output format for interacting with humans.

Modern Decoder Architectures

Today's decoder-only models incorporate numerous improvements over the original GPT architecture:

• Pre-normalization: Applying layer normalization before (rather than after) each sub-layer, using RMSNorm instead of LayerNorm.
• Rotary Position Embeddings (RoPE): Encoding relative position information through rotation matrices applied to queries and keys.
• Grouped Query Attention (GQA): Sharing key-value heads across multiple query heads to reduce memory usage during inference.
• SwiGLU activation: Replacing the standard ReLU activation in the feed-forward network with the SwiGLU gated activation function.
• Flash Attention: An IO-aware attention algorithm that reduces memory usage and improves speed through careful memory management.

The KV Cache: Making Generation Practical

A critical optimization for decoder models is the key-value (KV) cache. During autoregressive generation, the attention computation for token N requires the keys and values from all previous tokens 1 through N-1. Without caching, you would recompute these values at every step, wasting enormous computation.

The KV cache stores the key and value tensors from previous positions, so each new token only needs to compute its own query, key, and value and attend over the cached keys and values. This reduces the per-token computation from O(N) to O(1), making long-sequence generation practical. However, the KV cache also consumes significant GPU memory, which is why techniques like GQA and multi-query attention have become important.