Transformer Architecture: Attention Is All You Need Explained

In 2017, a team at Google published a paper titled "Attention Is All You Need" that would reshape the entire field of artificial intelligence. The Transformer architecture they introduced replaced the recurrent neural networks that had dominated sequence modeling for years with a radically different approach based entirely on attention mechanisms. Today, Transformers power virtually every state-of-the-art AI system, from GPT-4 and Claude to DALL-E and AlphaFold. Understanding how they work is essential for anyone working in AI.

Before Transformers: The RNN Era

To appreciate why Transformers were revolutionary, we need to understand what came before them. Recurrent Neural Networks (RNNs) and their variants (LSTMs, GRUs) processed sequences one token at a time, maintaining a hidden state that carried information forward. This sequential processing created two fundamental problems:

• The vanishing gradient problem: Information from early in a sequence was progressively diluted as it passed through many sequential steps, making it difficult to learn long-range dependencies.
• The parallelization bottleneck: Because each step depended on the output of the previous step, RNNs could not be parallelized across sequence positions. This made training painfully slow on modern GPU hardware designed for parallel computation.

Attention mechanisms were originally introduced as an addition to RNNs, allowing the model to "look back" at earlier positions directly rather than relying on the hidden state. The Transformer's radical insight was: what if attention is not an add-on but the entire architecture?

The Self-Attention Mechanism

Self-attention is the core operation of the Transformer. It allows every position in a sequence to directly attend to every other position, computing a weighted combination of all values based on their relevance to the current position.

The mechanism works through three learned projections of the input:

• Queries (Q): What each position is looking for.
• Keys (K): What each position offers as a match.
• Values (V): The actual information each position provides.

The attention score between any two positions is computed as the dot product of the query from one position with the key from another, scaled by the square root of the dimension. These scores are passed through a softmax to create attention weights, which are then used to weight the values. The formula is elegantly compact:

Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) V

This simple operation has remarkable properties. It creates direct connections between all positions regardless of distance, solving the long-range dependency problem. And because all positions can be computed simultaneously, it is fully parallelizable.

"The Transformer's key innovation is replacing sequential computation with parallel attention, allowing direct information flow between any two positions in a sequence regardless of distance."

Multi-Head Attention

A single attention operation captures one type of relationship between positions. Multi-head attention runs multiple attention operations in parallel, each with different learned projections. This allows the model to simultaneously attend to information from different representation subspaces at different positions.

For example, one head might learn to attend to syntactic relationships (subject-verb agreement), another to semantic similarity, and a third to positional proximity. The outputs of all heads are concatenated and projected through a final linear layer.

In the original Transformer, 8 attention heads were used, each with a dimension of 64, together matching the model dimension of 512. Modern large models use many more heads -- GPT-3 uses 96 heads, and larger models use even more.

Positional Encoding

Because self-attention treats the input as a set rather than a sequence -- it has no inherent notion of order -- the Transformer needs an explicit mechanism to encode positional information. The original paper used sinusoidal positional encodings: fixed mathematical functions of different frequencies added to the input embeddings.

Modern Transformers have explored various alternatives:

• Learned positional embeddings: Trainable vectors for each position, used in models like GPT-2.
• Rotary Position Embeddings (RoPE): Encode relative positions by rotating the query and key vectors, used in LLaMA and many modern models.
• ALiBi (Attention with Linear Biases): Add a linear bias to attention scores based on position distance, used in BLOOM.

The Feed-Forward Network

After each attention layer, the Transformer applies a position-wise feed-forward network (FFN) -- the same two-layer neural network applied independently to each position. This might seem simple, but research has shown that the FFN layers store a significant amount of the model's factual knowledge.

The standard FFN consists of two linear transformations with a non-linear activation in between. Modern variants use SwiGLU or GeGLU activations, which have been shown to improve performance. The FFN typically expands the dimension by 4x before projecting back down, giving the network more capacity to process information.

The Complete Architecture

The full Transformer combines these components into a stack of identical layers, each containing:

1. Multi-head self-attention with residual connection and layer normalization.
2. Feed-forward network with residual connection and layer normalization.

The original Transformer had both an encoder (which processes the full input bidirectionally) and a decoder (which generates output autoregressively). This encoder-decoder structure was designed for sequence-to-sequence tasks like machine translation.

Since then, three main variants have emerged:

• Encoder-only: BERT and its successors, optimized for understanding tasks.
• Decoder-only: GPT and most modern LLMs, optimized for generation.
• Encoder-decoder: T5, BART, optimized for tasks that require both understanding and generation.

Why Transformers Won

The Transformer's dominance is not an accident. Several properties make it uniquely suited to modern AI:

• Parallelism: Every operation can be parallelized across sequence positions, making Transformers extraordinarily efficient on GPU hardware.
• Scalability: Transformers scale predictably with more parameters, data, and compute, enabling the creation of increasingly capable models.
• Universality: The same architecture works for text, images, audio, video, proteins, and virtually any sequential or structured data.
• Expressiveness: Self-attention can learn arbitrary pairwise relationships, giving Transformers enormous representational flexibility.