The Transformer Architecture: From Scaled Dot-Product Attention to Modern Frontier Variants

1. Executive Summary#

When Vaswani et al. published Attention Is All You Need in 2017, they solved the fundamental bottleneck of Recurrent Neural Networks (RNNs and LSTMs): sequential time dependency. By replacing recurrence with self-attention, Transformers turned sequence modeling into highly parallelizable matrix multiplications. However, scaling from a 65M-parameter translation model in 2017 to modern 400B+ frontier models required re-engineering nearly every component of the original architecture.

2. The Foundational Mechanics (2017 Blueprint)#

A. Scaled Dot-Product Attention#

For an input sequence, embeddings are projected via learned weight matrices into Queries (Q), Keys (K), and Values (V). The attention formula is:

Attention(Q, K, V) = Softmax((Q * K^T) / sqrt(d_k)) * VWhy divide by sqrt(d_k)? When the head dimension d_k is large, the dot products Q * K^T grow large in magnitude, pushing the Softmax function into regions with near-zero gradients. Scaling by 1 / sqrt(d_k) preserves unit variance so gradients flow cleanly during backpropagation.

B. Multi-Head Attention (MHA)#

Instead of computing attention once across the full hidden dimension d_model, MHA projects Q, K, and V into h lower-dimensional subspaces (d_k = d_model / h). This allows Head 1 to track syntactic dependencies (e.g., verb-subject agreement), Head 2 to resolve coreferences (e.g., pronoun to noun), and Head 3 to attend to adjacent tokens simultaneously.

C. Why Decoder-Only Won Over Encoder-Decoder#

While the original Transformer used an Encoder-Decoder architecture for machine translation, modern foundation models (GPT, Llama, Mistral, Qwen, DeepSeek) converged on Causal Decoder-Only architectures. In a causal decoder, a triangular mask forces each token t to attend only to tokens <= t. This turns every single token position in a training document into a simultaneous next-token prediction target, maximizing training efficiency across trillions of tokens.

3. Five Architectural Upgrades in Modern LLMs#

1. Pre-Norm (RMSNorm) vs. Post-Norm (LayerNorm)#

• Original (2017): Applied LayerNorm after the residual connection (x + Sublayer(x)), which required delicate learning-rate warmup to prevent gradient explosions in deep networks.
• Modern Standard: Applies RMSNorm (Root Mean Square Normalization, Zhang & Sennrich 2019) before the attention and MLP blocks. RMSNorm drops the mean-centering calculation of LayerNorm, reducing compute overhead by ~10–15% while stabilizing deep gradient flow.

2. Rotary Position Embeddings (RoPE) vs. Sinusoidal Embeddings#

• Original (2017): Added absolute sinusoidal positional vectors to token embeddings at the bottom layer.
• Modern Standard (Su et al., 2021): RoPE encodes position by rotating the Query and Key vectors in complex 2D planes at every attention layer. Crucially, the dot product between a rotated Query at position m and a rotated Key at position n depends strictly on their relative distance (m - n), enabling strong length extrapolation and context extension methods (such as YaRN and NTK-aware scaling).

3. Grouped-Query Attention (GQA) vs. Multi-Head Attention (MHA)#

• The Bottleneck: During inference, storing separate Key and Value tensors for every attention head balloons the KV cache in GPU memory.
• Modern Standard (Ainslie et al., 2023): GQA divides Query heads into G groups, where each group shares a single Key and Value head. A 64-query-head model with 8 KV heads reduces KV-cache memory footprint and bandwidth requirements by 8x with virtually zero perplexity degradation compared to MHA.

4. SwiGLU Feed-Forward Networks vs. ReLU MLPs#

• Original (2017): Used a two-layer MLP with a ReLU activation: max(0, xW_1 + b_1)W_2 + b_2.
• Modern Standard (Shazeer, 2020): Replaces standard MLPs with SwiGLU (Swish-Gated Linear Unit), which uses three weight matrices (Gate, Up, and Down projections) where one projection acts as a learnable multiplicative gate (Swish(xW_gate) * (xW_up)). This gated expressiveness consistently lowers validation loss per FLOP.

5. FlashAttention (Dao et al., 2022 & 2023)#

• The Bottleneck: Standard attention computes the N x N score matrix S = Q * K^T, writes it to slow GPU HBM, reads it back to compute P = Softmax(S), writes P to HBM, and reads P back to multiply by V.
• The Solution: FlashAttention uses GPU kernel fusion and online tiling (via the online softmax trick) to compute exact attention inside tiny, ultra-fast on-chip SRAM blocks without ever materializing the N x N intermediate matrix in HBM—cutting memory complexity from O(N^2) to O(N) and accelerating training and prefill by 2x to 4x.