From PPO to DPO to GRPO: The Evolution of LLM Alignment and Test-Time Reasoning

1. Executive Summary#

Pre-training gives an LLM broad world knowledge, but post-training determines whether it can follow complex instructions and reason through multi-step math, code, and system design problems. Over the past three years, post-training has undergone a radical simplification—culminating in GRPO (Group Relative Policy Optimization) and RLVR (Reinforcement Learning with Verifiable Rewards), the engines behind modern reasoning models.

2. Generation 1: RLHF with PPO (Proximal Policy Optimization)#

Classical RLHF (used in early InstructGPT and ChatGPT) required holding four separate models in GPU memory simultaneously:

1. Policy (Actor) Model: The LLM being trained.
2. Reference Model (Frozen): Prevents the actor from drifting too far via a KL-divergence penalty.
3. Reward Model (Frozen): Scores the final response based on human preference pairs.
4. Value (Critic) Model: Estimates the expected future reward at every token position to compute advantage estimates (GAE).

The Engineering Bottleneck: Training a Critic network that is just as large as the Policy model doubles trainable VRAM requirements, introduces severe hyperparameter sensitivity, and frequently suffers from reward hacking.

3. Generation 2: DPO (Direct Preference Optimization)#

Rafailov et al. (2023) proved that you can analytically solve for the optimal policy directly from the reward formulation using a change of variables. DPO eliminates both the Reward Model and the Critic Model, turning alignment into a supervised binary cross-entropy loss over (prompt, chosen_response, rejected_response) triples.

Where DPO Falls Short: Because DPO is typically run off-policy on static datasets, the model cannot explore novel reasoning trajectories or learn from its own live mistakes during multi-step problem solving.

4. Generation 3: GRPO & RLVR (The DeepSeek-R1 Breakthrough)#

To train reasoning models that think via extended Chain-of-Thought (Test-Time Compute Scaling), on-policy Reinforcement Learning returned—but without the heavy Critic model.

How GRPO Works:#

1. Group Sampling: For a single prompt q (e.g., a competitive programming problem), the policy samples a group of G distinct outputs (e.g., G = 16 completions): {o_1, o_2, ..., o_G}.
2. Verifiable Rule-Based Scoring (RLVR): Instead of a neural reward model that can be tricked by verbose formatting, outputs are scored by deterministic verifiers: unit test suites, compilers, symbolic math checkers, or exact-match parsers, producing rewards {r_1, r_2, ..., r_G}.
3. Group-Normalized Advantage: Instead of training a Critic network to predict the baseline reward, GRPO simply uses the group's mean and standard deviation as the baseline:
   Advantage_i = (r_i - mean(r)) / std(r)
4. Policy Update: The model boosts the probability of trajectories that scored above the group average and suppresses those below, bounded by a clipped surrogate objective and KL penalty.

Why This Matters for Senior Engineers: Whenever you have a domain with a deterministic verifier—SQL query generation, Kubernetes YAML validation, TypeScript compilation, or financial reconciliation—you can build an RLVR + GRPO pipeline to make a small 7B–14B open-weights model outperform massive general-purpose models on your specific task.