Post-training — SFT, RL, and GRPO

SFT and RL are two directions of the same KL

Up to a constant, SFT minimizes forward KL and RL minimizes reverse KL. That single fact explains most of how they differ in practice.

For distributions $P$ and $Q$ :

$D_{KL}(P \,\|\, Q) = \mathbb{E}_{x \sim P}[\log P(x) - \log Q(x)] \quad \text{(forward)}$ $D_{KL}(Q \,\|\, P) = \mathbb{E}_{x \sim Q}[\log Q(x) - \log P(x)] \quad \text{(reverse)}$

For LLMs these are distributions over completions. The key difference is what you sample from to take the expectation: your dataset (offline) or the model itself (on-policy).

SFT ≈ forward KL

Call the data distribution $p^*$ , so an SFT example is $x \sim p^*$ . The training objective is negative log-likelihood:

$\mathbb{E}_{x \sim p^*}[-\log p_t(x)]$

and

$D_{KL}(p^* \,\|\, p_t) = \text{const} - \mathbb{E}_{x \sim p^*}[\log p_t(x)]$

The right-hand side is the SFT objective, so minimizing either gives the same result.

RL ≈ reverse KL

With current policy $p_t$ and reference policy $p_{\text{ref}}$ , we maximize

$\mathbb{E}_{x \sim p_t}[r(x)] - \beta \, D_{KL}(p_t \,\|\, p_{\text{ref}})$

from which

$D_{KL}(p_t \,\|\, p^*) = -\tfrac{1}{\beta}\mathbb{E}_{x \sim p_t}[r(x)] + D_{KL}(p_t \,\|\, p_{\text{ref}}) + \text{const}$

This is the reverse KL relative to the SFT expression, and equals the RL objective up to a constant and a factor of $1/\beta$ . Note the KL inside the RL objective is a regularization term — unrelated to the KL we are showing the objective is equivalent to.

Mode-covering vs. mode-seeking

SFT is mode-covering. The model is punished for assigning low probability to any completion in the data, so it spreads mass across every mode present. Because NLL blows up as probability approaches zero, the loss grows exponentially if the model nearly rules out something in the dataset.

RL is mode-seeking. It maximizes reward on on-policy completions, so it concentrates on high-reward outputs even at the cost of abandoning modes entirely. Assigning near-zero probability to a completion just means that completion stops being sampled — no exponential penalty. Reward can still be maximized over whatever remains covered.

Following “SFT, RL, and On-Policy Distillation Through a Distributional Lens.”

PPO → GRPO

If PPO is already familiar, GRPO is two changes:

No value function. The advantage no longer uses a critic — the network that takes a state and estimates return, whether as a separate model or a second head sharing a trunk with the actor. GRPO replaces it by normalizing rewards over a group of samples: current reward minus the group mean, divided by the group standard deviation.
The grouping itself. Draw several samples from the same old policy, evaluate them as a group, then update. This smooths out reward variance in stochastic environments.

The elegance shows up in contextual-bandit-style setups with a direct single-step reward, where PPO forces you to train a critic on the fly against data generated by the agent’s own interaction. GRPO removes that second network entirely and works with the reward directly.

Two framings I want to keep straight:

Clipping is a surrogate for the trust region. It bounds how far the update can move, standing in for an explicit constraint.
The policy ratio is an importance sampling weight. Clipping bounds that ratio (new policy over old) to stop destabilizing updates.

Reference: PPO vs GRPO.

Adapter landscape

Worth keeping the family straight: full SFT, direct preference optimization, PEFT methods (LoRA and relatives), and GRPO-style RL all solve different problems, and the choice depends on whether you have preferences, verifiable rewards, or just demonstrations.