Fixation of Dominated Strategies in Group-Normalized Policy Gradient

The problem

When a language model is trained with reinforcement learning from verifiable rewards (RLVR), for example with GRPO, the reward only checks whether the final answer is correct. The reasoning that produced the answer is never inspected. So a chain of thought that is wrong but happens to land on the right answer gets reinforced just as strongly as a correct one.

This happens more often than it sounds. Answer spaces are small: a guess on a four-way multiple-choice question is right a quarter of the time, and even free-form numeric answers cluster on small integers. The question I am working on is when this actually takes over training, and what it means for a single run that someone is about to launch.

Why the existing answer is not enough

Write $c$ for how often sound reasoning reaches the right answer, and $g$ for how often flawed reasoning gets there anyway. Averaged over all randomness, the direction of each update is the sign of $g - c$. This has recently been shown in the literature, and the practical advice that follows is simple: if sound reasoning succeeds more often than guessing ($c > g$), training will favour it.

I argue that this is necessary but not sufficient, because an average is not a prediction. GRPO estimates each update from a small group of samples, typically 4 to 16. In a case where sound reasoning is genuinely better ($c = 0.50$, $g = 0.49$, group size 4), I measured a single update two million times: the noise is about 58 times larger than the signal.

That noise matters because the outcome is permanent. If bad luck pushes the model to produce sound reasoning only rarely, a group of four almost never contains a sound answer, so there is nothing left in the gradient to pull it back. A model cannot learn from something it never does.

When I ran 600 independent trainings in that setting, about two thirds locked onto sound reasoning and about one third locked onto flawed reasoning. None ended near the average. And the failure is invisible: the fully flawed model scores 49% on the benchmark, against 50% for the sound one, well inside seed noise.

My approach

Training as a random walk. I model training as a two-strategy selection process: each update is a small pull in the right direction plus a large random wobble, with an absorbing barrier at each end.
A single number decides the outcome. I show that the pull and the wobble carry the same state-dependent factor, so their ratio stays constant along the whole run. That ratio combines the group size $n$, the learning rate $\eta$ and how much better sound reasoning actually is:
\[\Gamma = \frac{2n\,(g - c)\,\kappa}{\eta\,\sigma_A^2}\]
A closed-form answer. With $\Gamma$ constant, the problem becomes the classical gambler’s ruin, which gives the probability that flawed reasoning takes over directly from $c$, $g$, $n$ and $\eta$.
No simulation needed. I derived the exact mean and variance of the GRPO gradient estimator as finite sums. Along the way, $\sigma_A^2$ turns out to have a simple meaning: the probability that a group is not unanimous, i.e. the fraction of updates that do anything at all.

Results so far

Across 45 simulated configurations, every one collapses onto the single curve predicted by $\Gamma$ (figure above). The prediction error is indistinguishable from the Monte-Carlo noise of the simulation itself.
With the same advantage for sound reasoning, a group size of 4 ends in flawed reasoning about 31% of the time, while a group size of 16 does so only about 3% of the time. The sign of $g - c$ gives the direction; the group size and learning rate decide individual runs.
Doubling the learning rate halves $\Gamma$: larger steps buy noise faster than signal.
This gives a precise account of something practitioners see all the time: identical hyperparameters with different seeds producing noticeably different models.

What I am working on next

Adding PPO-style clipping and KL anchoring to the simulator, to test whether the collapse onto one curve survives them.
Testing the prediction on real language models. The current results use a simplified policy with a single dial between sound and flawed reasoning, and the next step is to see where real models agree or disagree.