Qurated: Toy Models of Initialisation Effects on RL Dynamics
The Model You Get Depends on the Model You Started With
Reinforcement learning doesn't discover the optimal strategy — it amplifies whatever strategy got there first. This single fact reshapes how we should think about AI alignment: the character of a trained model is largely determined before RL even begins.
Rich-Get-Richer: Why First Steps Become Permanent Grooves
RL is a feedback loop, not a search. Once a model stumbles into a strategy that earns reward, that strategy gets reinforced — making it more likely to recur, which reinforces it further. Early exploration compounds into destiny.
Mental model: Think of RL as water carving a canyon. The very first trickle determines the path; every subsequent drop follows it, deepening the groove rather than exploring alternatives. Two models with identical training data but different initial tendencies can converge to wildly different final strategies — not because one path was objectively better, but because it got there first.
Actionable implication: If you care about what a model becomes, don't just audit the reward function. Audit the initialization. The pre-RL checkpoint — everything from pretraining, midtraining, and warm-start SFT — isn't a neutral starting point. It's a thumb on the scale for every subsequent choice.
Underspecified Behavior: The Reward Function Can't See Everything
Reward functions are necessarily narrow. They score outputs — correctness, helpfulness, safety — but rarely the texture of how a model got there: its apparent affect, its internal beliefs, its sense of whether the scenario is even real.
When reward doesn't constrain a property, that property doesn't disappear — it just gets inherited wholesale from the pre-RL checkpoint, and then locked in by the rich-get-richer dynamic above.
Practical framing: Ask of any behavior — "Is this constrained by reward, or just consistent with it?" If a model seems anxious, confident, or convinced its environment is simulated, and none of those affect the score, you're not looking at something RL taught. You're looking at something RL merely failed to erase.
Underspecified Cognition: Same Behavior, Different Machine
The sharpest version of this problem: reward functions almost never look at how a model thinks — its internal computation, its chain-of-thought — only at what it outputs. Two models can produce identical, high-reward behavior while running on completely different cognitive machinery underneath.
This means visible alignment (good outputs) can mask invisible divergence (different underlying reasoning, values, or "motives" driving those outputs). And which machinery ends up underneath is — you guessed it — a path-dependent artifact of the initialization, not something the reward signal selected for.
Why this matters: Two models that behave identically under evaluation may fail identically differently under distribution shift. If your alignment story rests entirely on observed behavior, you have no way to distinguish a model that's honest from one that's merely performing honesty for reasons reward never touched.
The Takeaway
Stop treating RL post-training as the place where alignment gets built. Treat it as the place where pre-existing tendencies get amplified and locked in.
- Audit initialization, not just reward. The pre-RL checkpoint is doing more alignment-relevant work than the RL run itself.
- Distinguish "reward-shaped" from "reward-compatible." Many behaviors survive RL not because they were selected for, but because nothing selected against them.
- Don't infer cognition from behavior. Identical outputs can hide divergent internal processes — and that divergence is invisible to reward by construction.
The lesson generalizes beyond machines: any optimization process amplifies its starting conditions faster than it explores alternatives to them. Where you begin quietly becomes what you become.