Imitation Learning
Imitation learning trains a policy from demonstrations. Instead of discovering behavior only through reward, the model learns from expert trajectories or high-quality examples.
For demonstration data , behavioral cloning uses supervised learning:
For language models and coding agents, supervised fine-tuning on high-quality traces is a form of imitation learning.
The central failure mode is distribution shift. The learned policy changes the states it visits:
If demonstrations only cover expert states, small policy errors can compound as the model enters states with no demonstrated recovery action.
Key Ideas
- Demonstrations provide a dense learning signal when rewards are sparse or expensive.
- The model can copy expert behavior but may fail when it reaches states not covered by demonstrations.
- Imitation learning is often a first stage before preference optimization or reinforcement learning.
- Trace quality matters more than volume when demonstrations encode tool use, verification, or reasoning style.
Variants
| Variant | Signal | Main Risk |
|---|---|---|
| Behavioral cloning | state-action pairs | covariate shift after mistakes |
| Dataset aggregation | expert labels on learner-visited states | expert query cost and feedback consistency |
| Inverse reinforcement learning | infer reward from demonstrations | reward ambiguity |
| Offline imitation from traces | logs, tool calls, edits, or conversations | private data leakage and spurious style copying |
Practical Checks
- Are demonstrations representative of the target deployment setting?
- Does the policy recover from mistakes, or only imitate clean trajectories?
- Are bad examples filtered or labeled separately?
- Does imitation preserve private or sensitive details that should be removed before public training or documentation?
- Are tool calls, verifier steps, and failure recovery included, or only successful final outputs?
- Is the evaluation on held-out trajectories, live interaction, or downstream task success?