In-Context Learning
In-context learning is when a model adapts behavior from examples, instructions, or evidence placed in the context window without changing model weights.
A prompt with examples can be viewed as:
where is an instruction and are examples.
What Changes
In-context learning changes the conditional distribution, not the model parameters:
where is a set of examples or evidence. This is different from fine-tuning, where training updates .
Example Selection
Example quality matters. A small demonstration set can be selected by similarity:
or by coverage of expected cases:
Random examples, mislabeled examples, or examples with hidden leakage can make in-context learning look stronger than it is.
Instruction and Demonstration Conflict
If instructions and examples disagree, the model may follow either:
For workflows that require reliability, examples should illustrate the instruction rather than silently redefining it.
Evaluation Risk
Few-shot prompts can leak labels if examples are drawn from the evaluation set or from near-duplicates:
For retrieval-augmented workflows, retrieved examples should be checked against the test unit and split unit.
Checks
- Are examples representative of the desired behavior?
- Are instructions consistent with examples?
- Are labels or answers leaked into evaluation prompts?
- Is the behavior stable under reordered or paraphrased examples?
- Would fine-tuning or a tool be more reliable than prompting?
- Are examples selected before seeing target labels?
- Are examples current, public-safe, and free of private data?
- Is the prompt tested on edge cases, not only happy-path examples?
- Is the output format enforced with structured output when downstream tools consume it?