Show, Don't Tell: Few-Shot & Examples

Chapter 01 framed a prompt as conditioning: every token you place in context reshapes the distribution over what comes next. Examples are the most aggressive form of conditioning available, because they exploit a circuit the model already runs on every forward pass. Vol II · Chapter 03 introduced induction heads — attention heads that find an earlier occurrence of the current pattern and copy what followed it. A few-shot prompt is, structurally, bait for exactly that circuit: input → output, input → output, input → ? is the repeated-pattern format induction heads were discovered completing. No weights change. The "learning" in in-context learning is pattern-matching over the residual stream, executed fresh on every call.

Two findings sharpen the picture, and both should change how you write prompts:

• Examples mostly tell the model which task, not how to do it. Min et al. (2022) showed that on many classification benchmarks, replacing the gold labels in few-shot examples with random labels barely dents accuracy. What carried the performance was the input distribution, the label space, and the format — the demonstration's shape. The model already knew how to classify sentiment; the examples told it that sentiment classification, in this exact format, is what's happening here. This is often called task recognition as opposed to task learning.
• But bigger models do read the labels. Wei et al. (2023) found that as scale grows, models increasingly override their semantic priors and follow flipped or arbitrary label mappings in the examples. Frontier models genuinely extract input→output rules from context — a capability some theoretical work models as implicit regression or gradient-descent-like updating inside the forward pass. That account remains contested; the empirical part is not.

Since the GPT-3 paper plotted accuracy against shot count in 2020, the same shape has recurred across tasks and model generations: a steep rise that flattens fast. The single biggest jump is zero → one — the first example resolves the format, the label space, and most of the task ambiguity at once. Each additional example refines boundaries with diminishing returns. A saturating exponential captures the shape well enough to reason with:

The parameters cluster by task family. Format-following tasks (emit this JSON, this tag style, this report skeleton) have a huge \(a_\infty - a_0\) gap and tiny \(\kappa\): one or two examples and you're done, because format is precisely what demonstrations transmit best. Classification and extraction saturate more slowly — around 4–8 shots — since later examples still sharpen category boundaries. Reasoning-heavy tasks barely move: a worked example changes the style of the solution trace, not the model's ability to solve (Chapter 04 picks up that thread). Explore the three regimes:

The same exponential, in code. Each task family gets its own \((a_\infty, a_\infty{-}a_0, \kappa)\); the table makes the diminishing marginal return concrete — read down any column and watch the per-shot gain collapse. Edit tau for reasoning and watch its curve refuse to move regardless:

# k-shot accuracy curve (ILLUSTRATIVE) — acc(k) = ceil - gap*exp(-k/tau)
import numpy as np
k = np.arange(0, 9)
tasks = {  # (ceiling, gap, tau) — hand-built per EQ P3.1, NOT measured
    "format": (0.97, 0.55, 0.7),
    "classif": (0.88, 0.33, 3.2),
    "reason":  (0.66, 0.05, 4.0),
}
print("ILLUSTRATIVE — hand-built shapes, not measurements")
print("  k " + "".join(f"{n:>9}" for n in tasks))
for kk in k:
    row = [ceil - gap*np.exp(-kk/tau) for ceil, gap, tau in tasks.values()]
    print(f"{kk:>3} " + "".join(f"{a:>9.3f}" for a in row))
fmt = tasks["format"]
print("\nformat marginal gain, shot 1 vs shot 2:")
g1 = (fmt[0]-fmt[1]*np.exp(-1/fmt[2])) - (fmt[0]-fmt[1])
g2 = (fmt[0]-fmt[1]*np.exp(-2/fmt[2])) - (fmt[0]-fmt[1]*np.exp(-1/fmt[2]))
print(f"  +{g1*100:5.1f} pts then +{g2*100:5.1f} pts  (geometric decay)")
acc_fmt = [fmt[0]-fmt[1]*np.exp(-kk/fmt[2]) for kk in k]
plot_xy(k.tolist(), acc_fmt)

The printed table is the same EQ P3.1 the instrument above draws — the value of this view is the marginal-gain line: format-following banks most of its lift on the first shot and almost nothing after the second, exactly the geometric decay the equation predicts. The reasoning column barely leaves its starting value, the algebraic face of "examples don't teach a model to reason."

The long-context caveat. "Many-shot" in-context learning (Agarwal et al., 2024) showed that with hundreds to thousands of examples — feasible once contexts crossed 100K tokens — some tasks keep improving well past where the classic curve flattens, occasionally approaching fine-tuning quality. The exponential above describes the 0–32 shot regime where almost all practical prompting lives; treat the far tail as a separate tool with fine-tuning-like economics (Vol II · Chapter 06), amortized only if you cache the prefix.

The instinct is to pick your prettiest, most typical examples — three clean inputs with three clean outputs. That teaches the model a task narrower than yours. Production inputs are mostly edge cases wearing a trench coat: the empty field, the two-languages-in-one-sentence ticket, the review that praises the product while demanding a refund. Since examples define task boundaries (§3.1), an example spent on the happy path is a wasted boundary — the model already assumed the happy path.

Dynamic selection. When you have a pool of candidate examples, retrieving the nearest neighbors of the current input — embed the query, embed the pool, take top-\(k\) by cosine similarity — reliably beats a fixed example set (the KATE result, Liu et al. 2021, since reproduced broadly). It is the same move as RAG, aimed at demonstrations instead of facts. Two cautions. First, similarity retrieval quietly destroys diversity: five neighbors of an unambiguous input are five near-identical demos, so production systems usually blend retrieved neighbors with a fixed diverse core. Second, retrieval changes the prompt prefix per request, which invalidates prefix caching (Vol II · Chapter 08) — at scale, the static-set discount is real money.

Few-shot prompts are not sets; they are sequences, and the model reads them with position-dependent attention. Zhao et al. (2021) measured the damage on GPT-3: across permutations of the same four examples, SST-2 sentiment accuracy ranged from near-chance to state-of-the-art. They isolated three biases — majority-label bias (predictions drift toward the most frequent label in the demos), recency bias (the last example's label bleeds into the prediction most), and common-token bias. Recency is the one ordering controls. A minimal model of the skew:

The instrument shows one ordering at a time; the cell below sweeps all orderings of the same balanced four-demo block, applies EQ P3.2, and reports how often the recency-weighted prior lands on the final example's label. With perfectly balanced labels a position-blind model would sit at 50%:

# recency bias over orderings (ILLUSTRATIVE) — EQ P3.2, beta>0
import numpy as np
from itertools import permutations
rng = np.random.default_rng(0)
labels = np.array([1, 1, 0, 0])    # 2 POSITIVE (1), 2 NEGATIVE (0) — balanced
beta = 0.65
w = np.exp(beta * np.arange(1, 5))  # recency weights, last demo heaviest
perms = list(set(permutations(range(4))))
agree, mass = [], []                # match + prior mass on the LAST label's class
for _ in range(2000):
    order = perms[rng.integers(len(perms))]
    lab = labels[list(order)]
    p_pos = (w * lab).sum() / w.sum()
    p_last = p_pos if lab[-1] == 1 else 1 - p_pos
    agree.append(int((p_pos >= 0.5) == (lab[-1] == 1)))
    mass.append(p_last)
agree, mass = np.array(agree), np.array(mass)
print("balanced labels, position-blind baseline: 50.0%")
print(f"shuffles simulated         : {agree.size}")
print(f"prior matches LAST label   : {100*agree.mean():.1f}%  (recency skew)")
print(f"mean prior mass on LAST cls : {100*mass.mean():.1f}%  (vs 50% if blind)")

Set beta = 0 and the match rate falls to the coin-flip baseline — pure majority-label bias with no recency. Any \(\beta > 0\) pushes the prior toward whatever label sits last, which is why "end on your most representative example" is the single cheapest ordering fix. This is the toy model, not a transformer; what it reproduces is the direction and the order-sensitivity, not a specific model's magnitude.

What to do about it. Four mitigations, in increasing order of effort: (1) balance labels and end on the most representative example, since the last slot leaks hardest; (2) never sort examples by label — alternate or randomize within the balanced set; (3) for evaluation, average over several orders rather than trusting one (Chapter 07); (4) contextual calibration — measure the model's output on a content-free input like "N/A" and divide it out — recovers most of the lost accuracy when you control the decoding stack. Modern instruct models are meaningfully better calibrated than the GPT-3 these biases were measured on, but the bleed has not gone to zero — it has gone subtle, which is worse for debugging.

Everything in your examples leaks into the output: the casing of keys, the order of fields, trailing punctuation, whether lists end with a period, the average response length. Most discussions treat this as a hazard. It is also the most reliable format-specification mechanism that exists — more reliable than describing the format in prose, because a description must be parsed and interpreted while a demonstration is simply continued. Compare:

# DESCRIBED — the model must translate prose into structure
Return JSON with keys "sentiment" (one of positive|negative|mixed),
"confidence" (a float between 0 and 1, two decimals), and "evidence"
(an array of verbatim quotes, at most two).

# DEMONSTRATED — the model continues the pattern
Input:  "Battery life is superb but the hinge broke in a week."
Output: {"sentiment": "mixed", "confidence": 0.86, "evidence": ["superb", "broke in a week"]}

One example pins down a dozen micro-decisions the description left open: key order, float precision, quote style, whether evidence is verbatim or paraphrased. The production pattern is describe once, demonstrate twice — a short prose spec for the rules that examples can't carry (ranges, enums, fallbacks), then two examples that settle everything else. Chapter 05 replaces this with constrained decoding where available; few-shot formatting remains the portable fallback that works on every model.

Positive examples define the target; they say nothing about the boundary. When the failure mode is the model doing something almost right — summaries that editorialize, refusals that over-trigger, SQL that works but scans the whole table — the fastest fix is a contrastive pair: the same input with a good output, a bad output, and an explicit label on each saying why. The WHY annotation is what separates this from merely doubling your shot count: it converts an instance into a rule the model can apply to unseen cases.

# Contrastive pair for a support-summary task
Input: [47-message thread about a delayed refund]

GOOD:  "Customer requested refund 12 May; agent escalated 19 May;
       refund pending finance approval. Customer contacted support 4×."
// WHY GOOD: only verifiable facts, dates preserved, no sentiment language

BAD:   "Frustrated customer has been chasing an overdue refund for weeks
       while support repeatedly dropped the ball."
// WHY BAD: editorializes ("dropped the ball"), drops dates, asserts blame

Three rules keep contrastive prompts from backfiring. Label loudly — the GOOD/BAD markers must be unmissable, because an unlabeled bad output is just another demonstration and will be imitated. Make the why specific — "too informal" teaches less than "uses sentiment adjectives instead of dates". And end on good: §3.4's recency bias applies to quality exactly as it applies to labels, so the last thing in the example block should always be behavior you want continued. Contrastive pairs are also the natural home for near-misses harvested from production — every bad output your evals catch (Chapter 07) is a free BAD half waiting for its annotation.

Examples are a lever, not a ritual. Three situations where adding them subtracts value:

And always, the unglamorous one: cost. Every shot is paid on every call, forever. Eight 120-token examples are ~960 tokens of overhead — per request, that is rounding error; at ten million requests a month it is ten billion input tokens spent re-teaching a model the same four boundary cases. Prefix caching (Vol II · Chapter 08) discounts a static example block substantially, which is a real argument for fixed sets over per-query retrieval at high volume — and when the example block stops fitting the budget at all, the escalation path is the one Vol II · Chapter 06 opened: distill the behavior into the weights and delete the demos.

• Brown et al. (2020). Language Models are Few-Shot Learners. — the GPT-3 paper that introduced in-context few-shot learning and first plotted accuracy against shot count.
• Min et al. (2022). Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? — shows that label correctness matters far less than the demonstrations' format, input distribution, and label space.
• Zhao et al. (2021). Calibrate Before Use: Improving Few-Shot Performance of Language Models. — measures majority-label, recency, and common-token bias, and introduces contextual calibration as the fix.
• Lu et al. (2022). Fantastically Ordered Prompts and Where to Find Them. — demonstrates the extreme sensitivity of few-shot accuracy to example ordering and how to select good permutations without labelled data.
• Liu et al. (2022). What Makes Good In-Context Examples for GPT-3? — the KATE result: retrieving nearest-neighbour demonstrations of the query beats a fixed example set.
• Wei et al. (2023). Larger Language Models Do In-Context Learning Differently. — finds that at scale models increasingly override semantic priors and follow flipped or arbitrary label mappings in the examples.
• Agarwal et al. (2024). Many-Shot In-Context Learning. — shows that hundreds-to-thousands of examples, enabled by long context, can keep improving tasks well past the classic saturation point.