Mechanistic Interpretability & Sparse Autoencoders

Training tells you that a network works. It does not tell you how. The weights are the only artifact, and they encode whatever algorithm gradient descent happened to find. Mechanistic interpretability is the program of recovering that algorithm: not "which input pixels mattered" (the question saliency maps ask) but "what computation is this set of weights actually running, expressed in terms a person can audit." The unit of explanation is the circuit — a subgraph of features and the weights connecting them that together implement an identifiable function, such as "detect that the current token repeats an earlier bigram, then copy what came next."

The motivation is partly scientific and partly safety. If a model can be made to lie, refuse, or pursue a goal, you would like to find the internal mechanism that does so, verify it causally, and ideally intervene. Saliency and attention-weight visualization were the first attempts; both proved too coarse, because they describe where a network looked, not what it concluded. The mechanistic program instead insists on three commitments: name the intermediate variables (features), describe the computation over them (circuits), and confirm each claim with a causal experiment rather than a correlation.

Two facts make this hard, and they structure the rest of the chapter. First, the natural unit you can read off directly — the neuron — usually does not correspond to a single concept. Second, the reason is not noise but an efficient encoding the network chose deliberately: superposition, packing more features than it has dimensions. Sections 13.2 and 13.3 establish the problem; 13.4 gives the tool that addresses it; 13.5 supplies the causal tests; 13.6 is the honest accounting of what we can and cannot yet claim.

If features lined up with neurons, interpretability would be a labeling exercise: read each neuron's top-activating inputs, write down the concept. Some neurons do behave this way — a few are cleanly monosemantic, firing for exactly one human-recognizable thing. Most are not. A typical neuron is polysemantic: its top activations include unrelated concepts — French text, DNA codons, and HTTP headers all lighting up the same coordinate. There is no single caption you can write for it that is faithful.

Polysemanticity is the central obstacle. It is not that the network failed to organize itself; it is that the natural basis (the neurons) is the wrong coordinate system. The concepts are still in there, but they live along directions that cut across many neurons at once. Reading the activation vector in the neuron basis is like reading a paragraph with the word boundaries deleted: every letter is present, but the units of meaning are not where the gaps are.

Why would training produce non-axis-aligned features? Because there is no pressure to align them. Gradient descent optimizes the loss, and the loss is invariant to how you rotate the hidden basis: any rotation of the representation, undone by the next layer's weights, computes the same function. The network is free to place features wherever is cheapest, and §13.3 argues that "cheapest" usually means "not one per neuron."

The leading explanation for polysemanticity is superposition: a network represents more features than it has dimensions by storing them as directions that are not orthogonal but merely close to orthogonal. With \(d\) dimensions you get only \(d\) perfectly orthogonal directions, but you can pack many more nearly-orthogonal ones into the same space — the Johnson–Lindenstrauss bound says the number you can fit while keeping pairwise overlap below \(\varepsilon\) grows exponentially in \(d\). The price is interference: any two features whose directions are not exactly perpendicular leak a little into each other's readout.

Superposition is only viable because real features are sparse: on any given input, almost none of the thousands of possible features are active. If two rarely-active features happen to share a slightly-overlapping direction, they almost never fire together, so the cross-talk in EQ 13.2 almost never materializes. Sparsity is what lets the network get away with the overpacking. Elhage et al. (2022) showed in toy models that as feature sparsity rises, networks transition from a "one feature per neuron" regime into dense superposition, often arranging features into tidy geometric structures (antipodal pairs, pentagons, tetrahedra) that minimize worst-case interference.

# Superposition: pack F features into d dims, measure interference (EQ 13.2/13.3)
import numpy as np
rng = np.random.default_rng(0)
d, F, k = 8, 40, 3          # dims, features, active-per-input (L0)

D = rng.normal(0, 1, (d, F))
D /= np.linalg.norm(D, axis=0, keepdims=True)   # unit feature directions
G = D.T @ D                                       # Gram matrix of cosines
off = G[~np.eye(F, dtype=bool)]                   # off-diagonal = interference
print(f"F/d packing ratio     : {F/d:.2f}  (>1 means superposition)")
print(f"mean |cosine| off-diag: {np.abs(off).mean():.4f}")
print(f"max  |cosine| off-diag: {np.abs(off).max():.4f}")

errs = []                                          # read-out error vs sparsity
for kk in (1, 2, 4, 8):
    e = []
    for _ in range(400):
        a = np.zeros(F); idx = rng.choice(F, kk, replace=False)
        a[idx] = rng.uniform(0.5, 1.5, kk)         # kk active features
        x = D @ a                                   # superposed activation
        a_hat = D.T @ x                             # project to read each feature
        e.append(np.sqrt(np.mean((a_hat - a)**2)))
    errs.append(np.mean(e))
    print(f"L0={kk}: mean recon RMSE over {F} features = {errs[-1]:.4f}")
print("\nfewer active features -> less collision -> lower error: sparsity buys capacity.")
plot_xy([1,2,4,8], errs)                            # error rises with density

If superposition is the disease, dictionary learning is the cure, and a sparse autoencoder (SAE) is how the field runs it at scale. The idea: train a small auxiliary network to re-express each activation vector \(x\) as a sparse, non-negative combination of a large, learned dictionary of directions — exactly the form of EQ 13.1. If we force the code to be sparse, individual dictionary entries are pushed toward each capturing one concept, recovering the monosemantic features the neuron basis hid.

What stops the SAE from learning the trivial identity (one dictionary atom per input)? The loss. It balances two terms: reconstruct \(x\) faithfully, and keep the code sparse.

Training an SAE is then ordinary gradient descent on EQ 13.5 over a large bank of cached model activations. Three engineering realities dominate the practice:

• Dead features. Some dictionary atoms stop activating on any input and contribute nothing. A large fraction can die during training. Practitioners track the dead-feature count and revive them (resampling, auxiliary "ghost" losses, or careful initialization), because a dead atom is wasted dictionary width.
• The width / L0 / fidelity trilemma. Wider dictionaries (\(m\) larger) and lower \(\lambda\) both buy lower reconstruction error, but lower \(\lambda\) also raises L0 (less sparse, less monosemantic), and wider dictionaries cost compute and tend to split one concept into many near-duplicate atoms. There is no single best point; you choose where on the frontier to sit.
• Feature splitting and absorption. As you add width, a coarse feature ("a dog") fractures into finer ones ("a dog, in French", "a dog, as subject"). Useful sometimes, misleading others — it means the count of "features" is partly an artifact of dictionary size, not a fact about the model.

The landmark demonstrations are Anthropic's "Towards Monosemanticity" (Bricken et al., 2023), which extracted thousands of interpretable features from a one-layer transformer, and "Scaling Monosemanticity" (Templeton et al., 2024), which pushed SAEs to a production-scale model (Claude 3 Sonnet) and found millions of features, including abstract and multilingual ones, that could be located and manipulated. Those results are the reason SAEs went from a curiosity to the field's default decomposition tool.

# Train a tiny SAE on synthetic sparse data by plain gradient descent (EQ 13.4/13.5)
import numpy as np
rng = np.random.default_rng(0)
d, m, k = 6, 20, 2          # input dim, dictionary width (overcomplete), true active-per-input

Dtrue = rng.normal(0, 1, (d, m)); Dtrue /= np.linalg.norm(Dtrue, axis=0, keepdims=True)
def batch(B):                                   # sparse codes -> superposed activations
    A = np.zeros((B, m))
    for r in range(B):
        idx = rng.choice(m, k, replace=False); A[r, idx] = rng.uniform(.5, 1.5, k)
    return A @ Dtrue.T, A                        # x = D a  (shape B x d), and true code

Wenc = rng.normal(0, .1, (m, d)); Wdec = rng.normal(0, .1, (d, m)); lr, lam = .05, .04
for step in range(600):
    x, _ = batch(128)
    f = np.maximum(0, x @ Wenc.T)               # encode: ReLU code (EQ 13.4)
    xh = f @ Wdec.T                             # decode
    e = xh - x
    gWdec = e.T @ f / len(x)
    gf = (e @ Wdec) * (f > 0) + lam * np.sign(f) * (f > 0)   # recon + L1 grads (EQ 13.5)
    gWenc = gf.T @ x / len(x)
    Wdec -= lr * gWdec; Wenc -= lr * gWenc
    Wdec /= np.linalg.norm(Wdec, axis=0, keepdims=True)      # keep atoms unit-norm

x, _ = batch(2000); f = np.maximum(0, x @ Wenc.T); xh = f @ Wdec.T
recon = np.sqrt(np.mean((x - xh)**2)) / np.sqrt(np.mean(x**2))
L0 = (f > 1e-3).sum(1).mean()
dead = int((f.max(0) < 1e-3).sum())
print(f"relative reconstruction error : {recon:.3f}")
print(f"mean L0 (active feats / input): {L0:.2f}   (true sparsity k = {k})")
print(f"dead dictionary atoms         : {dead} / {m}")
print("the SAE learns a sparse code that rebuilds x while firing only a few atoms.")

Features are nouns; circuits are the sentences. A circuit is a set of features wired together by the model's weights to compute something. The most thoroughly reverse-engineered example is the induction head (Chapter 03), a two-head circuit that implements in-context copying: a "previous-token" head writes each token's predecessor into the residual stream, and a second head then attends from the current token to the position after its earlier copy, predicting "what came next last time." Olsson et al. (2022) traced the abrupt appearance of induction heads to the sharp drop in training loss that coincides with models acquiring in-context learning — a rare case where a capability, a circuit, and a loss-curve kink were tied together.

Two cheap tools are the everyday workhorses of circuit-finding:

• The logit lens. Apply the model's final unembedding to the residual stream at intermediate layers, as if you had stopped early. This reveals the model's "current best guess" forming layer by layer, and where in depth a prediction crystallizes. It is approximate (the residual stream is not calibrated to be decoded early), but it is a fast first read on where a computation happens.
• Ablations. Zero out (or mean-out) a component — a head, a neuron, an SAE feature — and measure how much the output changes. A large change implicates the component; a negligible one rules it out. Ablation is correlational about importance but a direct causal test of necessity.

The gold standard, though, is activation patching (also called causal tracing or interchange intervention). It isolates which internal location causes a behavior by transplanting activations between two runs.

Patching is causal because it asks a counterfactual: had this one location held the other run's value, would the answer change? That is stronger than ablation (which only asks "does removing it hurt") and far stronger than attention-weight inspection (which asks nothing causal at all). The same logic, applied to SAE features instead of raw activations, is how the field tests whether a discovered feature is a genuine causal variable or merely a correlate.

# Activation-patching toy: swap one hidden coordinate, measure output change (EQ 13.6)
import numpy as np
rng = np.random.default_rng(1)
d = 8
Wout = rng.normal(0, 1, d)                       # toy readout: logit = Wout . h

def hidden(prompt_seed, fact):                    # a toy "model": fact lives in coord 3
    rng2 = np.random.default_rng(prompt_seed)
    h = rng2.normal(0, 0.3, d)                     # generic context
    h[3] = 2.5 * fact                              # the decisive coordinate
    return h

h_clean   = hidden(7, fact=+1)                     # clean run: correct fact
h_corrupt = hidden(7, fact=-1)                     # corrupted run: wrong fact
base = Wout @ h_clean
print(f"clean logit baseline: {base:+.3f}\n")

print(" patch coord | logit after patch |  delta")
for c in range(d):
    h = h_clean.copy()
    h[c] = h_corrupt[c]                            # patch ONE coordinate from corrupt run
    delta = (Wout @ h) - base
    flag = "  <-- decisive" if abs(delta) > 1.0 else ""
    print(f"     {c:2d}     |      {Wout @ h:+7.3f}     | {delta:+7.3f}{flag}")
print("\nonly the coordinate that stores the fact (coord 3) moves the logit -- causal tracing")
print("localizes the computation to that one place, exactly as ROME does in real MLPs.")

Once you have a feature direction and a causal test that it matters, you can steer: add the feature's decoder vector to the residual stream (or clamp its SAE activation to a chosen value) and watch the behavior change. Templeton et al. (2024) made this concrete and a little famous — clamping a "Golden Gate Bridge" feature high made the model fixate on the bridge across unrelated prompts. More usefully, steering can amplify or suppress features tied to safety-relevant behavior, offering a knob that operates on concepts rather than tokens.

Now the honest part, because this is a young field and overclaiming is the main hazard:

• SAE features are not ground truth. They are the output of a particular SAE with a particular width and \(\lambda\). Change those and you get different features. "The model has \(N\) features" is shorthand for "this SAE found \(N\) atoms," not a discovered constant of the network.
• Evaluation is genuinely hard. There is no gold label for "is this feature monosemantic." Proxies (auto-interpretation scores from an LLM judge, activation sparsity, steering specificity) all have failure modes. A feature can look clean on its top activations yet fire spuriously on the long tail.
• Reconstruction is incomplete. Even good SAEs leave a residual; the part of the activation they fail to reconstruct may contain exactly the computation you care about. Recent work also finds SAEs can miss features that probing finds, and can introduce artifacts (feature absorption, composition) that are properties of the SAE, not the model.
• Scaling cost. Training SAEs for every layer of a frontier model, wide enough to resolve rare features, is a major compute and storage undertaking — and the dictionary needed seems to grow with model scale.
• Circuits remain mostly hand-found. Fully reverse-engineering a nontrivial behavior end-to-end is still rare, labor-intensive, and validated case by case. We can read fragments of the algorithm, not yet the whole program.

The honest scorecard. Mechanistic interpretability has moved from "neurons are confusing" to a working pipeline: superposition explains the confusion, SAEs decompose it into features, and causal methods (patching, ablation, steering) test those features for real. That is genuine progress, and at frontier scale it has produced steerable, human-legible concepts that did not exist as tools two years ago. What it has not done is deliver a complete, verified account of any large model's behavior, or a feature set anyone calls canonical. Treat extracted features as useful, falsifiable hypotheses — powerful when they pass a causal test, provisional always.

1. Elhage, N. et al. (2022). Toy Models of Superposition. Anthropic / Transformer Circuits — the superposition hypothesis and feature geometry behind EQ 13.2–13.3.
2. Bricken, T. et al. (2023). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Anthropic Transformer Circuits — the SAE recipe of EQ 13.4–13.5 on a one-layer model.
3. Templeton, A. et al. (2024). Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Anthropic Transformer Circuits — production-scale SAEs and feature steering (§13.4, §13.6).
4. Olsson, C. et al. (2022). In-context Learning and Induction Heads. Anthropic / Transformer Circuits — the induction-head circuit and its link to in-context learning (§13.5).
5. Meng, K., Bau, D., Andonian, A. & Belinkov, Y. (2022). Locating and Editing Factual Associations in GPT (ROME). NeurIPS 2022 — causal tracing / activation patching of EQ 13.6, then weight editing.