Learning from Data

For seventy years, making a computer do something meant one thing: a person figures out the rules, writes them down precisely, and the machine follows them. This works beautifully when the rules are knowable — payroll, physics simulations, chess-piece movement. It collapses when they are not. Nobody can write down the rules for recognizing a face, transcribing mumbled speech, or deciding whether an email is spam. We do these things effortlessly, and we cannot say how.

Machine learning inverts the contract. The human supplies three things: a pile of examples of the job done correctly, a flexible function with adjustable numbers inside it, and a score that measures how badly the function currently does the job. The machine's only task is to adjust the numbers until the score improves. The rules are never written by anyone. They condense out of the data, the way a curve condenses out of scattered points.

Take spam. The hand-written version — if subject contains "FREE!!!" then spam — was the actual state of the art in the 1990s, and it aged badly: spammers read the rules too. The learned version is handed two million emails that humans already labeled spam or not spam and tunes itself to agree with those labels. When spammers adapt, you don't rewrite code; you feed in fresh examples and tune again. The maintenance burden moves from logic to data — which is the real reason this paradigm conquered the industry.

The last table row is not a throwaway. A learned system's failures are statistical: it will be wrong on some inputs, with no stack trace pointing at the offending line, because there is no offending line. Knowing how often it is wrong — and on which inputs — is most of the discipline you are about to learn.

To make "adjust the numbers until the score improves" precise, we need names for the pieces. The cleanest setting — and the one this whole volume lives in — is supervised learning: each example is a pair \((x, y)\), where \(x\) is the input and \(y\) is the correct answer, the label. Square footage and sale price. Email text and spam-or-not. A photo and the word "cat". Someone, somewhere, supervised: they supplied the right answers.

A model (the older literature says hypothesis, hence the letter \(h\)) is a function that takes \(x\) and emits a guess for \(y\). What makes it a learnable function is that its behavior depends on adjustable numbers — its parameters, also called weights. The simplest interesting model on Earth has exactly two:

Hold onto the geometry of that sentence: two knobs define a two-dimensional space of candidate lines, and "learning" is a search through that space. Every model in this encyclopedia is the same object scaled up. A frontier language model is a function with roughly \(10^{12}\) knobs instead of two — harder to search, impossible to visualize, but not a different kind of thing. The vocabulary you are acquiring on this page transfers without modification.

An honest caveat before we proceed. Not all learning is supervised. Models can learn structure from unlabeled data (unsupervised), from data that labels itself (self-supervised — how language models pre-train, Vol II Ch 04), or from trial-and-error reward (reinforcement learning). Supervised learning is where the vocabulary is cleanest, and the other regimes reuse nearly all of it.

"Fits the data" must become a number, or the machine has nothing to improve. For one example, the natural measure of failure is the residual — the gap between prediction and truth, \(h(x_i) - y_i\). To grade the model on the whole dataset, square each residual and average:

Now feel it in your hands. Below are 25 measurements from a noisy linear process. Your job is the machine's job: turn the two knobs and drive the disagreement down. The red stalks are the residuals — the exact quantities EQ M1.2 squares and averages.

And here is the same arithmetic with the curtain pulled back — the identical 25 points, two candidate knob settings, scored in four lines of numpy. The second candidate beats the instrument's target; neither is optimal.

import numpy as np

# The exact 25 points behind Instrument M1.1
x = np.array([0.117, 4.055, 2.578, 3.680, 2.475, 2.910, 5.537, 9.212,
              4.253, 4.267, 3.306, 3.224, 6.843, 9.199, 9.605, 5.486,
              7.243, 8.232, 3.124, 8.334, 7.168, 8.049, 5.069, 8.135, 6.093])
y = np.array([3.041, 9.265, 5.475, 8.950, 4.430, 8.188, 14.821, 16.660,
              7.241, 12.262, 6.484, 5.253, 17.360, 19.157, 23.652, 10.813,
              17.723, 17.977, 5.106, 16.549, 17.812, 18.369, 8.996, 20.632, 14.282])

def mse(w, b):                      # EQ M1.2, verbatim
    return np.mean((w * x + b - y) ** 2)

candidates = [(1.0, 0.0), (2.0, 1.0)]
for w, b in candidates:
    print(f"h(x) = {w:.2f}x + {b:.2f}   ->   MSE = {mse(w, b):6.2f}")

plot_scatter(x, y)

Units, briefly. Squaring changes units: if \(y\) is in dollars, MSE is in dollars-squared, which no human can feel. Practitioners report \(\sqrt{\mathrm{MSE}}\) (RMSE) when they want interpretability. And MSE is one loss among many — classification tasks use cross-entropy (Chapter 04), and the freedom to choose the score is a design lever, not a footnote. What never changes: some single number measures disagreement, and learning means pushing it down.

Here is the trap at the heart of the field. If low loss on the examples were the goal, the perfect model would be a lookup table: store every \((x_i, y_i)\) pair, return \(y_i\) when asked about \(x_i\), achieve a loss of exactly zero. It is also perfectly useless — ask it about any \(x\) it hasn't stored and it has nothing to say. Zero training loss, zero learning. Memorization is not the goal. The goal is performance on data the model has never seen. That property is called generalization, and it is the only thing anyone is ever actually paying for.

The defense is almost embarrassingly simple, and it is the single most important habit in machine learning: before doing anything else, split the data. Tune the knobs on one part (the training set) and measure on a part the model never touched (the test set). The held-out score is a rehearsal for the future; the training score is just a record of the past. Formally, the quantity we minimize is a stand-in for the quantity we want:

To see the gap open wide, give a model too much flexibility. The instrument below fits the same 25 points two ways: a straight line (two knobs), and a degree-9 polynomial (ten knobs — enough to snake through nearly every training point individually). Both are fitted to the same 18 training points; 7 points are held out. Watch what each extra knob buys, and what it costs.

Run the same experiment yourself — a 90/10 split this time, fits via np.polyfit. With only 3 points held out the verdict is noisier than the instrument's (small test sets are unreliable juries — that is a real lesson, not an apology), but it points the same way:

import numpy as np

x = np.array([0.117, 4.055, 2.578, 3.680, 2.475, 2.910, 5.537, 9.212,
              4.253, 4.267, 3.306, 3.224, 6.843, 9.199, 9.605, 5.486,
              7.243, 8.232, 3.124, 8.334, 7.168, 8.049, 5.069, 8.135, 6.093])
y = np.array([3.041, 9.265, 5.475, 8.950, 4.430, 8.188, 14.821, 16.660,
              7.241, 12.262, 6.484, 5.253, 17.360, 19.157, 23.652, 10.813,
              17.723, 17.977, 5.106, 16.549, 17.812, 18.369, 8.996, 20.632, 14.282])

perm = np.random.default_rng(0).permutation(len(x))
train, test = perm[:22], perm[22:]      # 90 / 10 split
z = x / 10                              # rescale so degree 9 stays well-conditioned

for deg in (1, 9):
    c = np.polyfit(z[train], y[train], deg)
    mse_tr = np.mean((np.polyval(c, z[train]) - y[train]) ** 2)
    mse_te = np.mean((np.polyval(c, z[test])  - y[test])  ** 2)
    print(f"degree {deg}:  train MSE = {mse_tr:5.2f}   test MSE = {mse_te:5.2f}")

held_out = np.isin(np.arange(len(x)), test).astype(int)
plot_scatter(x, y, held_out)            # blue = the 3 points the fit never saw

Assemble the pieces and you get the universal cadence of machine learning — the loop every chapter in this encyclopedia will replay at larger scale:

In Instrument M1.1, you were the ADJUST box — eyes on the residuals, hands on the sliders. That works for two knobs. It does not work for ten, and it is unthinkable for \(10^{12}\). The entire next chapter is about firing you from the job: calculus can read the slope of the loss surface and announce, for every knob simultaneously, which direction reduces disagreement. That announcement is called the gradient, and following it is called gradient descent — the algorithm that trains essentially everything, from the straight line above to the largest models ever built.

What will change as this volume proceeds: the model grows from a line to a network of millions of units; the loss changes shape for new tasks; the data swells from 25 points to trillions of tokens; ADJUST acquires momentum, schedules, and tricks. What will never change: predict, measure, adjust. When the architecture of Volume II towers over you, find the four boxes. They are always there.

• Mitchell, T. (1997). Machine Learning. — the cleanest formal statement of "learning = improving at a task from experience," and the source of the task/experience/performance framing.
• Hastie, T., Tibshirani, R. & Friedman, J. (2009). The Elements of Statistical Learning (2nd ed.). — the canonical reference for supervised learning, loss functions, and the train/test split.
• Domingos, P. (2012). A Few Useful Things to Know About Machine Learning. — distils the field's hard-won folk wisdom: generalization, overfitting, and "data beats a cleverer algorithm."
• Vapnik, V. (1995). The Nature of Statistical Learning Theory. — the formal account of why minimizing training error is not the same as learning, and what closes the gap.
• Wolpert, D. (1996). The Lack of A Priori Distinctions Between Learning Algorithms. — the "no free lunch" result: no learner is best across all problems, so assumptions are unavoidable.
• Goodfellow, I., Bengio, Y. & Courville, A. (2016). Deep Learning, Ch. 5. — a modern, self-contained primer on the learning-algorithm anatomy: model, loss, optimizer, generalization.