Overfitting and Underfitting

Two ways to be wrong

Imagine studying for a driving test. If you learn only that "green means go," you'll miss lots of situations: that's underfitting. If you memorize the exact 50 practice questions and panic on a new one, that's overfitting.

In machine learning, we want a model that captures the real pattern in data and generalizes to new examples.

• Underfitting: the model is too simple to capture the signal.
• Overfitting: the model captures signal and random noise.
• Good fit: the model learns the underlying structure and performs well on unseen data.

A classic clue is comparing training error and validation/test error:

• High training error + high validation error → underfitting
• Very low training error + much higher validation error → overfitting

The goal isn't to win on training data. It's to do well on tomorrow's data.

Bias, variance, and the shape of the curve

Think of fitting points on a graph. A straight line through curved data is like using a ruler to trace a rainbow: too rigid. A 15th-degree polynomial that wiggles through every point is like drawing squiggles to hit each dot exactly: too flexible.

This is the bias-variance tradeoff:

• High bias models make strong simplifying assumptions → underfitting
• High variance models react too much to small quirks in the data → overfitting

A simple way to measure error is:

error rate = wrong predictions / total predictions

Worked example:

If a model misses 18 out of 100 validation examples,

error rate = 18 / 100 = 0.18 = 18%

As model complexity increases, training error usually goes down. Validation error often goes down at first, then rises once the model starts fitting noise.

How to fix each problem

If a model underfits, make it more capable. If it overfits, make it less fragile.

To reduce underfitting:

• Use a more expressive model
• Add useful features
• Train longer
• Reduce excessive regularization

To reduce overfitting:

• Get more training data
• Use a simpler model
• Apply regularization like L1/L2
• Use dropout for neural networks
• Prune decision trees
• Use early stopping

Here's a tiny scikit-learn example using regularization:

from sklearn.linear_model import Ridge

model = Ridge(alpha=1.0)  # larger alpha = more regularization
model.fit(X_train, y_train)
print(model.score(X_train, y_train))
print(model.score(X_val, y_val))

If alpha is too small, the model may overfit. If it's too large, the model may underfit.

How to spot it in practice

Professionals rarely judge a model from one score. They look at training curves, validation metrics, and whether the pattern makes sense.

A practical checklist:

• Compare training vs. validation performance
• Use a proper train/validation/test split
• Try cross-validation when data is limited
• Tune one thing at a time: depth, degree, alpha, epochs
• Prefer the simplest model that achieves strong validation results

A useful mental image: underfitting is wearing blurry glasses; overfitting is zooming in so far that dust looks like the map. In both cases, you lose the real structure.

The winning model doesn't memorize yesterday's examples. It learns the rule that generated them.