Regularization Techniques

Why regularization exists

Imagine hiring a student to predict house prices. One student memorizes every house in the practice set; another learns the broad patterns: bigger homes cost more, downtown locations raise prices. On new houses, the pattern-learner wins. Regularization pushes machine learning models toward that second behavior.

When a model is too flexible, it can fit noise instead of signal. This is called overfitting: low training error, disappointing test error. Regularization adds a penalty for complexity, discouraging extreme parameter values.

For linear regression, we often minimize:

loss = training error + penalty

That penalty says: “Fit the data, but don’t use wildly large weights unless they truly help.” The result is usually a slightly worse fit on training data, but a better fit on unseen data—exactly what we want in practice.

L2 and L1: two popular ways to penalize complexity

The two classic regularizers are L2 and L1.

• L2 regularization adds the squared size of weights: lambda * sum(w_i^2)
• L1 regularization adds the absolute size of weights: lambda * sum(|w_i|)

For weights [3, -4] and lambda = 0.1:

• L2 penalty = 0.1 * (3^2 + (-4)^2) = 0.1 * 25 = 2.5
• L1 penalty = 0.1 * (|3| + |−4|) = 0.1 * 7 = 0.7

L2 tends to shrink weights smoothly, spreading importance across features. L1 tends to push some weights all the way to zero, effectively performing feature selection.

A useful analogy: L2 is like telling every team member to speak more softly; L1 is like asking some people to stop talking entirely.

Choosing the strength: the role of lambda

Regularization is controlled by a hyperparameter, often called lambda or alpha. It determines how much we punish large weights.

• If lambda = 0, there is no regularization
• If lambda is too small, overfitting may remain
• If lambda is too large, the model may become too simple and underfit

Suppose a polynomial model predicts exam scores. With lambda = 0, it twists to match every student's noisy result. With lambda = 100, it may flatten so much that it misses real trends. Good practice is to try several values and compare validation performance.

from sklearn.linear_model import Ridge

model = Ridge(alpha=1.0)  # alpha is lambda-like
model.fit(X_train, y_train)
print(model.score(X_val, y_val))

The best regularization strength is usually found by experimentation, not guessing.

Beyond linear models: regularization as a general habit

Regularization appears across machine learning, not just in linear regression. In neural networks, weight decay is essentially L2 regularization. Dropout randomly turns off neurons during training, preventing the network from relying too heavily on any one path. Early stopping halts training when validation performance stops improving, acting like a practical regularizer.

The unifying idea is simple: models should earn complexity, not get it for free. If two models explain the data similarly well, prefer the simpler one.

When you see a gap like “training accuracy 99%, validation accuracy 84%,” think regularization. It is one of the first tools to reach for when a model looks brilliant in rehearsal but shaky on opening night.