← Back to Curriculum
Training & Evaluation·30 min·Intermediate
⚖️

Bias & Variance

Overfitting, underfitting, and regularization

Every model walks a tightrope between memorizing the training data and failing to learn enough. Here we learn to spot the fall and catch it with regularization.

Before We Begin

What we are learning today

Bias and variance are two sides of the same coin. A high-bias model is too simple—it underfits, missing the real pattern like a student who skims the textbook. A high-variance model is too complex—it overfits, memorizing every quirk of the training data like a student who crams specific answers instead of understanding concepts. Regularization techniques like L1, L2, dropout, and early stopping add a gentle penalty that keeps the model honest.

How this lesson fits

Building a model is one thing; training it well and knowing whether it actually works is another. This module covers the engine room of ML: how optimization drives learning, how bias and variance shape model behavior, how to pick the right scoreboard, and how to run experiments you can trust.

The big question

How do we train a model effectively, measure its true performance, and make sure our results are not just a fluke?

Trace a gradient descent step and explain how the learning rate controls itDiagnose overfitting vs. underfitting from training curvesChoose the right evaluation metric for a given problemDesign a fair experiment with baselines and reproducibility

Why You Should Care

The bias-variance trade-off is the single most important diagnostic framework in ML. Students who understand it can look at a training curve and immediately prescribe a fix: more data, simpler model, or regularization. It prevents the most common beginner mistake—chasing higher training accuracy while real-world performance crumbles.

Where this is used today

  • ✓Medical models: ensuring a diagnosis tool generalizes to new hospitals, not just the training hospital
  • ✓Finance: preventing a trading model from fitting to historical noise
  • ✓Autonomous vehicles: making sure a perception model handles unseen road conditions

Think of it like this

Think of studying for an exam. If you barely read the material, you will fail because you did not learn enough (underfitting). If you memorize every example word-for-word but never understand the principles, you will freeze on any new question (overfitting). The sweet spot is deep understanding without rote memorization—and regularization is the study strategy that keeps you there.

Easy mistake to make

Overfitting is not "the model is too good." It means the model learned patterns that exist only in the training data and do not generalize. A truly good model scores well on data it has never seen.

By the end, you should be able to say:

  • Define bias and variance in plain, intuitive language
  • Read training vs. validation curves to diagnose overfitting and underfitting
  • Explain how L1, L2 regularization, dropout, and early stopping reduce variance
  • Describe the trade-off and why you cannot minimize both to zero simultaneously

Think about this first

A model scores 99% on the training set but only 60% on new data. What is going on, and what would you try first?

Words we will keep using

biasvarianceoverfittingunderfittingregularizationdropoutearly stopping

The Tightrope

Every model walks a tightrope between underfitting (too simple, misses the pattern) and overfitting (too complex, memorizes noise). The bias-variance tradeoff is the diagnostic framework that tells you which side you're falling toward.

High BiasToo simple. Underfits. Like skimming the textbook \u2014 misses the real pattern.
Just RightCaptures the signal, ignores the noise. Generalizes well.
High VarianceToo complex. Overfits. Memorizes every quirk instead of understanding.

Interactive: Watch Overfitting Happen

The true relationship is quadratic. Orange dots are training data, green dots are test data. Increase the polynomial degree and watch the model wiggle to fit training noise while test error explodes.

\u25cf Train (MSE: 5.6)\u25cf Test (MSE: 6.1)

Low degree \u2014 the model is too simple (high bias / underfitting).

The U-Curve

As complexity increases, training error always goes down. But test error follows a U-shape: it decreases, hits a sweet spot, then climbs back up as the model starts memorizing noise.

\u2014 Train MSE\u2014 Test MSE

Regularization: Keeping the Model Honest

Regularization adds a penalty for complexity, forcing the model to stay simpler even when it has the capacity to overfit. It's the study strategy that prevents rote memorization.

L1 (Lasso)L+λ∑∣wi∣L + \lambda \sum |w_i|

Drives some weights to exactly zero \u2014 automatic feature selection.

L2 (Ridge)L+λ∑wi2L + \lambda \sum w_i^2

Shrinks all weights toward zero \u2014 no feature is completely dropped.

Dropout

Randomly disables neurons during training. Forces the network to not rely on any single pathway.

Early Stopping

Monitor validation loss. Stop training the moment it starts rising \u2014 before overfitting sets in.