Baselines, ablations, and reproducibility
Science demands proof, not just results. Learn to run experiments that are fair, repeatable, and actually convincing.
Before We Begin
A result without context is just a number. Did the model improve because of your clever new feature, or because you got lucky with the random seed? This lesson teaches the scientific method applied to ML. We start with baselines—simple models that set the floor—then use ablation studies to isolate which components actually help. We finish with reproducibility: version-controlling data, fixing random seeds, and documenting every choice so anyone can verify your work.
How this lesson fits
Building a model is one thing; training it well and knowing whether it actually works is another. This module covers the engine room of ML: how optimization drives learning, how bias and variance shape model behavior, how to pick the right scoreboard, and how to run experiments you can trust.
The big question
How do we train a model effectively, measure its true performance, and make sure our results are not just a fluke?
Why You Should Care
The ML community has a reproducibility crisis. Papers report results that no one can replicate. Teaching students to run clean experiments from the start builds scientific integrity and saves countless hours of debugging in the future.
Where this is used today
Think of it like this
Think of a science fair project. You would never claim "my plant grew taller because of the music I played" unless you had a control plant with no music, repeated the experiment multiple times, and wrote down every step. ML experimentation follows the exact same logic: control, compare, repeat, document.
Easy mistake to make
A single good result does not prove anything. Without a baseline comparison and repeated runs, you cannot tell if the improvement is real or just noise from a lucky data split.
Think about this first
Your classmate says their model got 92% accuracy. What is the first question you should ask before being impressed?
Words we will keep using
Did the model improve because of your clever new feature, or because you got lucky with the random seed? Experimentation is the scientific method applied to ML: control, compare, repeat, document.
Group A is the control. Group B gets a treatment with a true effect. Adjust the effect size and sample size to see when the difference becomes statistically significant.
An ablation table removes components one at a time and measures the impact. It's the ML equivalent of "which ingredient makes this recipe work?"
| Configuration | Accuracy | Δ from full model |
|---|---|---|
| Full model (BERT + features + augmentation) | 94.2% | — |
| − data augmentation | 92.1% | −2.1% |
| − custom features | 93.5% | −0.7% |
| − BERT (use TF-IDF instead) | 87.3% | −6.9% |
| − all three (baseline) | 85.0% | −9.2% |
BERT contributes the most (\u22126.9%), followed by augmentation (\u22122.1%). Custom features add only 0.7% \u2014 maybe not worth the engineering cost.
does not mean "definitely real." With 20 experiments at , on average one will be a false positive purely by chance. Correct for multiple comparisons or pre-register your hypothesis.