Optimization Basics

Gradient descent, learning rate, and loss functions

Meet the engine that powers every learning algorithm: gradient descent. We follow a model as it slides downhill toward better answers, one careful step at a time.

Before We Begin

What we are learning today

Every time a model "learns," it is really doing optimization: finding the set of weights that make a loss function as small as possible. The workhorse is gradient descent—compute the slope, take a step downhill, repeat. The learning rate controls the size of each step. Too big and you overshoot; too small and you crawl. We also survey common loss functions: mean squared error for regression, cross-entropy for classification, and why picking the right one shapes everything the model learns.

How this lesson fits

Building a model is one thing; training it well and knowing whether it actually works is another. This module covers the engine room of ML: how optimization drives learning, how bias and variance shape model behavior, how to pick the right scoreboard, and how to run experiments you can trust.

The big question

How do we train a model effectively, measure its true performance, and make sure our results are not just a fluke?

Trace a gradient descent step and explain how the learning rate controls itDiagnose overfitting vs. underfitting from training curvesChoose the right evaluation metric for a given problemDesign a fair experiment with baselines and reproducibility

Why You Should Care

Optimization is the literal mechanism of learning. Every neural network, every logistic regression, every fine-tuned LLM relies on some form of gradient descent. Understanding it removes the mystery from "training" and gives students the language to diagnose when learning stalls.

Where this is used today

✓Training neural networks (backprop is gradient descent applied layer by layer)
✓Hyperparameter tuning with learning-rate schedulers
✓Robotics: optimizing control policies in real time

Think of it like this

Imagine you are blindfolded on a hilly landscape and your only goal is to reach the lowest valley. You feel the slope beneath your feet and take a step downhill. That is gradient descent. The size of your stride is the learning rate—sprint and you might leap over the valley; tiptoe and sunset arrives before you get there.

Easy mistake to make

Gradient descent does not guarantee the absolute best answer. It finds a local minimum, which in deep learning is usually good enough—but it is not a magical global optimizer.

By the end, you should be able to say:

Define a loss function and explain what "minimizing loss" means in practice
Trace one step of gradient descent: compute gradient, apply learning rate, update weight
Describe what happens when the learning rate is too high or too low
Compare MSE and cross-entropy and when to use each

Think about this first

If you are lost in fog on a hill and can only feel the ground under your feet, what strategy would get you to the lowest point? What could go wrong?

Words we will keep using

loss functiongradientlearning rateconvergencelocal minimumMSEcross-entropy

The Engine of Learning

Every time a model "learns," it is doing optimization: finding weights that minimize a loss function. The workhorse is gradient descent \u2014 feel the slope, take a step downhill, repeat.

# One step of gradient descent
w = w - lr * \u2207L(w)

Loss FunctionMeasures how wrong the model is. Lower is better.

GradientThe slope of loss w.r.t. each weight. Points uphill \u2014 so we go the other way.

Learning RateStep size. Too big = overshoot. Too small = crawl.

Interactive: Roll Down the Loss Surface

The colored surface is the loss landscape. The white trail is the optimizer's path. Try different learning rates and momentum values.

Learning rate: 0.050Momentum: 0.00

Loss Over Iterations

Watch the loss curve. A smooth descent means stable training. Wild oscillations mean the learning rate is too high. A flat line that's still high means the model is stuck in a local minimum.

Common Loss Functions

Mean Squared Error (Regression)

\text{MSE} = \frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2

Penalizes large errors heavily (squared). Use when outliers matter.

Cross-Entropy (Classification)

L = -\sum_{i} y_i \log(\hat{y}_i)

Penalizes confident wrong predictions heavily. Standard for classification.

Optimizer Variants

SGDVanilla gradient descent. Simple, reliable, sometimes slow.

MomentumAccumulates velocity to roll past flat spots and local minima.

AdamAdaptive learning rate per parameter. The default choice in modern deep learning.

$\text{Adam: } w \leftarrow w - \frac{\eta}{\sqrt{\hat{v}} + \epsilon} \hat{m}$ \u2014 where $m$ is the running gradient mean and $v$ is the running gradient variance.