Gradient descent: lost in the Margalla fog

Sana and her cousin set off on Trail 5 in the Margalla Hills one December morning. By the time they reach the ridge, the fog has rolled in so thick that they can barely see their own shoes.

Their phones have no signal. They don't know where the path is. But they do know one thing: the car park is at the bottom, so any step downhill is a step in the right direction.

So Sana does something simple. She feels the ground with her foot in every direction, finds where it slopes down most steeply, and takes one step that way. Then she does it again. And again.

An hour later, they walk out at the bottom.

That is gradient descent, the method almost every AI model uses to learn.

The hill is the loss

In Part 1, Bilal the chai apprentice adjusted his recipe to reduce the loss, the number that says how wrong a cup is.

Now imagine every possible recipe as a point on a landscape. The height of the ground at each point is the loss for that recipe. Bad recipes are high peaks. The perfect recipe is the lowest valley.

Training a model means finding the bottom of this landscape. The problem is that a real model has billions of weights, so the "hill" has billions of directions. Nobody can see the whole thing. Like Sana in the fog, the model can only feel the ground right under its feet.

The gradient is the slope under your feet

The gradient is the mathematical name for "which way is steepest, and how steep". Calculus lets the computer work it out exactly for every weight at once. Backpropagation, from Part 1, is the efficient way of calculating it.

Gradient descent then follows a simple rule:

Work out the slope. Take a step in the downhill direction. Repeat.

The size of the step matters

How far should Sana step each time? This is the learning rate, and it is one of the most important settings in all of machine learning.

• Steps too small: Sana is safe, but it takes her all day to get down. A model with a tiny learning rate trains painfully slowly.
• Steps too big: Sana leaps right over the valley and lands on the opposite slope, maybe even higher than before. A model with a huge learning rate bounces around and never settles.
• Just right: big steps early on, smaller careful steps as the ground flattens. Many training methods do exactly this, slowly reducing the learning rate over time.

Feeling the ground with a few stones

Measuring the slope precisely would mean checking the model against every training example before each step. With millions of examples, that is far too slow.

So instead, the model looks at a small random handful (a mini-batch), estimates the slope from those, and steps. It's like Sana feeling the ground at just a few spots instead of surveying the whole hill. The estimate is a bit noisy, but it is fast, and the noise even helps the model shake itself out of small dips. This is called stochastic gradient descent (SGD). Popular variants like Adam also remember which direction they have been moving and keep some of that momentum, like a hiker who doesn't stop dead at every step.

The maths, briefly

Every step moves each weight $w$ against the gradient, scaled by the learning rate $\eta$:

$$w \leftarrow w - \eta \, \nabla_w L(w)$$

With mini-batches, the gradient is averaged over a small random batch $B$ of examples instead of the whole dataset:

$$\nabla_w L \approx \frac{1}{|B|} \sum_{k \in B} \nabla_w \ell(x_k, y_k; w)$$

Adam (Kingma and Ba, 2014) keeps running averages of recent gradients and of their squares, which gives every weight its own adaptive step size.

What to remember

• The loss is a landscape, and training means walking to the lowest point.
• The gradient tells you which way is downhill from where you stand.
• The learning rate is your step size: too small is slow, too big is chaos.
• Real training uses small random batches of examples to estimate the slope quickly.

But reaching the very bottom of the hill on your training data is not always good news. In Part 3, a student learns that the hard way.