Welcome back
Last time, we knelt in the garden and learned to sow our weights well: small, random, and scaled to each neuron’s number of inputs. With a good start in place, the network is ready to learn.
Today we look at how it takes its steps. Since Lesson 7 we’ve pictured gradient descent as a careful walker feeling for the downhill direction in fog. Now let’s swap the walking boots for a bicycle.
The walker’s trouble
Our foggy walker has one habit: at every step, she checks the slope under her feet and steps downhill. Nothing more.
That’s honest work, but it has two weaknesses.
First, the ground is bumpy. A small dip or ridge can turn her around, even when the real valley lies straight ahead. She zigzags, wasting steps.
Second, on long, gentle slopes she creeps along. The ground barely tilts, so her steps stay tiny, even though the path has pointed the same way for a long time.
The tools that fix these weaknesses are called optimisers. An optimiser is simply the rule that decides how each weight changes, given its gradient.
Momentum: letting speed carry you
A cyclist rolling downhill doesn’t stop and recheck the slope at every pebble. She has momentum. Her speed from the last few seconds carries her forward, and small bumps barely nudge her off course.
The momentum optimiser gives the network this same habit. Instead of stepping only by today’s gradient, each weight keeps a running memory of its recent steps, a bit like a speed. Each new gradient adjusts that speed a little, and the weight moves by the speed.
The results are lovely:
- When the gradients keep pointing the same way, speed builds and the network glides down long gentle slopes much faster.
- When the gradients flip back and forth across a narrow ravine, the zigzags mostly cancel out, and the path smooths into a straight line.
Adam: a gear for every wheel
Here’s another thing a skilled cyclist does. She changes gears. On steep, rough stretches she shifts so that each push moves her a little. On smooth flats she shifts so that each push moves her a lot.
In a network, different weights live on very different terrain. Some get huge, jumpy gradients. Others get faint, rare ones. A single learning rate for everyone (remember the shower tap from Lesson 8) is a compromise that suits nobody perfectly.
The Adam optimiser gives every weight its own gear. It keeps two running memories for each weight:
- The average direction of its recent gradients. This is momentum, as above.
- The average size of its recent gradients. This shows how rough that weight’s road has been.
Adam then scales each step. Weights with consistently big, noisy gradients take more careful steps. Weights with small, steady gradients take bolder ones. Every weight effectively gets its own learning rate, adjusted as training goes on.
What to carry with you
An optimiser decides how each weight moves. Momentum lets steady progress build up speed and smooths away the zigzags. Adam adds a personal gear for every weight, so each one takes steps that suit its own terrain.