Lai Learns Neural Networks

Lesson 07·Hiking in fog·4 min read

Of the Wanderer Who Findeth the Valley by Feel Alone

Gradient descent: finding better answers by always stepping downhill

“He could not see three paces before him, and yet he was not lost. He had only to attend, with a patience he had never before possessed, to the faint inclination of the earth beneath his feet. The mountain, it seemed, had been whispering the way down all along.”

Invented, in the spirit of Stefan Zweig

Welcome back

Last time, our archer looked at where her arrow landed. We met the loss function, which gives the network a single number for how wrong its guess was. Learning, we said, means making that number smaller.

But how? A network may have millions of weights. Which ones should change, and in which direction? Today we’ll answer that by going for a walk, in rather poor weather.

Lost on a foggy hillside

Imagine you’re hiking in the hills when a thick fog rolls in. You can barely see your own hands. Somewhere below is a valley with a warm cottage, and you’d very much like to reach it.

You can’t see the valley. You have no map. But you do have one thing: your feet. Standing still, you can feel which way the ground slopes. So you do the sensible thing. You feel for the direction that goes downhill most steeply, and you take a step that way. Then you stop, feel the ground again, and take another step.

Step by step, never seeing more than the ground beneath you, you make your way down into the valley.

That’s the whole idea of gradient descent.

The landscape of loss

Here’s how the picture maps onto a network.

Imagine every possible setting of the network’s weights as a place on a vast hillside. At each place, the height of the ground is the loss: how wrong the network would be with those weights. High ground means bad guesses. Low ground means good ones. The valley at the bottom is where the network performs best.

When a network starts out, its weights are random, so it’s standing somewhere up on the hillside in thick fog. It can’t see the whole landscape. There are far too many weights for that. But it can feel the slope right where it stands.

The gradient: feeling the slope

The slope under your feet has a proper name: the gradient. For each weight, the gradient answers a simple question:

“If I nudged this weight up a tiny bit, would the loss go up or down, and by how much?”

If nudging a weight up would make the loss go up, we should move that weight down. If nudging it up would make the loss go down, we should move it up. The gradient points uphill, so we step the opposite way. That’s what “descent” means.

Each round of learning, then, follows the same calm rhythm:

  1. Make a guess (the forward pass).
  2. Measure how wrong it was (the loss).
  3. Feel the slope for every weight (the gradient).
  4. Nudge every weight a little bit downhill.
  5. Repeat.

Thousands upon thousands of these small steps carry the network from the foggy heights down to a place where its guesses are good.

What to carry with you

In one line

Gradient descent improves a network by feeling which way each weight should change to lower the loss, then taking a small step downhill, over and over.

Next time

We’ll step into the morning shower. Turn the tap too little and you wait forever; turn it too much and you’re scalded. Next time we’ll meet the learning rate: how big each downhill step should be.