Should the Player Fall Ill, Let the Understudy Rise

Welcome back

Last time, we met two tailors and learned about overfitting: a network that fits its training examples so closely, quirks and all, that it fails on anything new. We saw two simple remedies: stop training at the right moment, and gather more data.

Today we’ll learn a few more remedies, gathered under one name: regularisation. And we’ll do it backstage, in a busy theatre.

A company that relies on its star

Imagine a small theatre putting on a play. They have one brilliant lead actor. Over weeks of rehearsal, the whole cast comes to rely on him. He covers for forgotten lines. He fixes awkward scenes on the spot. Everyone else quietly lets him carry the show.

Then, on opening night, he catches a cold.

Suddenly nobody knows what to do. Scenes fall apart. Without their star, the company is lost, because they never learned to stand on their own.

Now imagine a different director. During rehearsals, she sometimes sends a random actor home for the evening. “We’ll rehearse without you tonight,” she says. The others must step up, learn more lines and cover more ground. Every actor becomes more capable, and the play no longer depends on any one person. When someone falls ill on opening night, the show goes on.

Dropout

That second director’s trick has a direct twin in neural networks. It’s called dropout.

During training, on each pass through the loop, the network randomly switches off some of its neurons. A common choice is to switch off about one in five, or even half, in certain layers. Those neurons sit out that round. Next round, a different random handful sits out.

Because no neuron can count on any particular neighbour being present, each one has to learn something useful on its own. The network can’t build a fragile web where one neuron covers up for another’s quirks. Knowledge gets spread out, sturdy and shared.

When training is finished and the network is used for real, dropout is switched off. Every neuron takes the stage. And because they all practised working with gaps, together they perform more reliably on new examples.

Another kind of restraint: keeping weights small

There’s a second, quieter form of regularisation. Let’s go back to the theatre once more.

Some actors overact. They make every line enormous and every gesture huge. It might impress in one scene, but it throws the whole play off balance. A good director gently asks everyone to keep it modest.

In a network, large weights are a bit like overacting. A neuron with a very large weight reacts dramatically to tiny changes in its input, which is often a sign it’s chasing quirks in the training data. So we add a small penalty to the loss for weights that grow too large. This is called weight decay (you may also see it called L2 regularisation). The network now has two goals: be accurate, and stay modest. The result is usually a smoother, more general understanding.

What to carry with you