Of the Traveller Who Carrieth His Old Skills to Foreign Shores

Welcome back

Last time, in the busy newsroom, we met the Transformer. Every desk reads every page at once, many attention heads each look for their own kind of connection, and the whole process repeats layer after layer.

Those networks are enormous. Training one from scratch can take months and an unimaginable amount of text. So today’s question is a practical one: if you want a network to do something new, must you really start from nothing every time?

Packing your bags

Imagine you’ve just moved from your hometown to a city in another country. Everything feels new: the street signs, the money, the way people queue.

But think about how much you didn’t have to relearn. You already know how to read, how to cook, how to make a friend, how to tell when a shop is about to close. Years of hard-won knowledge came with you in your suitcase.

What you actually need to learn is the small, local part: the bus routes, a few phrases, where the good bakery is. You learn those in weeks, not decades, because you’re building on everything you already have.

Transfer learning

Transfer learning gives networks that same advantage. Instead of starting from random weights, you start from a network that has already been trained on a huge, general task.

Remember the cathedral from Lesson 23? A network trained on millions of everyday photos has already learned edges, textures, shapes and parts of objects in its early and middle layers. Those skills are useful almost everywhere. So if you now want it to recognise, say, different kinds of bread, you don’t have to teach it what an edge is. That came in the suitcase.

The big, general network is often called a pretrained model. It has done its schooling. Now it just needs some local knowledge.

Fine-tuning

The step of learning that local knowledge is called fine-tuning. You take the pretrained network and continue training it, gently, on your smaller, specific set of examples.

A few habits make this work well:

1. Small steps. You use a low learning rate, a gentle turn of the shower tap. You want to adjust the network, not wash away everything it knew.
2. Sometimes, freeze the early layers. Those general skills, like edges and shapes, or grammar and common words, are often left untouched. Only the later, more specialised layers are retrained.
3. Sometimes, add a new final layer. If your task has new categories, such as twelve kinds of bread instead of a thousand everyday objects, you swap in a fresh output layer to match.

Because so much is already learned, fine-tuning can work with hundreds of examples instead of millions, and finish in hours instead of months.

What to carry with you