Welcome back
Last time we stood on a soccer pitch and learned about weights: how much a neuron trusts each of its inputs. A neuron multiplies each input by its weight and adds everything up into one total.
That leaves a question we haven’t quite answered. Once a neuron has its total, what does it actually do with it? Today we sit down at a piano to find out.
The piano key that stays silent
Try this, if you ever have a piano nearby. Rest your finger on a key and press down very, very slowly. You’ll feel the key move, but you won’t hear anything. The hammer inside never quite reaches the string.
Press a little harder, and still nothing. Then, at some point, there’s a tipping point. Past it, a note rings out, and the harder you press, the louder it sings.
The piano is doing two things here:
- It has a threshold. Below a certain pressure, nothing happens.
- It shapes the response. Above the threshold, it turns your pressure into sound in its own particular way.
A neuron does both of these things too. The first is called bias. The second is called the activation function.
Bias: how easily the key sounds
Imagine two pianos. One is old and stiff, so you have to press hard before anything happens. The other is beautifully tuned and sounds at the lightest touch.
Bias is the neuron’s version of that stiffness. It’s one extra number that gets added to the total before the neuron decides what to do:
total = (each input × its weight, all added up) + bias
A large positive bias is like the light, eager piano: the neuron speaks up easily. A negative bias is like the stiff one: the inputs must be strong before anything comes through.
Just like the weights, the bias is something the network learns. Every neuron has its own bias, adjusted during training until it’s just right.
The activation function: how the note comes out
Once the neuron has its total, it passes that number through a small rule called the activation function. This rule decides what signal actually leaves the neuron.
The most popular one today is wonderfully simple. It’s called ReLU, and it says:
- If the total is negative, send out zero. Silence. The key wasn’t pressed hard enough.
- If the total is positive, send it out just as it is. The harder the press, the louder the note.
That’s the whole rule. There are other activation functions too. One older favourite, the sigmoid, gently squeezes any total into a number between 0 and 1, like a volume knob that can never go past its limits. But they all share the same purpose.
Why bother with this at all?
This part matters, so let’s take it slowly.
Without an activation function, a neuron could only ever add and multiply. And if every neuron only adds and multiplies, then a whole network of them, however large, behaves like one single, very plain neuron. It could only learn straight-line relationships: double the input, double the output.
The world isn’t like that. A melody isn’t simply louder or quieter; it rises, falls, pauses and resolves. The little bend that an activation function adds, the moment of silence followed by sound, is what lets a network learn rich, curving, surprising patterns. It is a small thing that makes everything else possible.
What to carry with you
Bias sets how easily a neuron speaks up. The activation function decides what it says, and its gentle “bend” is what lets networks learn complicated things.