What a neuron actually computes
Skip past the biology metaphor. A neuron in a neural network is not a simulated brain cell, it is two arithmetic operations, applied one after the other. First, a weighted sum: take every input number, multiply each by its own weight, add them all up, then add one more number called the bias. Second, pass that sum through a small, fixed function called an activation function. That's the entire computation. Nothing hidden, nothing mystical.
One neuron, three inputs
1inputs = [1.0, 0.5, -1.0]2weights = [0.6, -0.8, 0.3]3bias = 0.045weighted_sum = sum(i * w for i, w in zip(inputs, weights)) + bias6# (1.0 * 0.6) + (0.5 * -0.8) + (-1.0 * 0.3) + 0.0 = 0.6 - 0.4 - 0.3 = -0.178output = max(0, weighted_sum) # ReLU activation9# max(0, -0.1) = 0
The weights and the bias are the only numbers this neuron actually owns, and they are exactly what training adjusts. Everything a network “learns” is a change to these weight values, nothing else. A trained model with billions of parameters is, at the arithmetic level, billions of numbers exactly like the 0.6, -0.8, and 0.3 above.
Rule: a neuron is weighted-sum-then-activation. Nothing about the word “neural” implies anything more sophisticated is happening underneath a single unit. Sophistication comes from how many of these are wired together, not from any one of them individually.
Why the activation function isn't optional
It's reasonable to wonder why the weighted sum alone isn't enough, why bother squashing it through another function afterward. Here's the problem stacking creates if you skip that step: stacking two purely linear layers, weighted sum feeding directly into another weighted sum with no activation in between, produces something that is mathematically still just one linear layer. No amount of stacking helps.
Two linear layers collapse into one
1def layer(x, w, b):2 return w * x + b34x = 45layer1 = layer(x, w=2, b=3) # 2*4 + 3 = 116layer2 = layer(layer1, w=-1, b=5) # -1*11 + 5 = -678# Same result from a single combined layer:9combined_w = 2 * -1 # -210combined_b = -1 * 3 + 5 # 211single_layer = combined_w * x + combined_b # -2*4 + 2 = -6
Two layers, four numbers, and it still only computes a straight line. Stack a hundred linear layers and you still only get a straight line, just with different slope and offset. A straight line cannot represent most of what language requires: whether a sentence is sarcastic, whether a pronoun refers back three sentences or one, whether a number is odd. Those relationships are not straight lines through the data.
The activation function is what breaks the collapse. Because it bends the output, ReLU clips every negative value to zero, sigmoid squashes everything into a 0 to 1 curve, stacking layers with an activation in between actually builds something new at each layer, not a repackaged version of the first one. This is the entire reason depth in a “deep” neural network does anything at all.
In practice: modern transformers mostly use GELU or SwiGLU rather than plain ReLU or sigmoid, smoother variants that tend to train better at scale. The specific curve differs, but the reason one exists at all is exactly the collapse argument above.
Watch a neuron compute
Adjust the three inputs and their weights below and watch the weighted sum, labeled z, update live. Then switch between activation functions and watch the same z produce a different final output, and notice the shape of each curve: linear never bends, ReLU flattens everything negative to exactly zero, sigmoid and tanh both squash extreme values toward a ceiling and floor instead of letting them grow without bound.
x-axis: z (the weighted sum). y-axis: activation(z). The dot marks this neuron's current output.
That squashing matters beyond this one neuron. Feed a huge positive or negative number through sigmoid and the output barely changes no matter how much larger the input gets, the curve is nearly flat out there. That flattening is exactly why sigmoid fell out of favor in deep networks: a neuron whose output barely moves also barely teaches the network anything useful during training, a problem covered directly in the Loss and Backpropagation article.
From one neuron to a layer, from a layer to a matrix
A real network doesn't use one neuron, it uses many neurons side by side, each looking at the exact same inputs, each with its own independent set of weights. That group is called a layer. One neuron might learn to respond to something like negation, a different one in the same layer to something like tense, a third to something with no clean human name at all, purely because it helped the training objective.
Computing every neuron in a layer one at a time, in a loop, is exactly what a matrix multiplicationdoes in a single operation. Stack every neuron's weights as a row of a matrix, multiply that matrix by the input vector, and the result is every neuron's weighted sum, all at once. This is not an approximation of what a layer does, it is a literal restatement of the same arithmetic in a form GPUs can execute extremely fast, thousands of neurons at a time.
A layer of 2 neurons, as a loop and as a matrix multiply
1inputs = [1.0, 0.5, -1.0]23# As a loop, one neuron at a time4neuron_1_weights = [0.6, -0.8, 0.3]5neuron_2_weights = [0.1, 0.4, -0.2]6out_1 = sum(i * w for i, w in zip(inputs, neuron_1_weights))7out_2 = sum(i * w for i, w in zip(inputs, neuron_2_weights))89# As one matrix multiply, identical result10import numpy as np11weight_matrix = np.array([neuron_1_weights, neuron_2_weights])12outputs = weight_matrix @ np.array(inputs)
Stack several of these layers, each one's output feeding the next layer's input, activation functions between them, and that's a multilayer perceptron, the feedforward network embedded inside every transformer block. It is also, structurally, the entire machinery: everything from here through the end of this series, attention, transformer blocks, the whole model, is built from exactly these two operations, matrix multiplication and activation functions, arranged in different configurations.
In practice: when a paper says a model has “7 billion parameters,” it means the combined weight matrices and biases across every layer contain 7 billion individual numbers, each one adjusted during training, each one contributing to weighted sums exactly like the ones above.
What's still missing is how a network decides which earlier words in a sentence actually matter to the word it's currently processing. A plain feedforward layer treats every input position the same way every time, it has no mechanism for looking back at a sentence and deciding "that pronoun refers to this noun." That mechanism, built from the same matrix multiplication covered here plus the cosine-similarity-style comparison from the embeddings article, is attention, covered next.