Neural Networks From Zero: From a Single Number to a Billion Parameters

Almost every explanation of neural networks starts one step too late. It shows you a diagram of circles connected by lines, tells you “these are neurons,” and moves on. But it never answers the question a genuinely curious person asks first:

What are those circles even looking at?

When someone writes input = [0.2, 0.9, 0.1, 0.7], why are there four numbers? Why numbers at all? What if my input is the word “cat”? A photo? A ten-second voice clip? An entire paragraph? How does any of that turn into something a network can process?

This is day zero. If you don’t understand what an input is, what a single neuron does to it, and why each piece of a neuron exists, then everything downstream — attention, transformers, “7 billion parameters,” GPUs running out of memory — is memorization, not understanding. So we’re going to build the whole thing from the smallest possible unit, and by the end you’ll be able to look at any model and compute, by hand, both how big it is and how much math it costs to run.

Let’s start with the single most important sentence in this entire field.

A network only ever sees numbers

Here it is:

A neural network cannot see a word, a pixel, a sound, or a sentence. It can only see a list of numbers. Everything else is a translation problem that happens before the network.

That list of numbers has a name: a vector. When you saw [0.2, 0.9, 0.1, 0.7], that was a vector of length 4 — the network’s entire universe for that one input. The “4” isn’t magic; it’s just how many numbers this particular network was built to accept. A different network might take 784 numbers, or 4096. That fixed count is called the input dimension.

So the real question isn’t “how does a network understand a cat?” It’s: how do we turn a cat — the word, or the photo — into a list of numbers? That translation is where all the interesting representational choices live, and it’s completely separate from the network itself. Let’s do it for every kind of input.

Turning anything into numbers

A single quantity — the easy case. Say you’re predicting whether it’ll rain from today’s temperature. Temperature is already a number: 28.4. Your input vector is length 1: [28.4]. Add humidity and pressure and you have [28.4, 0.66, 1013] — length 3. Each slot in the vector is called a feature. This is the whole reason the classic examples use 3 or 4 numbers: they’re toy problems with 3 or 4 features. Nothing deeper.

A category — a color, a yes/no, a day of the week. These aren’t numbers, and you can’t just assign Monday = 1, Tuesday = 2, because that would trick the network into thinking Wednesday is “twice” Monday, which is nonsense. Instead we use one-hot encoding: one slot per possible value, all zeros except a single 1.

  Monday    → [1, 0, 0, 0, 0, 0, 0]
  Wednesday → [0, 0, 1, 0, 0, 0, 0]
  Sunday    → [0, 0, 0, 0, 0, 0, 1]

Seven categories → a vector of length 7. No false ordering, no fake arithmetic. Just “which one is it.”

A word — where it gets interesting. A word like “cat” is a category too, but the vocabulary has ~50,000 words, so a one-hot vector would be 50,000 numbers that are almost entirely zeros. Wasteful, and worse, it says every word is equally unrelated to every other — “cat” is exactly as far from “dog” as it is from “helicopter.” That’s clearly wrong.

So instead we learn a compact vector for each word, called an embedding — typically a few hundred to a few thousand numbers:

  "cat"        → [ 0.21, -0.44,  0.10, ..., 0.05]   (say, 768 numbers)
  "dog"        → [ 0.19, -0.41,  0.14, ..., 0.08]   ← close to "cat"
  "helicopter" → [-0.88,  0.30, -0.52, ..., 0.77]   ← far from both

The magic: these numbers are learned so that similar meanings end up close together in the vector space. “cat” and “dog” land near each other; “helicopter” lands far away. The network never sees the letters c-a-t — it sees 768 numbers that encode what “cat” means, in a geometry where distance is similarity. (How text gets split into these units in the first place — “tokenization” — and how embeddings are learned are their own deep dives. For today, just hold the shape: word → vector of numbers.)

An image. A grayscale photo is a grid of brightness values, each from 0 (black) to 255 (white). A small 28×28 image is 784 of those numbers. Flatten the grid row by row into one long list and you have a vector of length 784. A color photo has three such grids (red, green, blue), so a 224×224 color image is 224 × 224 × 3 = 150,528 numbers. A picture really is worth a (few hundred) thousand numbers.

Audio (an mp3). Sound is a wave — air pressure changing over time. To digitize it, we measure that pressure many times per second (44,100 times/second for CD quality). Ten seconds of audio is 44,100 × 10 = 441,000 numbers, each a sample of the wave’s height at that instant. A vector, again — just a very long, time-ordered one.

A paragraph. A paragraph is a sequence of words, so it becomes a sequence of embedding vectors — one vector per word (or token), in order. This is the one case where we don’t collapse to a single flat vector, because order matters (“dog bites man” ≠ “man bites dog”). Handling sequences of vectors while respecting order is exactly the problem transformers were invented to solve — but that’s getting ahead of ourselves. The atom is still the same: each word is a vector of numbers.

Here’s the whole picture in one table:

Every single one is just numbers. That is what the circles in the diagram are looking at. Now let’s build a circle.

The neuron: the atom of the whole field

Everything — every model with billions of parameters — is built from one tiny unit repeated a staggering number of times. Understand this one unit completely and the rest is assembly.

The unit is loosely inspired by a biological neuron, and the analogy is genuinely useful. A brain cell receives signals from many others through its dendrites. Some of those connections are strong, some weak. The cell adds up all its incoming signals (weighted by connection strength), and if the total is strong enough, it fires.

An artificial neuron does exactly that, with three ingredients:

In math, that entire picture is one short line:

  y = f( w₁·x₁ + w₂·x₂ + w₃·x₃ + w₄·x₄ + b )
      └─┬─┘ └──────────────┬───────────────┘
   activation        weighted sum + bias

Three ingredients: weights (w), a bias (b), and an activation function (f). Each of them exists for a specific reason, and the fastest way to understand why is to ask what breaks if you remove it.

Why weights? (importance knobs)

A weight is a dimmer switch on each input — a learned number that says “how much should this feature matter?” A large positive weight means “this input strongly pushes the answer up.” A weight near zero means “ignore this input.” A negative weight means “this input pushes the answer down.”

Predicting rain: the humidity input probably deserves a big weight, the day-of-week input a tiny one. The network doesn’t know that in advance — it learns the weights from data. Training a neural network is, almost entirely, the process of nudging billions of these dimmer switches until the outputs come out right.

Remove them — force every weight to 1 — and every input counts equally. The neuron loses all ability to say some features matter more than others. It can no longer learn. Weights are where the learning lives.

Why a bias? (the offset that frees the decision)

The bias is a single number added after the weighted sum, and it’s the ingredient people find most mysterious — until you see the one-dimensional version.

Strip a neuron down to one input: y = w·x + b. That’s the equation of a straight line, exactly the y = mx + c from school. The weight w is the slope; the bias b is the intercept — where the line crosses the axis.

Now remove the bias: y = w·x. You can only make lines that pass through the origin (0,0). You can spin the line to any angle, but it’s forever nailed to that one point. If the pattern in your data lives anywhere else — “predict rain when temperature is above 30” — a line through the origin simply cannot express it. The bias is what lets the neuron slide its decision boundary anywhere, instead of being pinned to zero.

Why an activation function? (the one that makes deep learning possible)

This is the subtle one, and it’s the reason the whole field is called deep learning. The activation f is a function applied to the neuron’s output — historically an S-shaped squash, today usually ReLU, which is almost comically simple: ReLU(z) = max(0, z) — keep positives, zero out negatives.

Why bother? Because a weighted sum plus a bias is a linear operation — it can only draw straight lines and flat planes. And here’s the killer fact: stacking linear operations gives you another linear operation. If layer 2 is just a weighted sum of layer 1, which is a weighted sum of the input, the whole thing collapses algebraically into a single weighted sum. A hundred layers with no activation is mathematically identical to one layer. All that depth buys you nothing.

The activation function inserts a bend — a non-linearity — between layers, and that bend is what stops the collapse. With bends between them, layers can compose: bend, then bend again, then again. Now the network can carve curved, intricate boundaries and represent genuinely complex relationships. The classic proof-by-example is XOR (“true when exactly one input is true”) — a pattern no single straight line can separate, and which becomes trivial the moment you have two neurons and a non-linearity to combine them.

Weights and bias let a neuron draw one straight boundary. The activation is what lets many neurons combine into something that isn’t straight at all. No activation, no depth, no deep learning.

That’s the entire atom: weights decide what matters, the bias frees the boundary from the origin, the activation bends the line so depth means something. Three ingredients, each earning its place.

Why one neuron is never enough

A single neuron draws a single boundary — one line, one plane. Real problems need many. So we do the obvious thing: use lots of neurons. But how we arrange them splits into two independent choices, and understanding the difference between them is understanding the shape of every model.

Many neurons side by side = width (one layer)

Put several neurons in a row, all looking at the same input, each with its own weights and bias. Each learns to detect a different feature of that input. Feed in an image and, side by side, one neuron might learn to spot a vertical edge, another a patch of red, another a curve. They all work in parallel on the same input, and together they produce a new list of numbers — one per neuron — describing the input in terms of the features they detected.

That row of neurons is a layer, and the number of neurons in it is the layer’s width. Width = how many different features you can detect at the same level of abstraction, simultaneously.

Layers behind layers = depth (a stack)

Now the powerful move: take the list of numbers that first layer produced and feed it as input to another layer. The second layer isn’t looking at raw pixels anymore — it’s looking at the features the first layer found, and it learns to combine them into bigger patterns. The number of layers stacked this way is the network’s depth.

This is the famous hierarchy, and it’s genuinely how deep vision networks organize themselves:

  raw pixels
     │
     ▼
  Layer 1  → detects edges & simple colors        ("there's an edge here")
     │
     ▼
  Layer 2  → combines edges into shapes            ("edges here form a curve")
     │
     ▼
  Layer 3  → combines shapes into parts            ("curves here form an eye")
     │
     ▼
  Layer 4  → combines parts into objects           ("eyes + nose + fur → cat")

Each layer builds on the one below, working at a higher level of abstraction. That’s why we go deep: depth is composition. Width gives you breadth of features at one level; depth gives you hierarchy across levels. They are different axes, and a good architecture needs both.

This whole arrangement — an input, one or more hidden layers, an output — is the Multi-Layer Perceptron (MLP), the original neural network and still a building block inside every transformer today.

So how many neurons and how many layers?

The honest answer: there is no formula, and anyone who gives you one is selling something. Width and depth are the network’s capacity — its room to learn patterns. Too little and it can’t capture the problem (underfitting); too much and it memorizes noise, wastes compute, and gets harder to train (overfitting). The right size depends on how complex the true pattern is, how much data you have, and how much compute you can afford.

In practice it’s chosen empirically — start from sizes that worked on similar problems, then adjust. But the rules of thumb are worth knowing: more depth tends to help when the problem is compositional and hierarchical (vision, language); more width helps when there are many independent features to track at once. Modern large models are both very deep and very wide, which is precisely why they have so many parameters — which brings us to the payoff.

From atoms to a model: counting the size

Here’s where everything connects, and where “7 billion parameters” stops being a marketing number and becomes something you can compute yourself.

First, collapse a whole layer into one clean object. A layer is just many neurons, each with a weight per input. Stack their weights and you get a matrix. A layer that takes d_in numbers in and produces d_out numbers out has:

• a weight matrix of shape d_out × d_in — that’s d_out × d_in individual weights, one for every (output neuron, input) pair,
• a bias vector of length d_out — one bias per output neuron.

And the entire layer’s computation — every neuron at once — is a single matrix multiply plus the biases, then the activation:

  y = f( W · x + b )        W is [d_out × d_in],  x is [d_in],  b is [d_out]

So counting a model’s size is now pure bookkeeping. A “parameter” is one single learned number — one entry in a weight matrix, or one bias. That’s all it is. “7 billion parameters” means 7 billion individual numbers that training tuned. To find the total, count the weights in every layer and add:

  parameters in one layer  ≈  d_in × d_out   (+ d_out biases, usually negligible)

  total parameters  =  sum over all layers

Worked example — a tiny 3-layer MLP that takes a 784-pixel image and classifies it into 10 digits:

  Layer 1:  784 → 256      784 × 256   =  200,704 weights
  Layer 2:  256 → 256      256 × 256   =   65,536 weights
  Layer 3:  256 → 10       256 ×  10   =    2,560 weights
  ────────────────────────────────────────────────────────
  total ≈ 268,800 parameters   (plus 522 biases)

That’s it. You just counted a model’s size by hand. Real transformers use the same arithmetic, just with a specific set of matrices per layer. As a rule of thumb, one transformer layer with hidden size d has roughly 12 × d² weights (four matrices for attention plus a wider feed-forward block). So a model with d = 8192 and 80 layers has about:

  80 layers × 12 × 8192²  ≈  64 billion weights   (+ embeddings → ~70B)

There’s your “70B model,” derived from nothing but “a parameter is one number in a matrix, and I can count matrices.”

And the size on disk or in memory follows immediately, because each parameter is just a number stored in some precision. In FP16, one number takes 2 bytes:

  70,000,000,000 params × 2 bytes  =  140 GB

That 140 GB is why a 70B model doesn’t fit on a single 80 GB GPU, why quantization (storing each number in 1 byte instead of 2) is such a big deal, and why “how many GPUs do I need?” has a concrete answer. It all traces back to one number per parameter.

The cost of running it: FLOPs

One last piece, and it’s the ground truth for every performance conversation you’ll ever have. We can count the math a model does just as precisely as we counted its size.

Look again at what one weight does inside W · x: it gets multiplied by its input, and that product gets added into a running sum. One multiply, one add — two arithmetic operations. Engineers call a floating-point operation a FLOP, so:

Every parameter costs 2 FLOPs each time one input passes through the network — one multiply and one add.

For one input (in language models, one “token”) flowing through a model with N parameters, the total work is therefore about:

  FLOPs per token  ≈  2 × N

For our 70B model: 2 × 70,000,000,000 = 140 billion FLOPs to process a single token. Not “a lot” — exactly 140 billion, a number you derived from first principles.

This single formula is the seed of an enormous amount of systems knowledge. It’s why processing a long prompt (many tokens at once, reusing the same weights) behaves completely differently from generating one token at a time (dragging all 140 GB of weights out of memory to do a comparatively tiny 140 GFLOPs of math). That contrast — compute-bound versus memory-bound — is the beating heart of why inference systems look the way they do. But that’s the next deep dive. For now, sit with what you can already do:

What you can now do

Start to finish, with no hand-waving, you can now explain:

• Input — a network only ever sees a vector of numbers, and every input type (number, category, word, image, audio, paragraph) is a translation into that vector. The encoding is where meaning gets packed in.
• The neuron — y = f(w·x + b): weights are learned importance knobs, the bias frees the decision boundary from the origin, and the activation’s bend is what makes depth meaningful.
• Width vs depth — many neurons side by side detect many features at one level; stacked layers compose features into a hierarchy. Both axes, for different reasons.
• Model size — a parameter is one number in a matrix; count the matrices and you have the parameter count; multiply by bytes-per-number and you have the memory.
• Model cost — two FLOPs per parameter per token; 2N for the whole forward pass.

That is the atom, the molecule, the organism, its size, and its running cost — the day-zero ground truth. Every advanced topic — attention, transformers, KV caches, quantization, distributed training — is built on exactly these pieces. When they show up, they won’t be magic. They’ll be assembly.

Next: why generating text one token at a time makes a GPU spend 99% of its time waiting on memory instead of computing — the compute-bound vs memory-bound story that shapes every serving system.