The one-sentence answer

A parameter is a single adjustable number inside a model, and training is the process of finding good values for all of them. When someone says a model has 70 billion parameters, they mean its behaviour is controlled by 70 billion such numbers. Nothing mystical is hidden in there. A model, once trained, is essentially a very large file of numbers plus a fixed recipe for combining them with your input.

The easiest way to feel this is to start at the opposite end, with a model so small that you can hold every parameter in your head.

A model with exactly two parameters

Suppose you want to predict house prices from size. You collect five sales, and you decide to describe the relationship with a straight line: price equals w times size plus b. That is the entire model. The two numbers w and b are its parameters. The first controls how steeply the price rises with size, and the second is a base price when size is zero. Before training, you have no idea what they should be, so both start at zero.

Training means nudging each number in whichever direction reduces the prediction error, over and over. Here is the complete program, which I ran:

# The smallest "model" there is: price = w * size + b   (2 parameters)
sizes  = [4, 6, 8, 10, 12]            # house size in hundreds of square feet
prices = [31, 43, 55, 69, 79]         # price in lakh rupees

w, b = 0.0, 0.0                       # both parameters start as ignorance
lr = 0.01
for step in range(20000):
    gw = gb = 0.0
    for x, y in zip(sizes, prices):
        err = (w * x + b) - y
        gw += 2 * err * x / len(sizes)
        gb += 2 * err / len(sizes)
    w -= lr * gw                      # nudge each knob against its error
    b -= lr * gb

print(f"learned w = {w:.2f}, b = {b:.2f}   (2 parameters)")
print(f"prediction for a 9-hundred sq ft house: {w * 9 + b:.1f} lakh")

Output:

learned w = 6.10, b = 6.60   (2 parameters)
prediction for a 9-hundred sq ft house: 61.5 lakh

The program discovered that each extra hundred square feet adds about 6.1 lakh rupees, with a base of about 6.6 lakh. Its prediction for a 900-square-foot house was 61.5 lakh. Two numbers, learned from five examples, now carry a small piece of knowledge about the housing market. That is all a parameter ever is: a knob whose setting was found by trial and error against data.

Think of the mixing console in a recording studio. A model with two parameters is a console with two sliders. A model with 70 billion parameters is a console with 70 billion sliders, each set by an automated process that listened to the result millions of times and moved the sliders a tiny amount each time.

From one line to a neural network

A straight line can only describe straight-line relationships. To capture richer patterns, such as recognising a face or continuing a sentence, models stack many simple units called neurons into layers. Each neuron takes numbers in, multiplies each by its own weight, adds them up with a bias (a constant nudge), and passes the result through a small bending function. A weight and a bias are both parameters.

The picture below is the smallest network that can learn the classic XOR puzzle, which outputs 1 when its two inputs differ and 0 when they match. A single straight line cannot solve it, but one hidden layer can.

Counting is mechanical. Every connection between two neurons is one weight. The first gap has 2 × 4 = 8 connections, and the second has 4 × 1 = 4. Every neuron that receives input also has one bias: 4 in the hidden layer plus 1 at the output. The total is 8 + 4 + 4 + 1 = 17 parameters. I trained exactly this network:

import numpy as np

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], float)
y = np.array([[0], [1], [1], [0]], float)          # XOR: true when the inputs differ
sig = lambda z: 1 / (1 + np.exp(-z))

def train(seed, steps=5000, lr=0.5):
    rng = np.random.default_rng(seed)               # the starting values are random
    W1, b1 = rng.normal(0, 1, (2, 4)), np.zeros(4)  # 2 inputs -> 4 hidden
    W2, b2 = rng.normal(0, 1, (4, 1)), np.zeros(1)  # 4 hidden -> 1 output
    n = W1.size + b1.size + W2.size + b2.size
    fwd = lambda: (lambda h: (h, sig(h @ W2 + b2)))(np.tanh(X @ W1 + b1))
    loss0 = float(np.mean((fwd()[1] - y) ** 2))
    for _ in range(steps):
        h, p = fwd()
        d2 = (p - y) * p * (1 - p)
        d1 = (d2 @ W2.T) * (1 - h ** 2)
        W2 -= lr * h.T @ d2; b2 -= lr * d2.sum(0)
        W1 -= lr * X.T @ d1; b1 -= lr * d1.sum(0)
    p = fwd()[1]
    return n, loss0, float(np.mean((p - y) ** 2)), p

n, l0, l1, p = train(seed=1)
print("parameters:", n, "(8 + 4 + 4 + 1)")
print("loss before training:", round(l0, 3))
print("loss after training :", round(l1, 4))
for row, out in zip(X, p):
    print(f"  inputs {row.astype(int)} -> {out[0]:.2f}")
ok = sum(train(s)[2] < 0.01 for s in range(20))
print()
print(f"different random starting values: {ok} of 20 runs learned XOR")

Output:

parameters: 17 (8 + 4 + 4 + 1)
loss before training: 0.24
loss after training : 0.0001
  inputs [0 0] -> 0.00
  inputs [0 1] -> 0.99
  inputs [1 0] -> 0.99
  inputs [1 1] -> 0.01

different random starting values: 19 of 20 runs learned XOR

The loss, a measure of how wrong the network was, fell from 0.24 to 0.0001, and the final predictions are almost perfectly 0, 1, 1, 0. Seventeen numbers now implement the XOR rule, and nobody wrote that rule down. Training found it.

The last line of the output deserves attention. Training began from random numbers, and in 19 of 20 attempts the network learned XOR, but in one it got stuck. The starting values matter, and training is a search, not a guarantee. Large models deal with this through careful initialisation and a lot of engineering, but the underlying nature of the process is the same.

How parameter counts are worked out

A fully connected layer with n_in inputs and n_out outputs has n_in × n_out weights and n_out biases. Chain a few layers and add the counts. Consider a simple digit recogniser that looks at a 28 by 28 grid of pixels, which is 784 numbers, passes them through a layer of 128 neurons, and then through a final layer of 10, one for each digit.

def linear(n_in, n_out):
    """A fully connected layer: one weight per connection, one bias per output."""
    return n_in * n_out + n_out

# 1. A small image classifier: 784 pixels -> 128 -> 10 digits
print("digit classifier:", f"{linear(784, 128):,} + {linear(128, 10):,} = {linear(784, 128) + linear(128, 10):,}")

# 2. GPT-2 small, counted from its published shape
d, layers, vocab, ctx = 768, 12, 50257, 1024
per_layer = (4 * d * d + 4 * d)          # attention: query, key, value, output (with biases)
per_layer += (8 * d * d + 5 * d)         # feed-forward: d -> 4d -> d (with biases)
per_layer += 4 * d                       # two layer norms
total = layers * per_layer + vocab * d + ctx * d + 2 * d
print(f"GPT-2 small    : {total:,}  (about {total / 1e6:.0f} million)")

# 3. A 70B-class design (Llama 2 70B shape: grouped-query attention, gated feed-forward)
d, layers, vocab = 8192, 80, 32000
kv_dim, ffn = 8 * 128, 28672             # 8 key/value heads of size 128
attn = 2 * d * d + 2 * d * kv_dim        # query + output, plus smaller key + value
mlp = 3 * d * ffn                        # gate, up and down projections
per_layer = attn + mlp + 2 * d           # plus two norms
total = layers * per_layer + 2 * vocab * d + d
print(f"70B-class model: {total:,}  (about {total / 1e9:.1f} billion)")
print(f"  attention is {layers * attn / total:.1%} of it, feed-forward {layers * mlp / total:.1%}, "
      f"embeddings {2 * vocab * d / total:.1%}")

Output:

digit classifier: 100,480 + 1,290 = 101,770
GPT-2 small    : 124,439,808  (about 124 million)
70B-class model: 68,976,648,192  (about 69.0 billion)
  attention is 17.5% of it, feed-forward 81.7%, embeddings 0.8%

That digit recogniser has 101,770 parameters: 100,480 in the first layer and 1,290 in the second. Notice that most of the parameters sit in the first layer, simply because 784 inputs times 128 outputs is a large product. This pattern repeats at every scale: parameters hide where two big dimensions meet.

Counting a real language model

The same program continues with two real architectures. Modern language models are built from transformers, whose layers each contain two main blocks: an attention block, which lets each word look at the other words, and a feed-forward block, which processes each word on its own. Both are made of big grids of numbers called weight matrices.

For GPT-2 small, a model released in 2019, the published shape is 12 layers, a width of 768, and a vocabulary of 50,257 tokens. Putting those through the counting rules gives 124,439,808 parameters, which matches the 124 million that the model is known by. So the counting really works; there is no hidden accounting.

Then I did the same for a 70-billion-class design, using the architecture numbers Meta published for Llama 2 70B: 80 layers, a width of 8,192, a feed-forward size of 28,672, grouped attention with 8 key-value heads, and a vocabulary of 32,000. The total came out at 68,976,648,192. That is about 69.0 billion, which is what gets rounded and sold as “70B”. Model names are marketing-friendly roundings, so a model called 70B might have 68 or 71 billion in reality.

The breakdown is instructive. About 17.5 percent of the parameters are in the attention blocks, 81.7 percent in the feed-forward blocks, and under one percent in the table that converts words to numbers. If you picture a language model as a giant recall system, the feed-forward blocks are where most of its capacity sits.

What a parameter is not

People pick up some wrong pictures, so let me clear them out.

• It is not a stored fact. No single parameter holds “Paris is the capital of France”. Knowledge is spread across enormous numbers of parameters working together, the way a photograph is spread across millions of pixels. Change one pixel and the picture survives; change one parameter and the model barely notices.
• It is not a neuron. Neurons are the units; parameters are the weights and biases that connect and adjust them. The 17-parameter network had only 5 neurons.
• It is not lines of code. The code that runs a 70B model is short, a few hundred lines. The size is in the numbers, not the instructions.
• It is not a direct measure of intelligence. A newer, well-trained 8B model can beat an older, poorly trained 30B one, because quality depends on the data, the training recipe and the design, not just the count.
• It is not the same as the brain’s connections. You will sometimes see comparisons with the number of synapses in a brain. The two are built and behave very differently, so such comparisons are loose at best.

Why the count decides what hardware you need

Here is the practical consequence, and the reason the number is on every announcement. Each parameter is a number that must be stored, and every one of them is used each time the model produces a word. So size translates directly into memory and computation.

Memory: bytes per parameter

The memory needed to simply hold the model is the parameter count times the bytes used for each parameter. Numbers can be stored with different precision: 32-bit floats take 4 bytes, 16-bit take 2, 8-bit take 1 and 4-bit take half a byte. I worked this out for 70 billion parameters:

import math

params = 70e9
print(f"Storing {params / 1e9:.0f} billion parameters")
for name, nbytes in (("32-bit float", 4), ("16-bit (fp16/bf16)", 2), ("8-bit integer", 1), ("4-bit integer", 0.5)):
    gb = params * nbytes / 1e9
    print(f"  {name:<20}{gb:>6.0f} GB   80 GB cards for the weights alone: {math.ceil(gb / 80)}")

print()
print("Training with Adam in mixed precision, a common recipe (bytes per parameter):")
parts = [("16-bit weights", 2), ("16-bit gradients", 2), ("32-bit master weights", 4),
         ("Adam momentum", 4), ("Adam variance", 4)]
for n, b in parts:
    print(f"  {n:<24}{b}")
total = sum(b for _, b in parts)
print(f"  {'total':<24}{total}   ->  {params * total / 1e12:.2f} TB before any activations")

Output:

Storing 70 billion parameters
  32-bit float           280 GB   80 GB cards for the weights alone: 4
  16-bit (fp16/bf16)     140 GB   80 GB cards for the weights alone: 2
  8-bit integer           70 GB   80 GB cards for the weights alone: 1
  4-bit integer           35 GB   80 GB cards for the weights alone: 1

Training with Adam in mixed precision, a common recipe (bytes per parameter):
  16-bit weights          2
  16-bit gradients        2
  32-bit master weights   4
  Adam momentum           4
  Adam variance           4
  total                   16   ->  1.12 TB before any activations

At 16-bit precision, the common format, the weights alone weigh 140 gigabytes. No single 80 GB graphics card can hold that, so you need at least two. Shrink each parameter to 8 bits and it just about squeezes onto one card; shrink to 4 bits and it takes about 35 gigabytes. That shrinking is called quantisation, and it is the reason that large models can run on much more modest hardware. The trade-off is a small loss in accuracy, which is usually acceptable at 8 bits and noticeable more often at lower precision.

Notice the caveat in the output: those figures are for the weights alone. Running the model also needs working memory for the conversation, which grows with the length of the text, so real deployments need headroom.

Training is far hungrier. A common recipe stores, for every parameter, a 16-bit copy, a 16-bit gradient, a 32-bit master copy and two 32-bit running statistics for the optimiser. That is 16 bytes per parameter, which for 70 billion comes to about 1.12 terabytes, before counting the temporary values computed during the forward pass. This is why training happens across racks of GPUs, while running a finished model can sometimes be done on one.

Computation: operations per word

The second cost is arithmetic. A widely used rule of thumb says that generating one token needs about two operations per parameter, and training needs about six operations per parameter per token of training data. Here is what that gives:

params, tokens = 70e9, 2e12                 # 70 billion parameters, 2 trillion training tokens
train_flops = 6 * params * tokens           # rule of thumb: about 6 operations per parameter per token
per_token = 2 * params                      # generating one token: about 2 operations per parameter
print(f"training work : {train_flops:.2e} operations")
print(f"one new token : {per_token / 1e9:.0f} billion operations")

peak, utilisation = 312e12, 0.40            # an A100-class GPU peak in 16-bit, and a realistic share of it
hours = train_flops / (peak * utilisation) / 3600
print(f"on one GPU at {utilisation:.0%} of {peak / 1e12:.0f} TFLOP/s: {hours / 1e6:.2f} million GPU-hours")
print(f"spread over 2,000 GPUs: about {hours / 2000 / 24:.0f} days")
print()
print(f"tokens per parameter in this recipe: {tokens / params:.0f}")

Output:

training work : 8.40e+23 operations
one new token : 140 billion operations
on one GPU at 40% of 312 TFLOP/s: 1.87 million GPU-hours
spread over 2,000 GPUs: about 39 days

tokens per parameter in this recipe: 29

Generating a single token with a 70B model takes roughly 140 billion arithmetic operations. Training on 2 trillion tokens needs around 8.4 times ten to the 23rd operations. On one A100-class GPU, using realistic assumptions of 40 percent of its peak speed, that is about 1.9 million GPU-hours, or roughly 39 days if you spread it over 2,000 GPUs. For comparison, Meta reported a figure in the same ballpark, about 1.7 million GPU-hours, for training the 70B model, so the rule of thumb lands close to reality. It is also why training a model is a headline-grabbing investment, while using it is comparatively cheap.

One last number in the output is worth pausing on: tokens per parameter. A 2022 study from DeepMind suggested that, for a fixed compute budget, models were being made too big and fed too little text, and recommended roughly 20 training tokens per parameter as a good balance. Their 70B-parameter model trained on 1.4 trillion tokens beat much larger models that had seen less text. The lesson is that parameter count is only half of the story. The other half is how much data the parameters were tuned against.

Size words you will meet

The mixture-of-experts row catches many people out. In such a model, the layers contain several specialist blocks and a small router chooses a few of them for each word. The full set of experts must sit in memory, so the memory bill follows the total parameter count, but only the chosen experts do any arithmetic, so the speed follows the active count. A model can therefore be large to store but quick to run.

Model sizes in real life

Different jobs call for very different sizes. Click through five everyday situations and notice that bigger is not always the right answer.

How to choose a size

When people ask me which model size they need, I do not start from the parameter count. I start from the job and the hardware. A short sequence of questions settles most cases.

Common mistakes and misconceptions

• Assuming more parameters always means a better model. Data quality, training length and design matter as much. Compare actual results on your task.
• Confusing parameters with training data. Parameters are the learned numbers; training data is the text the numbers were tuned on. Both are measured in billions, in different units.
• Forgetting the precision. “70B” says nothing about file size until you know the bytes per parameter. The same model is 140 GB at 16-bit and 35 GB at 4-bit.
• Ignoring working memory. Weights are the minimum. Long conversations need extra room on top.
• Treating the headline number as exact. Names are rounded; the real count of my 70B-class calculation was 69.0 billion.
• Mixing up total and active parameters. A mixture-of-experts model can quote a big total but compute with a fraction.
• Expecting one parameter to hold one fact. Knowledge is distributed across many parameters at once.
• Using a giant model for a tiny task. A spam filter does not need 70 billion parameters, and paying for them wastes money and energy.

A mental model that sticks

Picture an orchestra of seventy billion musicians, each holding a single instrument that can play only one note at a time, loud or soft. No individual musician plays a melody, and none of them knows the song. Yet when each one adjusts their volume to exactly the right amount, the combined sound becomes music. Training is the long rehearsal in which a conductor, listening to the result, tells every musician to turn up or down by a hair, repeated trillions of times. The “70 billion” is the head count of the orchestra, and the size of the concert hall you need follows directly from it.

Practice problems with answers

Try each one on paper before you tap to check.

Test yourself: five quick questions

Tap a question to reveal the answer. The correct option is marked.

Frequently asked questions

The post What Does “70 Billion Parameters” Actually Mean? appeared first on Learn With Examples.

The post What Are Tokens and Context Windows? appeared first on Learn With Examples.