Every AI announcement now name-drops a chip: the model was trained on thousands of GPUs, served on TPUs, and your new laptop has an NPU that apparently does something useful. If you have ever nodded along without being sure what separates them, you are in good company. The short version is that all four are processors, but each one is built around a different bet about what work will arrive. This guide walks through those bets with real numbers I measured on a plain two-core machine, a few small calculations anyone can repeat, and five everyday situations where you can see which chip wins and why.
The one idea behind all four chips
AI models, once you strip away the mystique, spend nearly all their time on one chore: multiplying big grids of numbers and adding up the results. Researchers call it matrix multiplication. A chatbot answering you, a phone sharpening a night photo, a car spotting a pedestrian: underneath, it is the same chore repeated billions of times.
So the useful question is not “which chip is smartest?” None of them are. The useful question is: how many of these multiply-and-add steps can this chip do at once, how fast can it fetch the numbers, and how much power does it burn doing it? Everything that follows is an answer to those three questions.
The short version. A CPU is a few brilliant generalists. A GPU is thousands of simpler workers doing the same step in parallel. A TPU is a custom machine built almost entirely for matrix maths. An NPU is a small, frugal version of that idea that lives in your phone or laptop. No chip is best; the right one depends on whether you are training, serving many users, or running quietly on a battery.
A kitchen to keep in your head
I have explained this to a lot of engineers over the years, and the kitchen picture has never failed me. Picture four ways to feed a stadium.
- CPU: the head chef. Brilliant, fast, can cook anything, follows complicated recipes with lots of decisions. But there is one of them (or a handful). Ask the head chef to chop ten thousand onions and the evening is lost.
- GPU: a thousand line cooks. Each is less skilled, but they all do the same simple job at the same moment. Ten thousand onions disappear in minutes. They are hopeless at improvising a new dish.
- TPU: a conveyor-belt machine. Built for one dish, made in industrial quantity. Nothing else comes off the line, but nothing is faster or cheaper per plate.
- NPU: the compact appliance in your home kitchen. Handles one family of dishes using almost no electricity. It will never feed a stadium, but it will make your breakfast without draining the battery.
The CPU: few workers, great at everything
The central processing unit is the chip that runs your operating system, your browser and nearly every program you have ever used. Modern CPUs have somewhere between a few and a few dozen powerful cores. Each core is a marvel of cleverness: it predicts which way a program will branch, reorders instructions on the fly, and keeps a hierarchy of fast memory nearby so it rarely waits.
All that cleverness costs silicon area and power. A big share of a CPU core is not doing arithmetic at all; it is working out what to do next. That is exactly right for messy, decision-heavy software, and wasteful for a job that is the same simple step repeated a billion times.
The CPU still matters enormously in AI. It loads the data, prepares it, launches the work on other chips and handles everything that is not a big matrix multiply. On small models, or when you only need one answer now and then, a CPU is often perfectly fine.
Measuring the gap: one at a time versus many at once
I do not want you to take any of this on faith, so here is an experiment you can run on any computer. It multiplies two 120 by 120 matrices, which is about 3.5 million arithmetic operations. First the slow, honest way: a plain loop, one operation at a time. Then using numpy, a library that hands the same job to optimised code that processes many numbers per instruction and uses every core.
import time, numpy as np
n = 120
rng = np.random.default_rng(0)
A = rng.random((n, n)); B = rng.random((n, n))
Al, Bl = A.tolist(), B.tolist()
t = time.perf_counter()
C = [[0.0] * n for _ in range(n)]
for i in range(n):
for j in range(n):
s = 0.0
for k in range(n):
s += Al[i][k] * Bl[k][j]
C[i][j] = s
loop = time.perf_counter() - t
t = time.perf_counter()
for _ in range(200):
D = A @ B
vec = (time.perf_counter() - t) / 200
print(f"one multiply, {n}x{n} matrices ({2*n**3/1e6:.1f} million operations)")
print(f" one operation at a time (plain loop): {loop*1000:8.1f} ms")
print(f" many at once (numpy, SIMD + threads): {vec*1000:8.3f} ms")
print(f" speed-up: about {loop/vec:,.0f}x")
print(" same answer:", np.allclose(np.array(C), D))
Output:
one multiply, 120x120 matrices (3.5 million operations)
one operation at a time (plain loop): 227.9 ms
many at once (numpy, SIMD + threads): 0.087 ms
speed-up: about 2,619x
same answer: True
Same answer, same machine, same chip. The only difference is how much was done at once, and it came to roughly 2,600 times faster. This was a two-core computer with no GPU at all, so what you are seeing is not “GPU magic”. It is the general principle of parallelism: when the same step repeats on lots of data, do many at once.
A GPU takes that principle to an extreme. Where my machine did a handful of operations per instruction, a modern GPU has thousands of arithmetic units working together. I did not have one to test, so I will not quote a GPU speed from my own run. The lesson transfers: arrange the work so thousands of lanes stay busy, and the speed-up is dramatic.
The GPU: thousands of simple workers
The graphics processing unit was built to paint pixels. Every pixel on your screen needs a similar small calculation, and there are millions of them, so GPUs grew thousands of small cores that all run the same instruction on different data. Around the late 2000s researchers noticed that neural networks have the same shape: huge piles of identical multiply-and-add steps. They borrowed the graphics hardware, and the deep learning era took off on the back of it.
Two features make GPUs the default for AI. First, raw parallelism: thousands of arithmetic units. Second, very fast memory sitting right beside the chip, which matters more than most newcomers expect, and we will come back to it. Today’s AI-focused GPUs also include dedicated matrix units (NVIDIA calls them tensor cores) that do small matrix multiplications as a single step.
The weakness is flexibility. A GPU is superb when thousands of workers do the same thing and poor when each worker needs to make different decisions. It also needs a CPU to feed it, and it draws a lot of power, so you find it in data centres, workstations and gaming PCs rather than in your pocket.
Why batching is the secret of GPU speed
Here is a second experiment, and in my experience it is the single most useful thing to understand about AI hardware. I made a 2048 by 2048 weight matrix, about 17 MB, and multiplied it by a growing number of input vectors at a time, called the batch.
import time, numpy as np
n = 2048
rng = np.random.default_rng(1)
W = rng.random((n, n), dtype=np.float32)
print(f"weights: {n}x{n} float32 = {W.nbytes/1e6:.0f} MB")
print(f"{'batch':>6} {'ms per call':>12} {'ms per item':>12} {'items/sec':>10}")
for b in (1, 4, 16, 64, 256):
X = rng.random((n, b), dtype=np.float32)
W @ X
reps = 30
t = time.perf_counter()
for _ in range(reps):
W @ X
ms = (time.perf_counter() - t) / reps * 1000
print(f"{b:>6} {ms:>12.2f} {ms/b:>12.3f} {b/ms*1000:>10,.0f}")
Output:
weights: 2048x2048 float32 = 17 MB
batch ms per call ms per item items/sec
1 0.66 0.664 1,506
4 1.53 0.381 2,623
16 1.92 0.120 8,328
64 4.80 0.075 13,324
256 16.43 0.064 15,580
One item at a time managed about 1,500 items a second. Handing the chip 256 items at once pushed it to about 15,600 a second, roughly ten times the throughput from exactly the same hardware. Each call costs more in total, but the cost per item collapses, because the weights are fetched once and reused for every item in the batch.
This explains why AI companies obsess over serving many users together. One person chatting is the worst case for a big chip. A hundred people being batched through the same weights is the best case.
Memory: the part everybody forgets
People compare chips by how fast they calculate. In practice, for the language models behind chatbots, the limit is usually how fast the chip can read the model. To produce one word, the chip has to stream essentially all the model’s weights from memory once. If the weights are 16 GB and memory delivers 100 GB per second, then one word takes at best 0.16 seconds, no matter how many calculators you have.
The ratio that captures this is called arithmetic intensity: how many calculations you perform for each byte you fetch. Low intensity means the chip sits idle waiting for data; this is the “memory-bound” case. High intensity means the arithmetic units are the bottleneck, the “compute-bound” case. The idea comes from a well-known 2009 paper that introduced the roofline model.
# Illustrative round numbers, NOT a specific product.
GB = 1e9
chips = {"laptop-class memory": 100 * GB, "gaming-GPU-class memory": 1000 * GB, "data-centre-GPU-class memory": 3000 * GB}
params = 8e9
for bits, name in ((16, "16-bit"), (4, "4-bit")):
size = params * bits / 8
print(f"8B model at {name}: {size/GB:.0f} GB of weights")
for c, bw in chips.items():
print(f" {c:<30} {bw/GB:>5,.0f} GB/s -> at most {bw/size:>6.1f} tokens/sec for one user")
print()
print("arithmetic intensity of a 4096x4096 layer in 16-bit (FLOPs per byte moved):")
n = 4096
for b in (1, 8, 64, 512):
flops = 2 * n * n * b
bytes_ = 2 * n * n + 2 * n * b * 2
print(f" batch {b:>3}: {flops/bytes_:>6.1f}")
Output:
8B model at 16-bit: 16 GB of weights
laptop-class memory 100 GB/s -> at most 6.2 tokens/sec for one user
gaming-GPU-class memory 1,000 GB/s -> at most 62.5 tokens/sec for one user
data-centre-GPU-class memory 3,000 GB/s -> at most 187.5 tokens/sec for one user
8B model at 4-bit: 4 GB of weights
laptop-class memory 100 GB/s -> at most 25.0 tokens/sec for one user
gaming-GPU-class memory 1,000 GB/s -> at most 250.0 tokens/sec for one user
data-centre-GPU-class memory 3,000 GB/s -> at most 750.0 tokens/sec for one user
arithmetic intensity of a 4096x4096 layer in 16-bit (FLOPs per byte moved):
batch 1: 1.0
batch 8: 8.0
batch 64: 62.1
batch 512: 409.6
Two things jump out. First, for a single user the best possible speed is set by memory bandwidth divided by model size. The numbers above are illustrative round figures, not a specific product, but the pattern is real: faster memory and a smaller model both mean more words per second. Second, look at the intensity table. At batch 1 each byte fetched supports about one calculation, which starves any fast chip. At batch 512 it is over 400, which keeps the arithmetic units busy. That is the same story as my batching experiment, told in a different unit.
This is why high-bandwidth memory is a headline feature of data-centre chips, and why your laptop, with much slower memory, runs a big model so much more slowly than the cloud does.
The TPU: a custom machine for matrix maths
The tensor processing unit is Google’s own chip, designed specifically for neural networks and put to work inside its data centres around 2015, according to the company’s own published papers. Instead of adapting graphics hardware, Google built something around a single idea: a systolic array.
Picture a big grid of tiny multiply-and-add cells. Weights are loaded into the grid and stay put. Input numbers flow in from one edge and march across the grid, rhythmically, like blood pumped through vessels, which is where the word “systolic” comes from. Each cell multiplies, adds its result to what is passing by, and hands it along. A result emerges from the far edge without ever returning to main memory in between.
That last point is the whole trick. The expensive part of computing is not the arithmetic; it is moving data. A systolic array reuses each number many times as it moves through the grid, so it does far more arithmetic per byte fetched. In the language of the last section, it is engineered for high arithmetic intensity.
The trade is flexibility. A TPU is superb at the large, regular matrix operations that dominate neural networks and awkward for anything else. It is also, mostly, something you rent from Google Cloud rather than buy. Other companies have built similar custom accelerators, sometimes called ASICs, because once an industry is spending billions on one workload, a chip built for exactly that workload becomes worth the engineering.
The NPU: AI on a battery
The neural processing unit is the newest name, and it lives in a different world: phones, tablets and laptops. Apple has had a “Neural Engine” in its phones for years, Qualcomm and others put similar blocks into their chips, and recent laptops are marketed around having an NPU, with performance quoted in TOPS, trillions of operations per second. Numbers and requirements change quickly here, so check a current spec sheet rather than trusting any figure from an article, including this one.
The NPU’s bet is efficiency. It uses the same matrix-friendly design as a TPU but at a scale and power budget a phone can afford, often well under a few watts. Photo enhancement, live captions, background blur on a video call, wake-word listening, on-device text suggestions: these are small models running constantly, where battery life matters more than raw speed. Running them on the CPU would be slow and hot; running them on the GPU would use more power than necessary.
How does an NPU get so frugal? Partly by using smaller numbers, which brings us to a trick every one of these chips shares.
Smaller numbers, cheaper chips
A model’s weights can be stored as 32-bit numbers, 16-bit, 8-bit or even 4-bit. Fewer bits mean less memory to fetch, less energy to move it and more calculations per second, because the arithmetic units are smaller and can be packed more densely. Specialised chips lean on this heavily; NPUs in particular are typically built around 8-bit integer arithmetic. The cost is accuracy, so the question is how much you lose.
import numpy as np
rng = np.random.default_rng(2)
n = 512
W = rng.normal(0, 0.05, (n, n)).astype(np.float32)
X = rng.normal(0, 1, (n, 64)).astype(np.float32)
ref = W @ X
def rel(y): return np.linalg.norm(y - ref) / np.linalg.norm(ref) * 100
def quant(W, bits):
q = 2 ** (bits - 1) - 1
s = np.abs(W).max() / q
return np.round(W / s).clip(-q, q), s
print(f"{'format':<10} {'bytes/weight':>12} {'matrix size':>12} {'output error':>13}")
print(f"{'float32':<10} {4:>12} {W.nbytes/1e6:>9.2f} MB {0.0:>12.3f}%")
print(f"{'float16':<10} {2:>12} {W.nbytes/2e6:>9.2f} MB {rel(W.astype(np.float16).astype(np.float32) @ X):>12.3f}%")
for bits in (8, 4):
q, s = quant(W, bits)
print(f"{'int'+str(bits):<10} {bits/8:>12} {W.size*bits/8e6:>9.2f} MB {rel((q*s) @ X):>12.3f}%")
Output:
format bytes/weight matrix size output error
float32 4 1.05 MB 0.000%
float16 2 0.52 MB 0.020%
int8 1.0 0.26 MB 1.120%
int4 0.5 0.13 MB 20.245%
I took a 512 by 512 layer of random weights and rounded it to each format, then measured how far the output drifted from the full 32-bit answer. Half precision changed it by only 0.02 percent while halving the memory. Eight-bit integers cut memory to a quarter at a cost of about 1.1 percent drift, which is usually tolerable. My crude four-bit version, which uses one scale for the whole matrix, drifted by about 20 percent; real four-bit schemes are much smarter, scaling small groups of weights separately, which is how they keep quality acceptable. The pattern is the useful part: each halving of bits saves memory and energy, and past a point the error starts to bite.
This is why the same model can have several versions: a big high-precision one for the data centre and a squeezed one for your phone’s NPU.
Training versus inference: two different jobs
One more distinction clears up a lot of confusion. Training is teaching the model: showing it enormous amounts of data and adjusting its weights, over and over, for days or months. It needs huge parallel compute, lots of memory to hold intermediate values, and fast links between chips, so it happens on big clusters of GPUs or TPUs. Inference is using the finished model to answer a question. It needs far less compute per answer, but it happens constantly, for every user, so cost per answer and delay matter enormously.
Training is a construction project; inference is a restaurant that serves meals all day. They want different hardware tuning, and that is why you see chips marketed for one or the other. NPUs, with rare exceptions, only do inference.
Side by side
| Chip | Built around | Best at | Weak at | Where you meet it |
|---|---|---|---|---|
| CPU | A few powerful, flexible cores | Varied, decision-heavy work; small models; orchestration | Huge parallel maths | Every computer and server |
| GPU | Thousands of parallel cores plus fast memory | Training and large-scale inference | Power use; irregular work | Data centres, workstations, gaming PCs |
| TPU | Systolic arrays for matrix maths | Large, regular neural-network workloads at scale | Anything outside that niche | Google Cloud |
| NPU | Low-power matrix units, often 8-bit | Small models on a battery | Big models; training | Phones, tablets, newer laptops |
Where this shows up in real life
Click through five situations and notice how the best chip changes with the job, the number of users and the power budget.
Training a model
A lab trains a new language model on a huge text collection. What hardware does the work?
What fits. Training means doing the same matrix maths across enormous batches, again and again, so thousands of GPUs or TPUs are linked together, with the data split between them. A CPU cannot keep up, and an NPU is not built for it. The cost is dominated by how many chips you rent and for how long, which is why efficiency gains matter so much.
Chatbot serving
Thousands of people are talking to a chatbot at once. How is it served?
What fits. Each generated word means reading the model’s weights from memory, so bandwidth sets the pace for a single user. Grouping many users into a batch reuses each read for all of them, which is why my batching test went from about 1,500 to 15,600 items a second. Providers also shrink models to 8-bit or lower to fit more work per chip.
Phone night photo
You take a dark photo and the phone brightens and sharpens it in a moment. Which chip runs the model?
What fits. Typically the phone’s neural engine or NPU, working alongside the image signal processor. A small model runs on every shot, so energy per photo matters more than raw speed. If the work landed on the CPU the phone would be slower and hotter. Exact designs vary by phone, so treat this as the general pattern.
Laptop captions
Your laptop transcribes a meeting live and blurs your background on a video call. Where does that run?
What fits. Newer laptops can send such small, steady models to the NPU so the CPU stays free and the battery lasts. But software has to be written to use it, so on many programs the work still runs on the CPU or GPU. Check whether the app you care about actually supports the NPU before buying a laptop for it.
Doorbell camera
A doorbell camera decides whether the movement is a person, a parcel or a cat. Why not send every frame to the cloud?
What fits. Running a small model on the device means instant alerts, lower bandwidth and fewer images leaving the home. It must live within a tiny power budget, so the model is small and uses low-precision numbers. Heavy lifting, such as improving the model, still happens in the cloud on bigger hardware.
How I would choose
When someone asks me which chip they need, I ask four questions in order. The answer usually falls out by the second.
Practical advice
- Start with the CPU you have. For a small model or occasional use, measure first. Many projects buy hardware they never needed.
- Check memory before speed. Before comparing calculation numbers, ask whether the model fits in memory and how fast that memory is. It decides more than the headline figure.
- Batch when you can. If you serve many requests, grouping them is often the cheapest speed-up available.
- Use a smaller number format. Try 8-bit or a good 4-bit version of a model and test the quality on your own task before paying for bigger hardware.
- Match the chip to the phase. Training and inference want different setups; do not price one using the other’s needs.
- Treat quoted performance numbers carefully. TOPS and FLOPS figures depend on number format, and peak numbers are rarely what real software achieves.
Common mistakes and misunderstandings
- “A GPU is just a faster CPU.” It is faster only at parallel work. For decision-heavy code it can be slower.
- “More TOPS always means a better NPU.” The figure depends on the number format, and real speed depends on memory, software support and the model.
- “TPUs replace GPUs.” They compete in some workloads and complement in others, and the software ecosystem around each matters as much as the silicon.
- Ignoring memory bandwidth. A chip with great arithmetic and slow memory will sit waiting when generating text for one user.
- Thinking NPUs train models. They are designed for efficient inference on small and mid-sized models.
- Assuming every program uses the NPU. Software must be written or compiled to target it; otherwise the work quietly lands on the CPU or GPU.
- Judging on peak numbers. Real workloads rarely hit the maximum on the spec sheet.
A mental model that sticks
Think of moving house. A CPU is a skilled mover who can carry anything, up stairs, around corners, and who knows which box is fragile. A GPU is a crew of hundreds who can empty a room of identical boxes in minutes but cannot decide where the piano goes. A TPU is a purpose-built conveyor from door to truck. An NPU is the sturdy trolley you keep in the hallway for the small daily loads. And in all four cases, the real limit is usually not how strong the movers are; it is how fast the boxes can be brought to the door, which is memory.
Practice problems with answers
Try each one before you tap to check.
1. How big are the weights of an 8-billion-parameter model at 4 bits per weight?
About 4 GB, since 8 billion × 0.5 bytes is 4 billion bytes.
2. With 100 GB/s of memory bandwidth, what is the best-case speed for that 4 GB model for one user?
About 25 words (tokens) per second, because 100 divided by 4 is 25.
3. A 70B model at 16-bit needs 140 GB. At 3,000 GB/s of bandwidth, what is the single-user limit?
About 21 tokens per second (3,000 divided by 140), assuming the whole model sits in that memory.
4. In my batching test, about how much did throughput rise from batch 1 to batch 256?
From about 1,506 to 15,580 items a second, a gain of roughly 10 times.
5. Why does batching help GPUs so much?
The weights are fetched from memory once and reused for every item in the batch, so each byte supports many more calculations.
6. Why do NPUs favour 8-bit numbers?
Smaller numbers need less memory, less energy and smaller arithmetic units. In my test 8-bit cut memory to a quarter at about 1.1 percent output drift.
7. Why can a laptop with an NPU still run an app on the CPU?
The software must be written to target the NPU. If it is not, the work falls back to the CPU or GPU.
Test yourself: five quick questions
Tap a question to reveal the answer. The correct option is marked.
What is the core job all four chips accelerate in AI?
- Sorting files
- Drawing menus
- Multiplying and adding big grids of numbers
- Compressing video
Neural networks spend most of their time in matrix multiplication, which is repeated multiply-and-add.
What does batching do for a GPU?
- Makes the model smaller
- Reuses each fetch of the weights for many items
- Turns off cores
- Lowers the precision
Fetching the weights once and using them for many inputs raises arithmetic per byte, and throughput rose about tenfold in my test.
Which limit often caps how fast a chatbot writes for one user?
- Screen size
- Disk colour
- Number of keyboards
- Memory bandwidth
Each word needs a pass over the weights, so bandwidth divided by model size gives the ceiling.
What is a systolic array?
- A grid of multiply-and-add cells that data flows through
- A kind of battery
- A cooling system
- A web protocol
Weights stay in place while inputs flow across the grid, so each number is reused without trips to main memory.
Which chip is typically built for small, always-on models on a battery?
- TPU
- NPU
- CPU only
- Server GPU
NPUs trade peak speed for very low power, usually with 8-bit arithmetic.
Frequently asked questions
What is the main difference between a CPU and a GPU?
A CPU has a few powerful, flexible cores for varied tasks. A GPU has thousands of simpler cores that do the same operation on lots of data at once, which suits AI’s matrix maths.
Is a TPU better than a GPU?
It depends on the workload. A TPU is built for large, regular neural-network maths and is mostly rented through Google Cloud. GPUs are more flexible and widely available. Compare cost and speed on your actual model.
What does an NPU do on my phone or laptop?
It runs small AI models efficiently, such as photo enhancement, live captions, background blur and voice features, using far less power than a CPU or GPU would for the same job.
Do I need an NPU to run AI on my laptop?
No. Many models run on the CPU or GPU. An NPU helps with small, always-on tasks and battery life, and only when the software supports it.
Why do AI chips talk about 8-bit and 4-bit numbers?
Smaller numbers cut memory use and energy and allow more calculations per second. The trade-off is some accuracy, which careful methods keep small.
What do TOPS mean?
Trillions of operations per second, a peak-speed figure usually quoted for 8-bit arithmetic. Real speed also depends on memory, software and the model, so use it only as a rough guide.
Key takeaways
- AI is mostly matrix multiplication, so chips differ in how many multiply-and-add steps they do at once, how fast they fetch data and how much power they use.
- In my test, the same 120 by 120 multiply ran about 2,600 times faster when done in bulk than one operation at a time, on a plain two-core machine.
- Batching 256 items instead of 1 raised throughput roughly tenfold in my test, because the weights are fetched once and reused.
- For chatbots, memory bandwidth often sets the speed limit: an 8B model at 16-bit is 16 GB, so 100 GB/s of memory allows at most about 6 words per second for one user.
- Eight-bit numbers cut memory to a quarter with about 1.1 percent output drift in my test, which is why NPUs and phones lean on them.
- CPU for general work, GPU for training and big serving, TPU for large regular workloads on Google Cloud, NPU for small always-on models on a battery.
Sources and further reading
- Jouppi et al., “In-Datacenter Performance Analysis of a Tensor Processing Unit” (2017), the paper describing Google’s first TPU and its systolic array.
- Williams, Waterman and Patterson, “Roofline: An Insightful Visual Performance Model for Multicore Architectures” (2009), the source of arithmetic intensity and the memory-bound versus compute-bound idea.
- Google Cloud: Introduction to Cloud TPU.
