I have been building things on top of language models since the days when “context window” meant 512 fragments and you budgeted them like a family budgets grocery money. The hardware got better. The models got enormous. And yet, in workshop after workshop, the same two questions come up first — and they are almost always asked in the wrong order.

People ask “why did the AI forget what I told it?” before they ask “what exactly is the AI storing in the first place?” That is like asking why your suitcase won’t close before checking what you packed. So let’s do it properly. First we’ll open up a token — see it, split it, count it. Then we’ll build the window it lives inside, and watch what happens when the window fills up.

By the end you’ll be able to look at a prompt and estimate its cost in your head, explain why the model insists “strawberry” has two Rs, know why Hindi or Tamil text costs nearly triple what English does, and — most usefully — know exactly what to trim when a system tells you your input is too long.

Part 1 — A token is not a word

Here is the first mental model most beginners build, and it is wrong: “the AI reads word by word, like I do.” Reasonable guess. Also the source of about half the confusion I see.

Think about why splitting on words fails. English alone has well over a million distinct word forms once you include names, typos, plurals, brand names, hashtags, and the thing your colleague types when they mean “definitely.” A model would need a lookup table with a million-plus entries, and the moment somebody typed Kanniyakumari or ngrok or bruhhhh, the table would shrug and return “unknown.”

Now try the opposite extreme: split on characters. Only about a hundred symbols to memorise — clean, complete, nothing is ever unknown. But now the word “encyclopedia” costs twelve slots instead of one, and the model has to learn spelling from scratch before it learns meaning. Sequences get impossibly long, and long sequences are expensive (we’ll get to why).

So the field landed in the middle. Subword tokenization: keep whole tokens for the text fragments that show up constantly, and break rare things into smaller pieces that can be reassembled. Common stuff is cheap. Weird stuff still works, just costs more.

Watch it happen

Reading about tokenization is far less convincing than watching a sentence get shredded. Type into the box below — or hit one of the preset buttons — and see the fragments appear. Each coloured block is one token. Notice the little spaces sitting inside the blocks: in most tokenizers a leading space is part of the token, which is why the and the are two different entries entirely.

How the splitting rules get invented

Nobody sits down and hand-writes the list of fragments. It’s learned from data, most often with an algorithm called Byte Pair Encoding (BPE). The idea is almost embarrassingly simple, and you can run it on paper.

Suppose your entire training corpus is these three words, repeated a lot:

Five rounds and the algorithm has independently discovered the root low and the suffixes er and est — real English morphology, learned purely by counting. Run this for millions of rounds on a slice of the internet and you get a vocabulary of roughly 50,000 to 200,000 fragments, ordered by how useful they are.

Two consequences fall straight out of this, and both matter in practice:

• Frequency decides price. Text that looked like the training data gets packed efficiently. Text that didn’t gets shredded into crumbs. Your name is probably one token if you’re called David and five tokens if you’re called Devanshi.
• The vocabulary is frozen at training time. A model trained mostly on English carries an English-shaped tokenizer forever. It can still read Hindi, Swahili, or Korean — it just pays a heavier toll per sentence.

Part 2 — Token maths you can do in your head

You will rarely need an exact count. You will constantly need a fast estimate: will this fit? These are the ratios I have used for years, and they hold up well enough for planning.

Let’s apply them to things you actually handle:

Part 3 — The five places tokenization gets weird

This is the part of the article I’d tell my past self to read first. Almost every “the AI is stupid” complaint I have investigated turned out to be a tokenizer artefact, not a reasoning failure.

1. The model cannot reliably count letters

Ask a model how many times the letter R appears in “strawberry” and you may get “two.” People post screenshots of this as proof that AI is a parlour trick. What actually happened is subtler: the model never received the letters. It received something like str + aw + berry — three opaque ID numbers. Asking it to count Rs is like asking you to count the letters in a word someone spelled out in Morse code, at speed, without writing anything down. Not impossible, but a genuinely awkward task given the input.

2. Numbers split in unhelpful places

Depending on the tokenizer, 148392 might become 148+392, or 1+48+39+2. The digits do not line up in neat place-value columns the way they do on paper, which is a large part of why models were historically shaky at long arithmetic. It also means an invoice number, a PAN, an order ID, or a phone number can be far more expensive than its length suggests — and is easier to mis-copy.

3. A trailing space quietly poisons your prompt

If your prompt ends with "The answer is " — with a space after “is” — you have handed the model a dangling space token and asked it to continue. But the natural next token is " 42", which already includes its own leading space. You’ve created a conflict the model must work around, and quality drops for no visible reason. Fifteen years in, and I still occasionally lose twenty minutes to a stray space at the end of a template string.

4. Whitespace is a real line item in code

Deeply indented code, four-space tabs, blank lines between functions — every bit of that costs tokens. A well-formatted 500-line Python file can cost noticeably more than the same logic minified. This is not an argument for writing ugly code; it is an argument for sending the model the three functions that matter instead of the whole file.

5. Rare and mixed-script text costs a fortune

Scientific nomenclature, chemical formulas, Sanskrit terms, transliterated place names, base64 blobs, UUIDs, and minified JavaScript all tokenize horribly. A single UUID can eat 20+ tokens. If you are pasting logs full of them, you are burning most of your budget on strings the model does not need to read carefully.

Every unusual character you send is a small tax. Every ordinary English word is a discount. That asymmetry is baked into the tokenizer, and no amount of prompt engineering removes it.

Part 4 — The context window: the desk, not the brain

Now the second half. You know what tokens are; the question is how many the model can look at simultaneously.

My favourite analogy comes from a colleague who used to work in an architecture office. Picture a draughtsman at a drawing board. The board is a fixed size. Everything he needs — the client brief, the site survey, last week’s revisions, the sheet he is drawing on right now — has to be laid out on that board at once. Anything that doesn’t fit goes on the floor. And here is the crucial part: he cannot remember what’s on the floor. Not “remembers it vaguely.” Cannot see it at all.

That board is the context window. It is measured in tokens, and it holds four things simultaneously:

Read that diagram twice, because it contains the single most misunderstood fact about chatbots: the conversation history is re-sent, in full, on every turn. The model is not sitting there remembering you between messages. It is stateless. Each time you press enter, the application quietly bundles up the entire transcript so far, appends your new message, and ships the whole parcel off again. That is why turn 40 of a conversation costs far more than turn 2, and why a long chat gradually slows down.

Watch a conversation fall off the edge

The simulator below runs a customer-support chat. Drag the slider to shrink or grow the window and watch which messages survive. The system prompt is pinned — it is always sent — and the newest messages have priority. Everything greyed out is on the floor.

Part 5 — Who is eating your window?

When a team tells me “we have a 128,000 token window, we’ll never hit it,” I ask them to write down the budget. It is always tighter than they expect. Here’s a real-shaped example — an internal HR assistant:

Notice what dominates: not the user’s question, which is maybe 30 tokens. The scaffolding dominates. That is the normal state of affairs, and it’s why the highest-leverage optimisation is almost never “write a shorter question.”

Input tokens and output tokens are not the same product

Every provider charges more per output token than per input token, often 3–5×, and beginners assume this is arbitrary. It isn’t. Reading your input happens in one parallel pass — the whole prompt goes through the model at once. Writing the reply happens one token at a time, sequentially, with a full pass through the network for each token produced. Output is genuinely the expensive half.

The practical takeaway: a request that reads 50,000 tokens and returns a 200-token summary is cheap. A request that reads 500 tokens and generates a 4,000-word essay may cost more. If your bill is climbing, look at what you are asking the model to write, not just what you are feeding it.

Part 6 — Why bigger windows didn’t solve everything

Around the time million-token windows arrived, a lot of people declared retrieval dead. Just dump everything in! It did not play out that way, for four reasons I now explain in every architecture review.

Attention cost grows faster than the input

In a standard transformer, every token compares itself with every other token. Double the input and you roughly quadruple that work. Modern implementations soften the curve considerably, but the shape of the problem remains: long contexts are disproportionately expensive in compute and in memory. That cost lands on you as latency and as money.

Information gets lost in the middle

Researchers have repeatedly measured a U-shaped accuracy curve: models are excellent at recalling material near the beginning of the context and near the end, and noticeably weaker on material buried in the middle. Put the critical clause on page 40 of a 90-page contract and recall genuinely suffers. It is the reading equivalent of remembering the first and last speaker at a conference and losing the eleven in between.

More context means more distraction

Give a model 50 pages when the answer lives in one paragraph, and you have added 49 pages of plausible-looking, semantically similar noise. Precision drops. I have watched a support bot’s accuracy improve by 12 points simply by retrieving four document chunks instead of twenty. Less, but better, beat more.

You pay for every token, every turn

A 300,000-token prompt sent on each of 20 conversational turns is six million input tokens for a single conversation. Multiply by a thousand users. This is the bill that surprises startups in month three.

A large context window is a bigger desk, not a better draughtsman. What you place on the desk still decides the quality of the drawing.

Part 7 — What happens when you overflow

Four different things, and knowing which one your tool does explains a lot of otherwise baffling behaviour.

Here’s the failure I see most often in production, told as a small story. A logistics company built an assistant that read a shipment manifest and answered questions about it. It worked beautifully in testing. In production it started confidently reporting wrong container weights. The cause: their largest customers had manifests that pushed past the window, their framework silently truncated the head of the file — including the column headers — and the model, doing its best with headerless numbers, guessed which column was which. No error was ever logged. Nobody had built a token counter into the pipeline.

Count your tokens before you send them. That is the whole lesson. It takes four lines of code and it prevents an entire genus of bug.

Part 8 — The practical playbook

Nine tactics, roughly in the order I reach for them.

A worked example: 200 support tickets

Say you want themes across 200 support tickets averaging 400 words each. Naive approach: paste all of them, about 107,000 tokens, into one prompt and ask for themes. It fits in a large window — and produces a bland, shallow answer, because the middle 150 tickets get skimmed.

What actually works: batch the tickets in groups of 20. For each batch, ask for a compact structured summary — five themes, each with a one-line description and a count. That’s ten calls of ~11,000 input tokens each, returning ~300 tokens apiece. Then a final call takes those ten summaries (3,000 tokens total) and merges them into the master list. Total input is similar, output is small, and every ticket got real attention rather than a glance. This map-then-reduce shape is the workhorse pattern of practical LLM engineering, and it exists entirely because of context limits.

Part 9 — Five myths worth deleting

Quick reference

Check yourself

Five questions. No score is recorded anywhere — this is just for you.

Frequently asked questions

Where this leaves you

Two ideas, and everything downstream follows from them. Text becomes tokens — fragments, not words, priced by how ordinary they are. Tokens live inside a window — a hard-edged desk, not a memory, holding your instructions, your documents, your entire chat history and the answer being written, all at the same time.

Once those two things are solid, the rest of the field stops looking like magic. Retrieval is a technique for choosing what goes on the desk. Prompt caching is a technique for not re-reading the same corner of it. Summarisation buys space. Chunking buys attention. Even the odd failures — the miscounted letters, the sudden amnesia at message 40, the report that stops mid-sentence — become predictable, and therefore preventable.

Try this before you close the tab: take a prompt you use regularly, paste it into the splitter near the top of this page, and look at the number. Most people discover their standard prompt is two or three times more expensive than they assumed, and that a third of it is boilerplate the model never needed. That’s a five-minute experiment that has saved teams I’ve worked with real money — and it starts with just looking at the tokens.

tokenscontext windowtokenizationBPEprompt engineeringLLM basics