Technology & Computing · AI & machine learning

How text becomes numbers

Before a language model reads anything, the text is cut into tokens and each token becomes a point in a space where distance stands for meaning.

4 short lessons · about 24 minutes · 6 sources · updated November 2026

Two rows of colored paper tiles, the lower row shorter because pairs from the upper row have become wide yellow tiles, beside a small box holding one yellow tile.

What you will learn

  • Trace how repeated pair merging shortens a string of text
  • State what changed when the compression method was adapted for language models
  • Predict how changing the vocabulary size changes the tokens a string produces
  • Explain how working in bytes removes the unknown-symbol problem

Lesson 1 of 4 · 6 min

How a 1994 compression trick cuts text into tokens

Before you readA rule from 1994, made to shrink files, now cuts text into pieces for chatbots. What does it do over and over?

Correct.Not quite. The answer is Merges the most common neighboring pair.

It merges the most common neighboring pair into one new symbol. This lesson follows that rule from file compression to the tokens chatbots read.

Before a chatbot reads your message, a program chops the text into pieces called tokens. The chopping rule wasn't invented for AI. Philip Gage described it in 1994 as a way to make files smaller, and it's called byte-pair encoding.

The rule is one step, repeated. Find the most common pair of neighboring characters. Replace every copy of that pair with one new symbol the text doesn't use. Write the swap down in a lookup table. Then do it again.

Try it on eleven letters, aaabdaaabac. The pair aa turns up most, so swap it for Z. That gives ZabdZabac, and the table says Z means aa. Now ab is the most common pair. Swap it for Y and you get ZYdZYac.

Look at what happened. ZY now appears twice, so the rule can merge its own merges. Swap ZY for X and the text is XdXac, five symbols where there were eleven. No pair repeats anymore, so the rule has nothing left to do.

Undoing it is easy. Apply the swaps in reverse order, reading each one from the table. That's why the table is stored with the compressed file. Without it, XdXac means nothing.

Check yourselfIn byte-pair encoding, how is the pair chosen for replacement at each step?

Correct.Not quite. The answer is The most common adjacent pair in the current text is replaced everywhere it occurs.

Each round looks at the text as it stands, finds the most frequent adjacent pair and substitutes a symbol for every instance of it. Because the text changes after each substitution, a pair that wasn't the most common at the start can become the most common later. That's how longer sequences get built up out of shorter ones.

Language models use the same loop for a different purpose. A tokenizer is the program that cuts text into pieces before the model sees it. It isn't trying to make the smallest possible file. It's building a fixed list of pieces, the vocabulary, with a number for each. A token is one of those pieces, and the model works with the numbers.

Building the vocabulary goes like this. Start with every single character as a token. Merge the most frequent pair of neighboring tokens into a new, longer token. Keep going on a large body of text, and stop when the vocabulary reaches a size chosen in advance.

That stopping rule is the real change. Compression stops when nothing repeats. A tokenizer stops when the list is full.

The vocabulary used by GPT-3.5 and GPT-4 has 100,256 regular tokens. The first 256 are single bytes, the small units computers store text in, and merging made the other 100,000. Some tokens are tiny and some are whole words. In that list " the", with its space in front, is token 279, and " telecommunications" is a single token too.

The single characters never leave the vocabulary. So any new word can still be spelled out from smaller pieces, and nothing is unrepresentable.

Try to recallThe compression version of byte-pair encoding and the language model version run the same merging loop. What's different about them, and what follows from that difference?

They differ in purpose, and so in when they stop. The compression version keeps merging until no pair repeats, because it wants the shortest possible text. The language model version wants a vocabulary of tokens, each with a number, at a size chosen in advance. So it merges the most frequent pair until the vocabulary is full, then stops.

That makes vocabulary size a design choice instead of an outcome. And because the single characters stay in the vocabulary, any new word can still be built from smaller pieces.

A token, then, is any chunk of text that was common enough to win a place in the vocabulary before it filled up. Some are single characters and some are whole words. How big the vocabulary gets is a choice someone made in advance.

Lessons in this topic

  1. How a 1994 compression trick cuts text into tokens

    Follow byte-pair encoding as it merges the most common pair of characters again and again, and see what changed when language models borrowed it.

    Essentials · 6 min

  2. Why the same text can be nine tokens or four

    See how one setting, the vocabulary size, changes the tokens a text becomes, and how working in bytes lets a tokenizer handle any script.

    Essentials · 6 min

  3. How computers learn what words mean from their neighbors

    Word embeddings turn each word into a point in space, so that near means similar. See how counting a word's neighbors makes that possible.

    Deeper · 6 min

  4. How models tell a golf club from a club sandwich

    A single vector for each word blends all its meanings together. See how newer models fixed that by giving every use of a word its own vector.

    Deeper · 6 min

Three quick stories from this topic

Bias in word vectors

A word map built from Google News linked man to computer programmer and woman to homemaker

Word embeddings place words by the company they keep in text. A widely used set was trained on news articles, and a 2016 study found that the analogies it produced included "man is to computer programmer as woman is to homemaker".

A map made from usage

An embedding gives each word a position worked out from the words that appear around it. Whatever patterns the text holds, stereotypes included, end up in the distances.

What the researchers did next

The study's authors showed that gender bias lies along a direction in the embedding. They proposed a way to remove it while keeping useful links such as the one between queen and female.

The alphabet is one token

GPT-4's tokenizer treats the whole lowercase alphabet, a to z, as a single token

Its vocabulary of 100,256 pieces was built by merging, over and over, whichever neighboring pieces appeared together most often in its training text. Common words became single tokens. So did the string abcdefghijklmnopqrstuvwxyz, which is token number 68,612.

Spaces are part of the token

In the same vocabulary, " the" with a space in front is token 279, while "the" without one is token 1,820. The tokenizer treats them as different pieces.

Built by merging

The vocabulary starts from the 256 possible bytes. The most frequent neighboring pair is merged into a new token, again and again, until the list reaches its chosen size.

A 1994 compression trick

Chatbots cut text into tokens with a file-compression method published in 1994

Philip Gage's byte-pair encoding shrank text by swapping the most common pair of neighboring characters for one new symbol, again and again. Tokenizers run the same loop, and stop once they've built a vocabulary of a chosen size.

Stop when the vocabulary is full

Compression stops when nothing repeats. Tokenizing stops at a chosen size. For GPT-3.5 and GPT-4 that's 100,256 tokens, 256 single bytes plus 100,000 from merging.

Tokens are numbers

A token is a natural number that can stand for anything from one punctuation mark to a long compound word. Single characters stay in the vocabulary, so any word can still be built.

Sources

This topic is part of Why computers don't do what you'd expect, a guided path through related topics.

  • How a language model produces an answer

    See how tokens, context and learned patterns produce fluent answers, and why those answers still need checking.

    4 lessons · 17 min

  • What artificial intelligence is

    AI is one label for many machines, from spam filters to chatbots. See what they share, how learned systems differ from hand-written rules, and how to test a claim about what AI can do.

    4 lessons · 16 min

  • How machine learning works

    A machine-learning system isn't given the rules. It's shown examples and adjusts itself until its guesses improve. See how training works, why a model can ace its lessons and fail in the world, and how it's tested.

    4 lessons · 16 min

  • How processors execute instructions

    Follow a tiny calculation through the instructions, memory addresses and circuits that make a computer work.

    4 lessons · 22 min

  • How recommendation feeds decide what you see

    Follow how a feed narrows billions of items to a screenful, what its score rewards, and how the design of a screen steers the choices you make.

    4 lessons · 17 min

  • How your browser's cache decides what to reuse

    Follow a stored response from fresh to stale to revalidated, and see why the hard part of caching is knowing when to stop trusting a stored copy.

    4 lessons · 22 min

All technology & computing topics · Take the quiz