Technology & Computing · AI & machine learning
How text becomes numbers
Before a language model reads anything, the text is cut into tokens and each token becomes a point in a space where distance stands for meaning.

What you will learn
- Trace how repeated pair merging shortens a string of text
- State what changed when the compression method was adapted for language models
- Predict how changing the vocabulary size changes the tokens a string produces
- Explain how working in bytes removes the unknown-symbol problem
Lesson 1 of 4 · 6 min
How a 1994 compression trick cuts text into tokens
Before a chatbot reads your message, a program chops the text into pieces called tokens. The chopping rule wasn't invented for AI. Philip Gage described it in 1994 as a way to make files smaller, and it's called byte-pair encoding.
The rule is one step, repeated. Find the most common pair of neighboring characters. Replace every copy of that pair with one new symbol the text doesn't use. Write the swap down in a lookup table. Then do it again.
Try it on eleven letters, aaabdaaabac. The pair aa turns up most, so swap it for Z. That gives ZabdZabac, and the table says Z means aa. Now ab is the most common pair. Swap it for Y and you get ZYdZYac.
Look at what happened. ZY now appears twice, so the rule can merge its own merges. Swap ZY for X and the text is XdXac, five symbols where there were eleven. No pair repeats anymore, so the rule has nothing left to do.
Undoing it is easy. Apply the swaps in reverse order, reading each one from the table. That's why the table is stored with the compressed file. Without it, XdXac means nothing.
Language models use the same loop for a different purpose. A tokenizer is the program that cuts text into pieces before the model sees it. It isn't trying to make the smallest possible file. It's building a fixed list of pieces, the vocabulary, with a number for each. A token is one of those pieces, and the model works with the numbers.
Building the vocabulary goes like this. Start with every single character as a token. Merge the most frequent pair of neighboring tokens into a new, longer token. Keep going on a large body of text, and stop when the vocabulary reaches a size chosen in advance.
That stopping rule is the real change. Compression stops when nothing repeats. A tokenizer stops when the list is full.
The vocabulary used by GPT-3.5 and GPT-4 has 100,256 regular tokens. The first 256 are single bytes, the small units computers store text in, and merging made the other 100,000. Some tokens are tiny and some are whole words. In that list " the", with its space in front, is token 279, and " telecommunications" is a single token too.
The single characters never leave the vocabulary. So any new word can still be spelled out from smaller pieces, and nothing is unrepresentable.
Try to recallThe compression version of byte-pair encoding and the language model version run the same merging loop. What's different about them, and what follows from that difference?
They differ in purpose, and so in when they stop. The compression version keeps merging until no pair repeats, because it wants the shortest possible text. The language model version wants a vocabulary of tokens, each with a number, at a size chosen in advance. So it merges the most frequent pair until the vocabulary is full, then stops.
That makes vocabulary size a design choice instead of an outcome. And because the single characters stay in the vocabulary, any new word can still be built from smaller pieces.
A token, then, is any chunk of text that was common enough to win a place in the vocabulary before it filled up. Some are single characters and some are whole words. How big the vocabulary gets is a choice someone made in advance.
Lessons in this topic
How a 1994 compression trick cuts text into tokens
Follow byte-pair encoding as it merges the most common pair of characters again and again, and see what changed when language models borrowed it.
Why the same text can be nine tokens or four
See how one setting, the vocabulary size, changes the tokens a text becomes, and how working in bytes lets a tokenizer handle any script.
How computers learn what words mean from their neighbors
Word embeddings turn each word into a point in space, so that near means similar. See how counting a word's neighbors makes that possible.
How models tell a golf club from a club sandwich
A single vector for each word blends all its meanings together. See how newer models fixed that by giving every use of a word its own vector.
Three quick stories from this topic
Bias in word vectors
A word map built from Google News linked man to computer programmer and woman to homemaker
Word embeddings place words by the company they keep in text. A widely used set was trained on news articles, and a 2016 study found that the analogies it produced included "man is to computer programmer as woman is to homemaker".
A map made from usage
An embedding gives each word a position worked out from the words that appear around it. Whatever patterns the text holds, stereotypes included, end up in the distances.
What the researchers did next
The study's authors showed that gender bias lies along a direction in the embedding. They proposed a way to remove it while keeping useful links such as the one between queen and female.
The alphabet is one token
GPT-4's tokenizer treats the whole lowercase alphabet, a to z, as a single token
Its vocabulary of 100,256 pieces was built by merging, over and over, whichever neighboring pieces appeared together most often in its training text. Common words became single tokens. So did the string abcdefghijklmnopqrstuvwxyz, which is token number 68,612.
Spaces are part of the token
In the same vocabulary, " the" with a space in front is token 279, while "the" without one is token 1,820. The tokenizer treats them as different pieces.
Built by merging
The vocabulary starts from the 256 possible bytes. The most frequent neighboring pair is merged into a new token, again and again, until the list reaches its chosen size.
A 1994 compression trick
Chatbots cut text into tokens with a file-compression method published in 1994
Philip Gage's byte-pair encoding shrank text by swapping the most common pair of neighboring characters for one new symbol, again and again. Tokenizers run the same loop, and stop once they've built a vocabulary of a chosen size.
Stop when the vocabulary is full
Compression stops when nothing repeats. Tokenizing stops at a chosen size. For GPT-3.5 and GPT-4 that's 100,256 tokens, 256 single bytes plus 100,000 from merging.
Tokens are numbers
A token is a natural number that can stand for anything from one punctuation mark to a long compound word. Single characters stay in the vocabulary, so any word can still be built.
Sources
- Byte-pair encoding, Wikipedia
- tiktoken/tiktoken_ext/openai_public.py at main · openai/tiktoken, OpenAI
- cl100k_base.tiktoken, OpenAI
- Implementing A Byte Pair Encoding (BPE) Tokenizer From Scratch, Sebastian Raschka
- Word embedding, Wikipedia
- Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings, arXiv (Bolukbasi, Chang, Zou, Saligrama and Kalai, 2016)





