Technology & Computing · AI & machine learning

How a language model produces an answer

See how tokens, context and learned patterns produce fluent answers, and why those answers still need checking.

4 short lessons · about 17 minutes · 4 sources · updated November 2026

What you will learn

  • Explain what a token is and how generated tokens extend a sequence.
  • Distinguish a likely continuation from a verified statement.
  • Explain how surrounding words can change the way a token is interpreted.
  • Distinguish information in a prompt from learned model parameters.

Lesson 1 of 4 · 4 min

Building an answer one token at a time

Before you readA chatbot answers "The library closes at 5 pm" with no web access. What most likely produced that time?

Correct.Not quite. The answer is A likely ending, from patterns it learned.

It most likely came from a likely-sounding ending, because the model predicts what comes next. This lesson shows how an answer is built and why that is not the same as checking.

Ask a chatbot why the sky is blue and a tidy paragraph appears in seconds. It's natural to picture the program looking the answer up and copying it out. It doesn't. A language model writes by predicting what comes next, one small piece at a time.

The pieces are called tokens. A token can be a whole word, part of a word or a single character. In English a token averages about four characters, so 400 tokens come to roughly 300 words.

Here's the loop. The model reads all the text so far and rates how likely each possible next token is. One token is chosen and added to the text. Then it reads the longer text and does it again.

Google's machine learning course shows it with a gap to fill. "When I hear rain on my roof, I ___ in my kitchen." Its example model gives "cook soup" a 9.4% chance, "warm up a kettle" 5.2% and "cower" 3.6%. The program can take the top choice or pick among the likely ones at random, so one question can get different answers.

This is autoregressive generation, meaning each step builds on the model's own earlier output. The same course calls these models "essentially autocomplete mechanisms". It's a common design, and others exist.

Try it with the unfinished sentence "The library closes at". A time would fit. So would "sunset", or "noon on Sundays". The model chooses among endings that sound right, given the context and the patterns it learned.

Notice what's missing. Nothing in that loop looked at a timetable. A natural ending tells you about language. It tells you nothing about when any real library shuts.

Check yourselfSuppose a language model splits "sunset" into the tokens "sun" and "set". After generating "sun", how does it continue?

Correct.Not quite. The answer is It adds "sun" to the text and predicts again from the longer text.

A token can be part of a word, so the model doesn't wait for whole words. It adds each chosen token to the text, and the longer text, including its own output, shapes the next prediction. Nothing in the loop checks whether the finished statement is true.

The patterns come from training. A large model has hundreds of billions of parameters, numbers that training adjusts. Words are hidden in real text and the model guesses them until it guesses well.

It learns which text tends to follow which. No step attaches a checked source to a sentence. So a sentence can fit its surroundings like a puzzle piece with the right shape and still carry the wrong date or name.

Try to recallWhy isn't a confident, grammatical answer from a language model enough evidence that its factual claim is correct?

The model builds its answer by predicting likely next tokens from learned patterns and the context. A likely continuation can still contain a false date, name or claim, because no step checks it. So the facts have to be checked against a source.

Lessons in this topic

  1. Building an answer one token at a time

    Follow a language model as it builds an answer by predicting one token after another, and see why sounding right is different from being checked.

    Essentials · 4 min

  2. How a language model works out what "it" means

    Use one ambiguous sentence to see how attention links words together, and why pasting text into a prompt isn't the same as training.

    Essentials · 4 min

  3. How to check an AI answer that cites its sources

    A language model can invent a convincing citation. Learn the two checks that catch it, and the problem a real source can still hide.

    Deeper · 4 min

  4. Three ways to make one AI model do a new job

    Compare fine-tuning, distillation and prompt engineering by what each changes inside the model, what it costs and what it gives up.

    Deeper · 5 min

Three quick stories from this topic

Answers worked out ahead

Google Search had a language model list over 800 synonyms for Covid vaccines in advance

The list covered more than 50 languages. The model worked through it ahead of time and Search stored the result, then used it to recognize vaccine queries in live traffic. A slow model is fine for a job done once in a while.

Slow is fine if it runs once

Very large models can be too slow for real-time use. With offline inference, the model works through the task in advance and caches its predictions. A slow run doesn't matter when it happens once a week or once a month. In the course's example, Google Search cached more than 800 synonyms for Covid vaccines in over 50 languages to recognize vaccine queries.

Stored answers keep the same limits

Precomputing only suits tasks where a real-time response isn't critical, and a cached prediction has the same limits as a live one. LLMs share the biases of the data they were trained or distilled on.

The case that didn't exist

A chatbot told a lawyer that a court case it had invented "does indeed exist"

In 2023 two New York lawyers filed a document citing six court decisions that ChatGPT had made up. Asked whether one was real, it said yes and named the legal databases that held it. A federal judge imposed a $5,000 penalty.

Why a model invents sources

A language model writes by predicting likely text. Google's course says such models hallucinate, meaning their predictions often contain mistakes, and a citation is one more piece of predicted text.

Two checks before you trust one

First confirm the work exists. Then read the passage and see whether it supports the exact claim. Asking the chatbot if it's sure isn't a check.

Autocomplete, says Google

Google's machine learning course calls large language models "essentially autocomplete"

In full, it describes them as "essentially autocomplete mechanisms" that can predict thousands of tokens, or pieces of text, in a row. A model predicts the next piece, adds it and predicts again. No step in that loop checks whether a sentence is true.

Predict, add, repeat

The model rates possible next tokens, one is chosen and added, and the longer text is fed back in. A token can be a word, part of a word or a single character.

Why the wording varies

The program can take the most likely token or pick at random among the likely ones. That's one reason the same question can get different answers.

Sources

This topic is part of What happens behind the screen, a guided path through related topics.

  • How text becomes numbers

    Before a language model reads anything, the text is cut into tokens and each token becomes a point in a space where distance stands for meaning.

    4 lessons · 24 min

  • What artificial intelligence is

    AI is one label for many machines, from spam filters to chatbots. See what they share, how learned systems differ from hand-written rules, and how to test a claim about what AI can do.

    4 lessons · 16 min

  • How machine learning works

    A machine-learning system isn't given the rules. It's shown examples and adjusts itself until its guesses improve. See how training works, why a model can ace its lessons and fail in the world, and how it's tested.

    4 lessons · 16 min

  • Why adding cores stops helping

    A fixed job has a part that won't split, and that part sets a ceiling no number of processors can lift.

    4 lessons · 24 min

  • What a database is

    See how tables, rows and keys organize records, how one SQL question pulls out an answer, why a spreadsheet stops being enough and when other kinds of database fit better.

    4 lessons · 16 min

  • Searching a million items

    See why a sorted list of a million items takes about 20 checks to search, when sorting first is worth the effort, and how a hash table skips the search altogether.

    4 lessons · 17 min

All technology & computing topics · Take the quiz