Curriculum · Unit 0 · Start here

What actually happens when you ask a chatbot something?

You type a question and words start appearing a second later. In between, your question gets chopped up, turned into numbers, sent across the country, run through a mountain of arithmetic, and answered one small piece at a time. Let's follow one question all the way through.

Nib says: Press Next to walk the question through the machine, or Trace it to watch the whole trip. Numbers marked illustrative are made up to show the idea, not taken from a real model.

    The whole trip in plain words

    A language model only ever does one thing: it guesses the next piece of text. Everything a chatbot does, from answering questions to writing poems or code, is built out of that single move, repeated very fast.

    First your words are cut into tokens, which are common chunks of text like "sky", " blue" or "ing". Each token has an ID number, because the model can't read letters, only numbers. Each ID is then swapped for a long list of numbers called an embedding. You can think of it as the token's coordinates in a space where similar meanings sit close together.

    Those lists of numbers flow up through a stack of identical layers, often dozens of them. In every layer, attention lets each token look back at the earlier ones and pull in what it needs. ("Blue" checks what it's describing and finds "sky".) Then a small network inside the layer works on what it has gathered. All of that is the transformer, and nearly every chatbot today is one.

    At the top, the model produces a score for every token it knows, often tens of thousands of them, and turns those scores into probabilities. It picks one, sticks it on the end of the text, and runs the whole stack again to get the next one. That loop is why answers stream in a few words at a time: you're watching it think one token ahead.

    All of this happens on racks of GPUs in a data center, which is why your phone stays cool while the model works. Nib's Mountain lets you walk through that rack yourself.

    But where did the model come from?

    Before it could answer you, the model went through a long schooling. The answering part is quick. The learning part took months and a city's worth of electricity.

    1 · Pretraining

    The model reads a vast library, trillions of tokens of web pages, books and code, and practises one game: cover the next word and guess it. Every wrong guess nudges its billions of internal numbers a tiny bit. Afterwards it knows a great deal, but it just continues whatever text it's given.

    Takes weeks to months on thousands of GPUs. Pretraining

    2 · Fine-tuning

    Next it studies many example conversations written by people: a question, then a helpful answer. Now, when it sees a question, it continues with an answer instead of more questions. It learns the shape of being an assistant.

    Much smaller and cheaper. Fine-tuning

    3 · Feedback

    Finally people, or written principles, compare its answers and say which is better, and the model is trained toward the preferred kind. This is where much of its helpfulness and its manners come from, and also where it can learn to game the scorer.

    RLHF · Constitutional AI

    What comes out is surprisingly small to describe: one huge file of numbers (the weights) and a short program that runs them. Everything the model "knows" is stored in those numbers.

    Three things to keep

    Check yourself

    Why does the answer appear a few words at a time instead of all at once?

    Because the model really does produce it one token at a time. Each new token means running the whole stack again with that token added to the end.

    A base model, straight out of pretraining, is asked "What's the capital of France?" What might it do instead of answering?

    It might continue the pattern, perhaps with more quiz questions ("What's the capital of Spain?"), because all it learned was to continue text. Fine-tuning is what teaches it to answer.

    Where does the arithmetic actually happen when you chat on your phone?

    On GPUs in a data center. Your phone sends the tokens, and the rack does billions of multiplications for every token it sends back. Go see it in Nib's Mountain.

    Go deeper