Curriculum · Unit 10 · Looking inside

Can we read what a model is thinking?

A language model is billions of numbers, and none of them comes with a label. For years that made models feel like sealed boxes. Since about 2023, researchers have learned to pull readable ideas out of the tangle, turn them up and down, and even trace a model's steps on the way to an answer.

Nib says: Press Next to watch one messy neuron get untangled into neat, one-idea features, or Trace it to play the whole thing. Then slide the feature dial below and see what happens when one idea gets turned way up. Anything marked illustrative is made up to show the idea.

    Turn one feature up

    Once you've found a feature, you can test it by pushing on it. Here's the idea behind the famous Golden Gate Bridge experiment, played as a toy.

    What should I cook tonight?
    0 / 10
    0 · left alone10 · clamped high

    Normal

    Replies are invented to illustrate the idea. They are not real model output.

    The idea in plain words

    You'd hope each neuron in a model stood for one thing. Mostly, it doesn't. Look at a single neuron and you'll often find it firing for a jumble of unrelated inputs. Researchers call these neurons polysemantic, meaning "many meanings". A leading explanation is superposition: a model has far more ideas to keep track of than it has neurons, so it packs them in as overlapping combinations instead of giving each idea a neuron of its own.

    In 2023, Anthropic's Towards Monosemanticity showed a way to unpack them on a very small model. The trick is a kind of dictionary learning: record the model's activity on lots of text, then train a second, simple network (a sparse autoencoder) to rewrite each burst of activity as a sum of just a few entries from a much bigger "dictionary". Many of those entries turned out to be features a person can name, each mostly about one thing.

    In 2024, Scaling Monosemanticity did the same on Claude 3 Sonnet, a real production model, and found millions of features, from famous places and people to more abstract ideas such as code bugs or flattery. One responded to the Golden Gate Bridge, in text and in pictures. When researchers clamped it high, the model brought up the bridge in almost every answer, and Anthropic briefly let the public chat with this "Golden Gate Claude". Turning a feature up and seeing the expected change is good evidence the label is right.

    Features are the words; the next question is the sentences. In 2025, On the Biology of a Large Language Model used attribution graphs to trace which features feed which on the way to an answer. It caught a model picking a rhyming word before writing the line that ends with it, and chaining two facts in its head, as in the figure below. The authors are careful to say these tools capture only part of what's going on, and work well on only some prompts.

    Why bother? Because judging a model only by what it says has limits. This field is called interpretability, and its hope is to check what a model is actually doing inside, which matters most when its words and its reasons might not match.

    Tracing a two-step thought

    Fact: the capital of the state containing Dallas is → Austin

    Dallas activates a Texas feature. Capital activates a say-a-capital feature. Together they activate say Austin, and the model outputs Austin. capital state Dallas say a capital Texas say “Austin” Austin
    Simplified from the kind of graph in the 2025 paper. Hop one: “Dallas” brings up “Texas”. Hop two: “Texas” plus “a capital is wanted” brings up “Austin”. The researchers checked the middle step by swapping in a different state's features, and the answer changed to that state's capital. Real graphs have many more nodes.

    Three things to keep

    Check yourself

    Why can't researchers just read each neuron and write down what it means?

    Because many neurons are polysemantic: one neuron fires for several unrelated things. The model seems to store more ideas than it has neurons by overlapping them, so the meaningful units are combinations of neurons, not single ones.

    What did clamping the Golden Gate Bridge feature high show?

    That the feature really was about the bridge and that it affects behavior: with it turned way up, the model kept bringing the bridge into answers that had nothing to do with it. Changing something inside and seeing the predicted change outside is strong evidence.

    Asked for “the capital of the state containing Dallas”, what middle step did researchers find, and how did they check it?

    The model worked out “Texas” internally, then used it to get to “Austin”. Swapping in features for a different state changed the answer to that state's capital, which shows the middle step was really being used.

    Go deeper