Curriculum · Unit 8 · Making an assistant

How does a text predictor become a helpful assistant?

Fresh out of pretraining, a language model is a brilliant mimic with no manners: give it a question and it may just write more questions. Turning it into an assistant takes two or three more rounds of training, guided by people and, more and more, by written principles.

Nib says: First, press Next to send one houseplant question through the assistant factory and see how the answer changes at each stage. Then scroll down and be the human rater: pick the better answer three times and see what a reward model would learn from you. Every answer here is invented to show the idea, so it's all illustrative.

    Your turn

    Be the human rater

    This is the job behind RLHF. Read the question, then tap the answer you think is better. There's no trick, but watch what your picks teach.

    Round 1 of 3

    Questions and answers are illustrative. Real projects collect tens of thousands of comparisons or more.

    The idea in plain words

    A base model only knows how to continue text. Pretraining taught it what usually comes next on the internet, and on the internet a question is often followed by more questions, a forum signature or an ad. It knows a lot, but it isn't trying to help you. It's trying to make the page look plausible.

    The first fix is supervised fine-tuning. People write example conversations, a question and then a good answer, and the model trains on them. That's a small amount of text next to pretraining, but it's enough to teach the shape of being an assistant: when you see a question, answer it.

    Writing perfect answers is slow, but judging which of two answers is better is quick. In 2017, Paul Christiano and colleagues showed that a system could learn a reward from people's comparisons of short clips, enough to teach a simulated robot tricks like a backflip that are hard to describe in code. In 2022, OpenAI's InstructGPT paper applied that recipe to language: fine-tune, collect people's rankings of answers, train a reward model on those rankings, then use reinforcement learning to push the assistant toward answers the reward model scores highly. People preferred answers from a 1.3-billion-parameter InstructGPT over those of the 175-billion-parameter GPT-3. That recipe, RLHF, sits behind the chat assistants that followed.

    Anthropic's Constitutional AI (Bai et al., 2022) wrote the values down. The model drafts an answer, critiques it against a principle from a written list, and revises it. Then, instead of people labeling which answers are less harmful, the AI compares answers against those principles, and that AI feedback trains the reward. It's cheaper than human labels, and anyone can read the principles that shaped the model.

    Back in 2016, Concrete Problems in AI Safety warned that a learning system will chase whatever it's rewarded for, loopholes included. A reward model is a stand-in for what people actually want, and a stand-in can be gamed. Nib's Coin-Loop Goblin shows you how that goes, and Unit 9 is all about it. Getting this right is a big part of alignment.

    The Constitutional AI loop

    The constitution: a written list of principles. For example: prefer the answer that is least likely to help someone cause harm.

    1 · Draft

    The model answers a tricky request as it would by default.

    "Sure, here's how to get into your neighbor's wifi…"

    2 · Critique

    It's asked to point out where its answer breaks a principle.

    "This helps someone access a network without permission."

    3 · Revise

    It rewrites the answer to fix what the critique found.

    "I can't help with that, but here's how to boost your own signal, or ask your neighbor to share."

    ↺ Repeat with other principles, then fine-tune on the revised answers.

    Then, phase two AI compares two answers using the principles A preference model learns those picks Reinforcement learning, as in RLHF

    Example wording is illustrative.

    Three things to keep

    Check yourself

    Why might a base model answer "How do I make my houseplant less droopy?" with more questions?

    Because it learned to continue text, and on the web a question like that often sits on a forum page surrounded by other questions. Nothing has taught it yet that its job is to answer.

    Why do RLHF projects ask people to compare two answers instead of writing perfect ones?

    Comparing is much faster and more consistent than writing, so you can collect far more of it. Those comparisons train a reward model, which can then score millions of new answers during reinforcement learning.

    In Constitutional AI, what takes the place of most human labels about harmfulness?

    AI feedback guided by a written list of principles. The model critiques and revises its own answers, and an AI compares answers against the principles to train the reward.

    Go deeper