In depth
How a model learns, and why it answers the way it does
From raw text to your answer: the whole life cycle of a language model, told without a single equation.
Reviewed on August 22, 2026
Where do the texts come from?
The raw material of a language model is text, in amounts no human will ever read: copies of the web (the non-profit Common Crawl project has archived pages since 2008), scanned books, encyclopedias, code, transcripts. That raw pile is then cleaned: duplicates, spam, misdetected languages and recognisable personal data are removed, and sources judged higher quality get more weight. The sorting matters as much as the volume: two models fed the same web, filtered differently, will not reach the same level. This is also where the copyright debate starts: those texts were written by people who, for the most part, never signed anything.
Read the discovery version
Picture a library so big you can't see the end of it: web pages, books, articles, forum chats, computer code. Before going to school, the AI gets all of that in its backpack. But someone sorts it first: duplicate pages, broken texts and nasty stuff are thrown out. And some authors complain: "That book is mine, nobody asked me!" It's a big argument between grown-ups, and it isn't over.
Pre-training: guessing what comes next, billions of times
Pre- rests on one very simple instruction: hide the next of a text and ask the model to predict it. When it gets it wrong, an algorithm nudges its (its internal "settings", hundreds of billions of them in the largest models) in the direction that would have made the right answer more likely. Repeated over trillions of tokens, this modest task forces the network to absorb grammar, facts, styles and elementary reasoning: whatever helps guess what comes next. The result is a "base model": a brilliant completer that cannot yet hold a conversation or follow an instruction. This phase keeps tens of thousands of chips busy for weeks; it is the one whose appetite our Conso page measures.
Read the discovery version
The AI's favourite game at school is called "guess what comes next". It is shown "The cat is sleeping on the…" and has to suggest "sofa", "bed" or "rug". At first it says anything. But after each mistake, its billions of tiny inner knobs get turned just a little, and it gets a bit less wrong. It plays this game billions of billions of times, in a building full of hot computers, until it knows the music of sentences in almost every language.
Fine-tuning: learning to be helpful and harmless
The base model is an excellent text imitator, not an assistant. transforms it in two steps. First, instruction tuning: it is shown tens of thousands of human-written question-answer pairs, so that it gets into the habit of answering rather than continuing the text. Then preference learning (RLHF, ""): annotators rank several answers, a reward model learns those tastes, and the assistant is adjusted to aim for what pleases. "Helpful and harmless" is the accepted phrase: genuinely address the request, without giving dangerous help or making things up. This phase, not pre-, is what gives an assistant its tone, its caution and its refusals.
Read the discovery version
After the big school, the AI can finish sentences, but it doesn't yet know how to be kind and useful. So people teach it good manners: they write examples of nice answers, then show it two answers and say "this one is better". Little by little, the AI starts to prefer answers that help, tell the truth and hurt nobody. A bit like a puppy you praise when it does well.
Answering: one word after another, with confidence
At use time, the model no longer changes: it reads your message cut into , computes a probability for every possible next token, draws one, appends it, and repeats until the final full stop. A setting called "" doses the randomness of that draw: low, the model almost always picks the most likely token and repeats itself; high, it varies more and derails sooner. Nothing in this mechanism tells a known fact from a plausible wording: when the information is missing, the most likely text often looks like a confident answer. Hence "", those errors stated with poise: an invented reference, a shifted date, a figure rounded at random. Today's assistants compensate by searching the web or your documents before writing, which reduces the problem without removing it.
Read the discovery version
When you ask a question, the AI doesn't look the answer up in a drawer: it builds it, piece by piece, guessing the most likely next word, just like at school. Sometimes it is allowed a bit of randomness, so it doesn't always say exactly the same thing. Because it picks what "sounds right", it can tell you something false in a very sure voice. That's why we check what it says, the way you'd check a friend who talks very fast.
What it does not do (and what we still don't know)
Three received ideas to take apart. The model does not learn from your conversation: its are frozen after ; what it "retains" during an exchange sits in its and vanishes when you close it, except for memory features that feed it notes at the start of the next session. Your messages can, however, be used to train the next version, depending on the provider and your settings: ChatGPT and Claude do so for consumer accounts unless you opt out, never for business plans. Finally, does it "understand"? It has no body, no lived experience, no intentions of its own, and some researchers see nothing more than a "stochastic parrot"; others observe in its circuits representations of the world that look like a form of understanding. The debate is open, and it is honest to say so.
Read the discovery version
The AI talks very well, but it is not a person who understands like you do. It hasn't lived, never tasted ice cream, never been afraid of the dark. It doesn't remember you from one conversation to the next, unless it is given a little notebook to write things down. And it doesn't learn while you chat: the big school is over. But scientists still argue: does it understand "a little" anyway? Nobody has the full answer.