Deep dive into LLMs like ChatGPT by Andrej Karpathy (TL;DR)

To visualize this, go to tiktokenizer.

One such post-training dataset is OASST1. Early post-training datasets were hand-curated by humans.
Now, models like UltraChat can generate synthetic conversations, allowing models to improve without as much human input.
You can visualize these mostly synthetic datasets here.

Hallucinations, Tool Use, and Memory

One major issue with LLMs is hallucination, where the model confidently generates incorrect or made-up information.

Why does this happen?

• During post-training, models learn that they must always give an answer.
• Even if the question doesn’t make sense, the model tries to generate a response instead of saying, “I don’t know.”

How Meta Deals with Hallucinations

Meta’s research on factuality (from their Llama 3 paper) describes a way to improve this:

1. Extract a snippet of training data.
2. Generate a factual question about it using Llama 3.
3. Have Llama 3 generate an answer.
4. Score the response against the original data.
5. If incorrect, train the model to recognize and refuse incorrect responses.

Essentially, this process teaches models to recognize their own knowledge limits.

Using Tools to Reduce Hallucinations

One way to fix hallucinations is to train models to use tools when they don’t know the answer. This approach follows the pattern:

userWho is Orson Kovacs?
assistantWho is Orson Kovacs?

[...search results...]

assistantOrson Kovacs is ....

With repeated training, models learn that if they don’t know something, they should look it up instead of making things up.

“Vague Recollection” vs. “Working Memory”

• Model parameters store vague recollections (like remembering something from a month ago).
• Context tokens function as working memory, giving models access to fresh information.

This is why retrieval-augmented generation (RAG) works so well: if the model has direct access to relevant documents, it doesn’t need to guess.

Knowledge of Self

If you prompt an untuned base model about who it is, it will likely hallucinate. For example, a non-OpenAI model might still say it was created by OpenAI simply because most internet data links AI models to OpenAI.

How to Fix This

By default, LLMs have no real knowledge of themselves. Without specific training, they default to generic AI responses.

Models Need Tokens to Think

LLMs don’t reason like humans. They generate tokens sequentially, meaning they need structured generation to think properly.

Example: Bad vs. Good Model Output

Bad Model Output:

Human: Emily buys 3 apples and 2 oranges. Each orange costs $2. The total cost of all the fruit is $13. What is the cost of apples?
Assistant: The answer is $3.

The model jumps to the answer without breaking it down.

Good Model Output:

Human: Emily buys 3 apples and 2 oranges. Each orange costs $2. The total cost of all the fruit is $13. What is the cost of apples?
Assistant: The total cost of the oranges is $4. 13 - 4 = 9, so the cost of the 3 apples is $9. 9/3 = 3, so each apple costs $3.

Here, the model works through the reasoning step-by-step.

Why This Matters

• If a model jumps straight to an answer, it might just be guessing.
• If it walks through the solution step-by-step, it’s more reliable.
• The model breaks down the problem into smaller steps. Since there are finite layers in the model, one token output cannot be processed indefinitely. Breaking down the problem into smaller steps allows the model to process the problem in a way that is more likely to yield the correct answer.

For math and logic tasks, it’s best to ask the model to use external tools rather than relying on its own reasoning.

Reinforcement Learning

Once a model is trained on internet data, it still doesn’t know how to use its knowledge effectively.

• Supervised fine-tuning teaches it to mimic human responses.
• Reinforcement learning (RL) helps it improve by trial and error.

How RL Works

Instead of relying on human-created datasets, RL lets the model experiment with different solutions and figure out what works best.

Example Process:

We generated 15 solutions.
Only 4 of them got the right answer.
Take the top solution (each right and short).
Train on it.
Repeat many, many times.

No human is involved in this process. The model generates different solutions to the same problem, sometimes they are in millions. Then it compares them to pick the ones that reached the correct answer and then trains on the winning solutions.

The pre-training and post-training processes are very well defined but the RL process is still under a lot of active research. Andrej talks about it here as well. Companies like OpenAI do a lot of research on it
but it’s not public. That is why the release of DeepSeek was such a big deal. Their paper talks more about it. It talks very publicly about RL and FT for LLMs and how it brings out a lot of reasoning capabilities in them.

An example taken from the Deepseek paper shows us that as the time passes, the model is able to use more tokens to get better at reasoning.

You can see that the model has this “aha” moment here, it’s not something that you can explicitly teach the model through just training on a dataset. It’s something that the model has to figure out on it’s own through reinforcement learning. Pros of this technique is, the model is becoming
better at reasoning but the con is it’s consuming more and more tokens to do so.

One thing we can learn from the research paper on mastering the game of Go is that RL actually helps the model become better at reasoning than their human counterparts.
The model isn’t just trying to imitate the human but it’s coming up with its own strategies through trial and error to win the game.

One very unique thing noted during the AlphaGo’s game was a move called Move 37.
It’s a move that was not part of the training data but the model came up with its own strategy to win the game. The researchers predicted that the chance of it being played
by a human would be 1 in 10000. So you can see how the model is capable of coming up with its own strategies.

RL is still heavily unexplored and there is a lot of research going on in this area. It is entirely possible if given the chance, the LLM might come up with a language
of its own to express its thoughts and ideas because it discovered that it’s the best way to express its thoughts and ideas.

Learning on unverifiable domains a.k.a Reinforcement Learning from Human Feedback (RLHF)

It is very easy to exclude humans from the RL process in verifiable domains. LLMs can act as a judge for its own performance.

However, in unverifiable domains, we need to include humans in the loop.

For example, for a prompt Write a joke about pelicans, it is not very easy to find a way to automatically judge the quality of the joke.
The LLM will generate the jokes without issues, but judging their quality at scale is not possible.

Furthermore, including humans in this process at scale is not feasible. This is where RLHF comes in. You can read more about it in this paper.

In order to do RLHF at scale, you basically train a separate reward model which could be a transformer without the extra layers surrounding it.
You use humans to judge the rank it is giving to the responses and then you use that to train the reward model until you are satisfied with the results.
Once that is done, you can use the reward model to judge the quality of the responses generated by the LLM at scale.

RLHF Upside

• Enables RL in unverifiable domains like joke-writing or summarization.
• Often improves models by reducing hallucinations and making responses more human-like.
• Exploits the “discriminator-generator gap“—humans find it easier to evaluate an answer than generate one.
  • Example: “Write a poem” vs. “Which of these 5 poems is best?”

RLHF Downside

• The reward model is just a simulation of human preferences, not an actual human. This can be misleading.
• RL can game the system, producing adversarial examples that exploit weaknesses in the reward model.
  • Example: After 1,000 updates, the model’s “top joke about pelicans” might be complete nonsense (e.g., “the the the the the the the the”).
• This is known as Adversarial Machine Learning. Since there are infinite ways to game the system, filtering out bad responses isn’t straightforward.
• To prevent this, reward model training is capped at a few hundred iterations—beyond that, models start over-optimizing and performance declines.

Preview of Things to Come

Future LLMs will expand in several key areas:

• Multimodal Capabilities → Not just text, but also understanding and generating images, audio, and video.
• Agent-Based Models → Moving beyond single tasks to long-term memory, reasoning, and correction of mistakes.
• Pervasive & Invisible AI → AI will be integrated into workflows in a way that becomes second nature.
• Computer-Using AI → AI models that interact with software and take actions beyond just text generation.
• Test-Time Training → AI adapting itself in real-time to improve accuracy on the fly.

Keeping Track of LLMs

If you’re interested in following developments in this space, here are some great resources:

• LM Arena → Benchmarking new language models.
• AI News → A newsletter covering AI research.
• X (Twitter) → Many researchers share updates here.

Where to Find LLMs

Want to try different LLMs? Here’s where to find them:

• Proprietary Models → OpenAI (GPT-4), Google (Gemini), Anthropic (Claude), etc.
• Open-Weight Models → DeepSeek, Meta (Llama), etc. Try them via Together.ai.
• Run Locally → Use Ollama or LM Studio.
• Base Models → Explore Hyperbolic.