Insight
Training AI with Pure Reinforcement Learning: Insights from DeepSeek R1
What DeepSeek-R1 shows about training LLMs to reason with pure reinforcement learning, and how it differs from supervised fine-tuning.

Introduction
Large Language Models (LLMs) have made tremendous strides in recent years, yet the question of how to make machines truly “reason” remains a central challenge. Traditional methods of training AI often rely heavily on supervised fine-tuning, requiring vast datasets of labeled examples. This approach has yielded impressive results in domains like image recognition, language translation, and conversational AI, but its limitations become evident when tackling tasks requiring creativity, adaptability, and reasoning beyond pre-learned patterns.
Enter DeepSeek R1, a groundbreaking large language model that takes a fundamentally different approach: it relies on pure reinforcement learning (RL) to train its reasoning capabilities. Developed as part of the ongoing pursuit of more general AI systems, DeepSeek R1 showcases the immense potential of RL to enable self-improvement without needing massive supervised datasets.
DeepSeek R1 doesn’t just solve problems, it thinks through them. It leverages Chain-of-Thought (CoT) reasoning, a simple yet powerful prompting technique that makes the model articulate its thought process step-by-step. This allows the model to refine its own reasoning, identify errors, and correct them on the fly, making it more accurate over time.
In this post, we’ll explore the insights gained from training AI with pure reinforcement learning, as demonstrated by DeepSeek R1. We’ll break down its key features, innovative training techniques, and why it represents a paradigm shift in AI development. Whether you’re an AI enthusiast or a newcomer curious about the next frontier in machine learning, this is a story of how machines are beginning to learn like humans: through trial, error, and self-reflection.
Who is DeepSeek?
Liang Wenfeng, DeepSeek AI CEO. Chinatalk Media | Sohu
DeepSeek is a Chinese AI company founded by Liang Wenfeng that focuses on developing fundamental AI technology rather than market-ready products. The company is committed to making all its AI models open source, enabling anyone to use and modify them.
Before founding DeepSeek, Wenfeng established High-Flyer, one of China’s top four quantitative hedge funds, valued at $8 billion. High-Flyer provides full funding for DeepSeek, which has no plans to seek additional investment.
DeepSeek has launched chat applications for both web and mobile platforms, along with an API platform for developers interested in using their models.
Notably, DeepSeek managed to train V3 at a significantly lower cost of $6 million while using less powerful NVIDIA GPUs due to US export restrictions. They relied on NVIDIA H800 GPUs, which have approximately half the data transfer rate of the H100 (unavailable in China). Despite these limitations, DeepSeek-V3’s training required only 2.8 million GPU hours, compared to Llama 3 405B’s estimated 30.8 million GPU hours. These hardware restrictions appear to have driven DeepSeek to optimize their model training and inference processes.
DeepSeek V2 and V3
To understand the significance of the R1 model, let’s examine its predecessors.
DeepSeek-V2 introduced two major innovations: DeepSeekMoE (Mixture-of-Experts) and DeepSeekMLA (Multi-head Latent Attention).
The Mixture-of-Experts technique, first introduced by Robert A. Jacobs and Geoffrey Hinton in their 1991 paper “Adaptive Mixtures of Local Experts” , uses multiple expert networks to divide complex problems into manageable regions. This approach enhances both accuracy and efficiency in machine learning. While ChatGPT 3.5 activates its entire model during training and inference, MoE divides the model into specialized experts. When processing a query, only the relevant experts activate, reducing power consumption and increasing speed. GPT-4 implements this MoE architecture with an estimated 16 experts, each containing approximately 110 billion parameters.
DeepSeekMoE in V2 enhances the traditional Mixture-of-Experts by:
-
Specialized vs. Shared Experts: Using both specialized experts for specific tasks and shared experts for general knowledge.
-
Efficient Training: Improving load-balancing and routing during training, which reduces communication overhead and makes training more efficient. This contrasts with traditional MoE, which sacrifices training efficiency for efficient inference.
DeepSeekMLA (multi-head latent attention) tackles a major inference bottleneck: memory consumption. It reduces the memory needed for storing the context window by compressing the key-value pairs associated with each token. This allows for larger context windows and more efficient inference overall.
The results are impressive, as shown in the DeepSeek-V2 paper:
MMLU accuracy vs. activated parameters, among different open-source models
Training costs and inference efficiency of DeepSeek 67B (Dense) and DeepSeek-V2
V3 enhanced DeepSeek’s previous innovations by improving load balancing and implementing multi-token prediction during training. These advancements led to remarkable cost savings, with DeepSeek spending just $5.576 million to train V3. This demonstrated that large language models could be trained efficiently and economically.
Benchmark performance of DeepSeek-V3 and its counterparts.
DeepSeek-V3 architecture:
DeepSeek-V3 architecture
You can find more details about DeepSeek’s V3 architecture here.
What Makes DeepSeek R1 Unique?
Benchmark performance of DeepSeek-R1