DeepSeek R-1 Model Overview and how it Ranks against OpenAI’s O1

DeepSeek is a Chinese AI company “dedicated to making AGI a reality” and open-sourcing all its designs. They began in 2023, but have been making waves over the past month approximately, and especially this past week with the release of their two most current thinking models: DeepSeek-R1-Zero and the advanced DeepSeek-R1, likewise referred to as DeepSeek Reasoner.

They’ve released not just the designs however likewise the code and evaluation triggers for public use, along with an in-depth paper describing their technique.

Aside from creating 2 extremely performant models that are on par with OpenAI’s o1 model, the paper has a lot of important info around reinforcement learning, chain of idea thinking, timely engineering with thinking models, and more.

We’ll begin by concentrating on the training process of DeepSeek-R1-Zero, which uniquely relied entirely on reinforcement learning, instead of traditional supervised knowing. We’ll then proceed to DeepSeek-R1, how it’s reasoning works, and some prompt engineering finest practices for thinking models.

Hey everyone, Dan here, co-founder of PromptHub. Today, we’re diving into DeepSeek’s newest design release and comparing it with OpenAI’s thinking models, particularly the A1 and A1 Mini designs. We’ll explore their training procedure, reasoning abilities, and some essential insights into prompt engineering for reasoning designs.

DeepSeek is a Chinese-based AI business dedicated to open-source advancement. Their current release, the R1 thinking model, is groundbreaking due to its open-source nature and ingenious training methods. This includes open access to the designs, triggers, and research papers.

Released on January 20th, DeepSeek’s R1 achieved outstanding performance on various benchmarks, matching OpenAI’s A1 models. Notably, they also introduced a precursor design, R10, which serves as the foundation for R1.

Training Process: R10 to R1

R10: This model was trained exclusively utilizing reinforcement learning without supervised fine-tuning, making it the very first open-source model to accomplish high performance through this method. Training involved:

– Rewarding proper responses in deterministic tasks (e.g., math issues).
– Encouraging structured thinking outputs utilizing design templates with “” and “” tags

Through thousands of models, R10 developed longer reasoning chains, self-verification, and even reflective behaviors. For instance, throughout training, the design showed “aha” minutes and self-correction behaviors, which are unusual in standard LLMs.

R1: Building on R10, R1 added a number of improvements:

– Curated datasets with long Chain of Thought examples.
– Incorporation of R10-generated thinking chains.
– Human choice positioning for sleek reactions.
– Distillation into smaller designs (LLaMA 3.1 and 3.3 at various sizes).

Performance Benchmarks

DeepSeek’s R1 model carries out on par with OpenAI’s A1 models throughout lots of reasoning benchmarks:

Reasoning and Math Tasks: R1 competitors or surpasses A1 designs in precision and depth of thinking.
Coding Tasks: A1 designs generally carry out better in LiveCode Bench and CodeForces tasks.
Simple QA: R1 frequently surpasses A1 in structured QA tasks (e.g., 47% accuracy vs. 30%).

One noteworthy finding is that longer thinking chains generally improve efficiency. This lines up with insights from Microsoft’s Med-Prompt structure and OpenAI’s observations on test-time compute and reasoning depth.

Challenges and Observations

Despite its strengths, R1 has some restrictions:

– Mixing English and Chinese responses due to a lack of supervised fine-tuning.
– Less polished responses compared to talk designs like OpenAI’s GPT.

These problems were attended to throughout R1’s refinement process, consisting of supervised fine-tuning and human feedback.

Prompt Engineering Insights

A remarkable takeaway from DeepSeek’s research is how few-shot triggering abject R1’s performance compared to zero-shot or succinct customized prompts. This aligns with findings from the Med-Prompt paper and OpenAI’s recommendations to restrict context in thinking models. Overcomplicating the input can overwhelm the model and minimize precision.

DeepSeek’s R1 is a substantial action forward for open-source thinking models, demonstrating capabilities that rival OpenAI’s A1. It’s an interesting time to experiment with these designs and their chat interface, which is totally free to utilize.

If you have concerns or wish to discover more, inspect out the resources linked below. See you next time!

Training DeepSeek-R1-Zero: A support learning-only approach

DeepSeek-R1-Zero sticks out from many other state-of-the-art models since it was trained using only reinforcement learning (RL), no supervised fine-tuning (SFT). This challenges the current standard approach and opens new opportunities to train thinking designs with less and effort.

DeepSeek-R1-Zero is the first open-source model to verify that advanced reasoning abilities can be established purely through RL.

Without pre-labeled datasets, the model finds out through experimentation, fine-tuning its behavior, criteria, and entirely on feedback from the options it produces.

DeepSeek-R1-Zero is the base model for DeepSeek-R1.

The RL procedure for DeepSeek-R1-Zero

The training process for DeepSeek-R1-Zero included presenting the model with different thinking tasks, varying from mathematics issues to abstract reasoning challenges. The design produced outputs and was evaluated based on its performance.

DeepSeek-R1-Zero got feedback through a reward system that helped assist its knowing process:

Accuracy benefits: Evaluates whether the output is proper. Used for when there are deterministic outcomes (math issues).

Format rewards: Encouraged the model to structure its reasoning within and tags.

Training prompt template

To train DeepSeek-R1-Zero to produce structured chain of idea sequences, the scientists utilized the following prompt training design template, changing timely with the reasoning question. You can access it in PromptHub here.

This template triggered the model to explicitly outline its idea process within tags before providing the last answer in tags.

The power of RL in reasoning

With this training process DeepSeek-R1-Zero began to produce advanced reasoning chains.

Through thousands of training actions, DeepSeek-R1-Zero developed to solve increasingly complex issues. It found out to:

– Generate long thinking chains that allowed deeper and more structured analytical

– Perform self-verification to cross-check its own answers (more on this later).

– Correct its own mistakes, showcasing emerging self-reflective behaviors.

DeepSeek R1-Zero performance

While DeepSeek-R1-Zero is primarily a precursor to DeepSeek-R1, it still accomplished high performance on several criteria. Let’s dive into a few of the experiments ran.

Accuracy enhancements during training

– Pass@1 precision began at 15.6% and by the end of the training it enhanced to 71.0%, similar to OpenAI’s o1-0912 model.

– The red solid line represents performance with majority voting (similar to ensembling and self-consistency techniques), which increased precision further to 86.7%, going beyond o1-0912.

Next we’ll look at a table comparing DeepSeek-R1-Zero’s performance throughout several reasoning datasets versus OpenAI’s thinking models.

AIME 2024: 71.0% Pass@1, a little below o1-0912 but above o1-mini. 86.7% cons@64, beating both o1 and o1-mini.

MATH-500: Achieved 95.9%, beating both o1-0912 and o1-mini.

GPQA Diamond: Outperformed o1-mini with a score of 73.3%.

– Performed much worse on coding tasks (CodeForces and LiveCode Bench).

Next we’ll look at how the action length increased throughout the RL training process.

This graph shows the length of responses from the design as the training procedure progresses. Each “action” represents one cycle of the design’s learning process, where feedback is provided based upon the output’s performance, examined using the prompt design template discussed earlier.

For each question (corresponding to one step), 16 reactions were tested, and the average precision was calculated to guarantee stable examination.

As training advances, the model creates longer reasoning chains, allowing it to resolve progressively intricate thinking jobs by leveraging more test-time compute.

While longer chains do not always ensure better results, they normally associate with enhanced performance-a trend also observed in the MEDPROMPT paper (read more about it here) and in the initial o1 paper from OpenAI.

Aha minute and self-verification

Among the coolest elements of DeepSeek-R1-Zero’s advancement (which likewise applies to the flagship R-1 design) is just how good the model ended up being at reasoning. There were sophisticated thinking habits that were not explicitly programmed but emerged through its reinforcement learning process.

Over thousands of training steps, the model started to self-correct, reevaluate flawed reasoning, and confirm its own solutions-all within its chain of thought

An example of this kept in mind in the paper, described as a the “Aha minute” is below in red text.

In this instance, the model literally said, “That’s an aha moment.” Through DeepSeek’s chat feature (their variation of ChatGPT) this type of thinking typically emerges with phrases like “Wait a minute” or “Wait, however … ,”

Limitations and challenges in DeepSeek-R1-Zero

While DeepSeek-R1-Zero was able to carry out at a high level, there were some downsides with the design.

Language mixing and coherence concerns: The design occasionally produced actions that mixed languages (Chinese and English).

Reinforcement learning trade-offs: The lack of supervised fine-tuning (SFT) indicated that the model did not have the improvement needed for fully polished, human-aligned outputs.

DeepSeek-R1 was developed to attend to these concerns!

What is DeepSeek R1

DeepSeek-R1 is an open-source thinking design from the Chinese AI lab DeepSeek. It builds on DeepSeek-R1-Zero, which was trained completely with support learning. Unlike its predecessor, DeepSeek-R1 incorporates monitored fine-tuning, making it more improved. Notably, it exceeds OpenAI’s o1 model on a number of benchmarks-more on that later.

What are the primary distinctions in between DeepSeek-R1 and DeepSeek-R1-Zero?

DeepSeek-R1 constructs on the foundation of DeepSeek-R1-Zero, which works as the base design. The 2 differ in their training approaches and general efficiency.

1. Training approach

DeepSeek-R1-Zero: Trained entirely with support knowing (RL) and no supervised fine-tuning (SFT).

DeepSeek-R1: Uses a multi-stage training pipeline that includes monitored fine-tuning (SFT) initially, followed by the same reinforcement learning process that DeepSeek-R1-Zero damp through. SFT helps enhance coherence and readability.

2. Readability & Coherence

DeepSeek-R1-Zero: Struggled with language mixing (English and Chinese) and readability issues. Its thinking was strong, however its outputs were less polished.

DeepSeek-R1: Addressed these problems with cold-start fine-tuning, making reactions clearer and more structured.

3. Performance

DeepSeek-R1-Zero: Still a really strong reasoning design, sometimes beating OpenAI’s o1, but fell the language blending issues lowered usability greatly.

DeepSeek-R1: Outperforms R1-Zero and OpenAI’s o1 on a lot of thinking standards, and the reactions are a lot more polished.

Simply put, DeepSeek-R1-Zero was an evidence of concept, while DeepSeek-R1 is the completely optimized version.

How DeepSeek-R1 was trained

To take on the readability and coherence concerns of R1-Zero, the researchers included a cold-start fine-tuning stage and a multi-stage training pipeline when developing DeepSeek-R1:

Cold-Start Fine-Tuning:

– Researchers prepared a top quality dataset of long chains of idea examples for initial supervised fine-tuning (SFT). This information was gathered utilizing:– Few-shot triggering with detailed CoT examples.

– Post-processed outputs from DeepSeek-R1-Zero, fine-tuned by human annotators.

Reinforcement Learning:

DeepSeek-R1 underwent the very same RL procedure as DeepSeek-R1-Zero to fine-tune its reasoning capabilities further.

Human Preference Alignment:

– A secondary RL phase improved the model’s helpfulness and harmlessness, ensuring much better positioning with user needs.

Distillation to Smaller Models:

– DeepSeek-R1’s thinking capabilities were distilled into smaller sized, effective designs like Qwen and Llama-3.1 -8 B, and Llama-3.3 -70 B-Instruct.

DeepSeek R-1 criteria performance

The researchers checked DeepSeek R-1 throughout a variety of benchmarks and against leading models: o1, GPT-4o, and Claude 3.5 Sonnet, o1-mini.

The standards were broken down into several categories, shown below in the table: English, Code, Math, and Chinese.

Setup

The following specifications were applied throughout all designs:

Maximum generation length: 32,768 tokens.

Sampling setup:- Temperature: 0.6.

– Top-p worth: 0.95.

– DeepSeek R1 surpassed o1, Claude 3.5 Sonnet and other models in the bulk of reasoning criteria.

o1 was the best-performing model in 4 out of the 5 coding-related criteria.

– DeepSeek performed well on imaginative and long-context task job, like AlpacaEval 2.0 and ArenaHard, surpassing all other models.

Prompt Engineering with reasoning models

My preferred part of the short article was the scientists’ observation about DeepSeek-R1’s sensitivity to triggers:

This is another datapoint that lines up with insights from our Prompt Engineering with Reasoning Models Guide, which recommendations Microsoft’s research study on their MedPrompt framework. In their research study with OpenAI’s o1-preview model, they found that frustrating thinking models with few-shot context deteriorated performance-a sharp contrast to non-reasoning designs.

The key takeaway? Zero-shot triggering with clear and concise instructions appear to be best when utilizing reasoning designs.