Skip to content
DeepSeek-R1's Score on a US Maths Contest Rose From 15.6% to 77.9% During Training With Rewards for Correct Answers Alone, and Nature Published the Referees' Reports Beside the Paper
AI & Technology

DeepSeek-R1's Score on a US Maths Contest Rose From 15.6% to 77.9% During Training With Rewards for Correct Answers Alone, and Nature Published the Referees' Reports Beside the Paper

Illustration by tuput

English

DeepSeek's R1 paper appeared in Nature on 17 September 2025 (volume 645, pages 633 to 638). It reports that a language model can learn to write long, self-checking solutions when it is rewarded only for right final answers, with no human-written worked examples to copy. The journal also published the reports of eight referees and the authors' replies.

· · 9 min read

On 17 September 2025 Nature published the paper behind DeepSeek-R1, the reasoning model from the Hangzhou company DeepSeek, in volume 645, pages 633 to 638. Its central result is that a language model can learn to write long, self-checking solutions when it is rewarded only for correct final answers, with no human-written worked examples to copy. On the 2024 American Invitational Mathematics Examination (AIME), a US maths contest for high-school students whose answers are single numbers, the first version of the model, R1-Zero, rose from an average of 15.6% correct to 77.9% over the course of training, according to the paper’s Figure 1. A majority vote across several sampled answers lifted it to 86.7%.

Nature published the referees’ reports and the authors’ replies alongside the paper, and its supplementary pages give the training bill: 294,000 dollars at the authors’ assumed GPU rental price. What follows explains how such a model is trained, where each idea came from, and what peer review changed.

What the paper claims

R1-Zero starts from DeepSeek-V3 Base, a model already pretrained on web text (the pretraining stage is covered in our training explainer). The authors then applied reinforcement learning, a trial-and-error method in which the model’s own answers are scored and the model is nudged toward the ones that score well. They skipped the usual supervised fine-tuning step, in which people supply worked examples for the model to imitate. The reward depended only on whether the final answer matched the known correct one. The only instruction on form was a template asking the model to put its reasoning between think tags and its answer between answer tags.

Over training, the paper says, responses grew longer and began to include checking, reflection and the exploration of alternative approaches, although the authors did not explicitly teach the model how to reason. R1-Zero had faults: its text was hard to read and sometimes mixed English and Chinese in one answer. So the team built DeepSeek-R1 in stages. It starts with thousands of cold-start examples written in a conversational style, adds reinforcement learning, then rejection sampling (keeping only the model’s good outputs) with a second round of supervised fine-tuning, and ends with a second reinforcement stage aimed at helpfulness and harmlessness.

DeepSeek’s release note dates the model to 20 January 2025 and says the code and models carry the MIT licence, with six smaller distilled models released too. On the model card, DeepSeek lists R1 at 79.8% on AIME 2024 against 39.2% for DeepSeek-V3, the model it was built on, using 64 sampled answers per question at temperature 0.6. The model’s successors are covered in our look at Chinese open-weight models. The Hugging Face page for the model listed about 51.5 million downloads over its lifetime when we opened it on 12 November 2026.

Three earlier ideas the paper builds on

The idea that a model does better when it writes out its steps came first from prompting. Jason Wei and colleagues at Google posted the chain-of-thought paper on 28 January 2022. They report that a 540-billion-parameter model, shown just eight worked examples, reached state-of-the-art accuracy on GSM8K, a set of school maths word problems. On 24 May 2022 Takeshi Kojima and colleagues showed that adding the words “Let’s think step by step” raised one large model’s accuracy from 17.7% to 78.7% on MultiArith and from 10.4% to 40.7% on GSM8K, with no examples at all.

A third strand was feedback on the steps themselves. In a paper posted on 31 May 2023, OpenAI researchers compared rewarding only the final result with rewarding each intermediate step. Step-level feedback won on the MATH dataset, with the best model solving 78% of a representative subset, and the team released PRM800K, 800,000 step-level labels written by people.

Each of these needed people: for examples, or for labels on every step. The Nature paper’s introduction argues that depending on human-annotated reasoning traces slows scaling and caps a model at the quality of the examples it was shown. Its test was whether a base model could find good reasoning strategies from the reward alone.

Why a maths answer can be scored by a machine

The authors use two kinds of rule-based reward on reasoning tasks. An accuracy reward checks the answer: a maths problem requires the final answer in a set format, such as inside a box, so a program can compare it with the key, and code is run by a compiler against predefined test cases. A format reward checks that the reasoning sits inside the tags. The two are added with equal weight.

They chose not to use a learned reward model, a second neural network that scores answers, on reasoning tasks. The paper says such models are open to reward hacking in large-scale training, meaning the model learns to please the scorer rather than solve the problem, and that retraining them costs extra computing.

The learning method is Group Relative Policy Optimization, or GRPO. For each question the model writes a group of answers, 16 in R1-Zero’s run. Each answer’s reward is compared with the group’s average and spread, and answers above average are made more likely. The Nature paper’s figure caption says this estimates the advantage from the group’s scores directly, without the separate value function used in the older PPO method. DeepSeek’s team introduced GRPO in the DeepSeekMath paper of 5 February 2024, where a 7-billion-parameter model reached 51.7% on the MATH benchmark; the Nature paper cites that work and describes the method in its Methods section.

R1-Zero trained for 10,400 steps. Each step used 32 questions with 16 answers each, so 512 answers, with a length cap of 32,768 tokens that was raised to 65,536 after step 8,200.

What the training curves showed

The paper reports that average response length rose steadily during training, and that the model came to write hundreds to thousands of tokens while checking its own work. Its Extended Data figure counts reflective words such as “wait”, “mistake” and “verify” in the outputs. The word “wait” was almost absent early, appeared now and then between steps 4,000 and 7,000, and rose sharply after step 8,000. The authors call an intermediate example an “aha moment”.

Referees treated this reading with care. One wrote that the effect had been reproduced at small scale by another group, and added that a separate paper suggests the aha moment may largely reflect the quality of the base model. Another noted that all the evaluations score final answers only, not the intermediate text, so the written steps are not measured for coherence. This article therefore describes them as intermediate steps the model writes out, and does not claim the model thinks.

What the training bill covers

The supplementary information lists R1-Zero at 101,000 H800 GPU hours, 5,000 hours for creating fine-tuning data and 41,000 hours for R1, 147,000 in all. At an assumed rental price of 2 dollars per GPU hour that comes to 294,000 dollars. The run used 64 by 8 H800 GPUs, which is 512 chips, for about 198 hours for R1-Zero and about 80 hours for R1. The authors say they prepared with A100 GPUs on a 30-billion-parameter model first.

The figure covers the reasoning stages only. Scientific American, reporting for Nature, notes that it comes on top of the roughly 6 million dollars DeepSeek spent on the base model. The supplementary pages describe DeepSeek-V3 as having 671 billion parameters, 37 billion of them active for each token, trained on 14.8 trillion tokens.

What the referees asked, and what changed

The manuscript was received on 14 February 2025, accepted on 17 July 2025 and published on 17 September 2025. Nature’s editorial says eight specialists reviewed it and that the reports and replies were published with it. Lewis Tunstall of Hugging Face, one of the named reviewers, told Nature the practice was “a very welcome precedent”. Huan Sun of Ohio State University said, “Other firms should do the same.”

Several changes are documented. One referee urged the authors to cut down on describing the model’s intermediate text in human terms. The authors removed the wording, and in their reply acknowledged that human-like patterns may lead users to trust the output too much, adding that risk to the revised text. Referees also asked about contamination, where test questions leak into training data (the audits of AI benchmarks show why makers’ figures are read with care). According to Nature’s editorial, DeepSeek supplied details of its safeguards and added evaluations on benchmarks published after the model’s release. The supplementary pages report 75% on AIME 2025 against 80% for OpenAI’s o1-1217, and 143.7 out of 150 on the AMC 12 2024 contest against 141.0 for o1-1217, all in DeepSeek’s own tests. They also concede that the n-gram filtering used against contamination cannot catch paraphrased test questions, so benchmarks released before 2024 may still be affected.

A third change was a safety section. Nature’s editorial says reviewers pointed to a lack of information about safety tests, and the paper now compares R1 with rival models; it concludes that R1’s inherent safety level is moderate, comparable to GPT-4o of 13 May 2024. On the question of OpenAI data, Scientific American reports that DeepSeek told referees R1 did not learn by copying reasoning examples generated by OpenAI models, while acknowledging that its base model was trained on web pages that will contain AI-generated text. Sun called the rebuttal “as convincing as what we could see in any publication”; Tunstall said he could not be fully certain, but that other labs’ replication attempts suggest the recipe works without it. The training data itself was not published, a limit one referee flagged for reproducibility.

What the authors list as unsolved

The paper’s own limitations section names several. R1’s structured output is weaker than that of existing models, and it cannot use tools such as search engines or calculators. It sometimes spends many tokens on easy questions. It is tuned for Chinese and English and may reason in English even when asked in another language. Few-shot prompting makes it worse, so the authors recommend describing the problem directly. Because software-engineering tests take long to run, large-scale reinforcement learning was barely applied there, and R1 shows no large gain over DeepSeek-V3 on those benchmarks.

The method needs a reliable verifier. For tasks such as writing, where no rule can score the answer, the authors used human-annotated data and only hundreds of reinforcement steps. A model-based reward for general data was confined to the last 400 of the second stage’s 1,700 steps, because longer exposure led to reward hacking.

Share
Copied!

Sources & further reading

  1. Guo and others (DeepSeek-AI): DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning, Nature 645, 633 to 638 (17 September 2025)
  2. Nature: Supplementary Information for the DeepSeek-R1 paper (cost table, evaluation settings, decontamination, fresh test sets)
  3. Nature: Peer Review File for the DeepSeek-R1 paper (referee reports and author responses)
  4. arXiv: DeepSeek-R1, Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (arXiv:2501.12948, submitted 22 January 2025, revised 4 January 2026)
  5. Ippolito and Zhang, Nature News and Views: AI can learn to show its workings through trial and error (17 September 2025)
  6. Nature editorial: Bring us your LLMs, why peer review is good for AI models (17 September 2025)
  7. Gibney, Scientific American (with Nature): Secrets of DeepSeek AI model revealed in landmark paper (17 September 2025)
  8. Wei and others (Google): Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (arXiv:2201.11903, 28 January 2022)
  9. Kojima and others: Large Language Models are Zero-Shot Reasoners (arXiv:2205.11916, 24 May 2022)
  10. Lightman and others (OpenAI): Let's Verify Step by Step (arXiv:2305.20050, 31 May 2023)
  11. Shao and others (DeepSeek): DeepSeekMath, Pushing the Limits of Mathematical Reasoning in Open Language Models (arXiv:2402.03300, 5 February 2024)
  12. DeepSeek: DeepSeek-R1 model card on Hugging Face (MIT licence, evaluation table)
  13. DeepSeek API Docs: DeepSeek-R1 Release (20 January 2025)

Researched and written with the help of AI tools and edited for accuracy. Provided for general information and discussion only, not professional advice. See our editorial standards and disclaimer. Spotted an error? Tell us.

#reasoning models#deepseek-r1#chain of thought#reinforcement learning#grpo#peer review#nature#ai training

Enjoyed this? Get the next one.

One good read at a time, straight to your inbox. No spam, unsubscribe anytime.

More in AI & Technology
Google DeepMind's AI Cyclone Model Averaged a 230 Kilometre Five-Day Track Error Against 370 for the European Ensemble in a Nature Paper, and Its Own Authors Call Rapid Intensification an Unsolved Problem
The model was scored on storms it had never seen, and the live check covers the Atlantic and East Pacific. Bangladesh, Myanmar and India recorded 84% of cyclone deaths from 1970 to 2019 by WMO's count, and the North Indian Ocean appears in the supplement as one labelled map panel.
Eight Google Authors Posted the Transformer Paper on 12 June 2017, and Its Big Model Scored 28.4 BLEU in English to German After 3.5 Days on Eight GPUs
A walk through the design behind today's chatbots, using the paper's own tables and Google's own example of a word that has to look at a river. It also covers the parts of the 2017 design that later models kept, changed or dropped.
Bharat-VISTAAR, Launched in Jaipur on 17 February 2026, Lets a Farmer Ask an AI Adviser Questions by Phone Call on 155261 and Draws on ICAR, IMD, Agmarknet and AgriStack Data
One call to 155261 reaches an AI adviser fed by ICAR practice guides, weather and mandi data. The launch releases, the language plan and the usage figures published so far, set out with dates and sources.
← all articles