Skip to content
Retrieval-Augmented Generation, Described in a 22 May 2020 Paper by Patrick Lewis and 11 Co-Authors, Lets a Chatbot Fetch Passages From an Index of 21 Million Before It Answers
AI & Technology

Retrieval-Augmented Generation, Described in a 22 May 2020 Paper by Patrick Lewis and 11 Co-Authors, Lets a Chatbot Fetch Passages From an Index of 21 Million Before It Answers

Illustration by tuput

English

Retrieval-augmented generation, or RAG, has a language model look up passages in a document collection and write its answer from them. The 2020 paper that named it set a state of the art on three open-domain question tasks, and its authors showed the model's answers change when the documents are swapped, with no retraining.

· · 7 min read

A language model keeps what it learned in training inside its own numbers, and updating those numbers means training it again. In a paper submitted to arXiv on 22 May 2020 and accepted at NeurIPS 2020, Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni and eight co-authors described a different arrangement. Their model, called retrieval-augmented generation or RAG, first searches a collection of documents and then writes its answer with the best passages in front of it. It set a state of the art on three open-domain question-answering tasks, and it let the authors change what the model knew by changing the collection.

The four steps, in plain words

The pipeline has the same shape in the 2020 paper and in the systems built since. Anthropic’s 2024 engineering post lists the preparation steps: cut the documents into chunks of text, usually no more than a few hundred tokens (a token is a short piece of a word); use an embedding model to turn each chunk into a vector, a list of numbers that encodes meaning; and store the vectors in a database that can be searched by similarity.

At question time the system turns the question into a vector the same way, fetches the chunks whose vectors sit closest to it, and adds them to the prompt sent to the text generator. Gao and nine co-authors, in a 2023 survey of the field, call this three-stage version “naive RAG”: indexing, retrieval, generation.

The numbers in the original are specific. Lewis and colleagues took the December 2018 English Wikipedia dump and split every article into disjoint 100-word chunks, which gave 21 million documents. They searched the index with a fast approximate method, retrieved 5 or 10 passages per question in training, and fed them with the question to BART-large, a 400-million-parameter text generator.

The retriever that came first

The searching half came from a paper submitted on 10 April 2020, six weeks before Lewis’s. Vladimir Karpukhin and seven co-authors, including Lewis, built dense passage retrieval, which matches questions to passages by meaning rather than by shared words. Their paper reports that it beat a strong BM25 system, the long-standing word-matching method, by 9 to 19 points of absolute accuracy at the top 20 passages. On the Natural Questions dataset the figures were 78.4 percent against 59.1 percent.

Meta AI’s blog post of 28 September 2020, written by Sebastian Riedel, Douwe Kiela, Patrick Lewis and Aleksandra Piktus, describes RAG as Facebook AI’s dense passage retriever joined to BART. An earlier approach, REALM, from Kelvin Guu and four co-authors on 10 February 2020, trained a retriever during the language model’s pre-training. Lewis’s paper says RAG reaches strong results without that expensive special pre-training, and that no separate re-ranker is needed for state-of-the-art performance.

The swap test

The Meta post puts the practical gain in one sentence: with RAG, “we control what it knows simply by swapping out the documents”. The paper gives the experiment behind it, which it calls index hot-swapping.

The authors built a second index from a December 2016 Wikipedia dump and made a list of 82 world leaders who had changed between the two dates. They asked the model “Who is the President of Peru?” and similar questions of each index. It answered 70 percent correctly with the 2016 index for the 2016 leaders and 68 percent with the 2018 index for the 2018 leaders. With the wrong year’s index, accuracy fell to 12 percent and 4 percent. The paper’s reading is that the model’s world knowledge can be updated by replacing its non-parametric memory, its term for the document store.

The same paper notes that the memory is raw text, so it is human-readable and human-writable. In a human comparison on Jeopardy-style question writing, judges found RAG more factual than the BART baseline in 42.7 percent of 452 paired outputs, and BART more factual in 7.1 percent.

Meta’s post says the model was released as part of Hugging Face’s Transformers library. Hugging Face’s documentation page dates the model’s contribution to that library to 16 November 2020 and describes the idea in the same terms: fetch relevant passages, then condition the answer on them, so that knowledge is updated by changing the index.

Where chunks lose their context

The weak point is the cutting. Anthropic’s post gives an example: a chunk reading “The company’s revenue grew by 3% over the previous quarter” says nothing about which company or which quarter, so a search for a particular firm’s results can miss it. Anthropic’s fix, Contextual Retrieval, has a model write a short note for each chunk, usually 50 to 100 tokens, saying where in the document it comes from, and puts that note in front of the chunk before indexing.

Anthropic reports its results as the share of relevant passages that failed to appear in the top 20 retrieved, averaged over codebases, fiction, arXiv papers and science papers:

  • Ordinary embeddings: 5.7 percent failed.
  • With contextual embeddings: 3.7 percent.
  • With contextual embeddings and contextual BM25: 2.9 percent.
  • With a reranking step added: 1.9 percent.

The figures are Anthropic’s own tests on its own datasets, run with Gemini’s Text 004 embeddings and a Cohere reranker, and the post does not describe independent replication. The same post adds a limit on the whole approach: if a knowledge base is smaller than about 200,000 tokens, roughly 500 pages, it can simply go into the prompt with no retrieval at all.

What retrieval does not remove

Retrieval lowers the rate of invented answers and does not bring it to zero. Shuster and four co-authors reported in 2021 that retrieval inside a chatbot substantially reduced hallucination, as judged by human evaluators. The Gao survey lists the failure that remains: the model can still write content the retrieved text does not support, and the retrieval step itself can pick misaligned chunks.

The clearest test of a product claim is in law. Magesh and five co-authors evaluated commercial legal research tools that use RAG and found each hallucinated between 17 and 33 percent of the time, less often than GPT-4 as a general-purpose chatbot, and well above zero, against vendor claims of “hallucination-free” legal citations. The tuput explainer on AI hallucination covers that study and the court orders that followed, and the one on how a large language model is trained explains the stored-in-the-numbers knowledge that RAG supplements.

Lewis’s own abstract lists provenance, the ability to show where an answer came from, as an open research problem. The 2023 survey is more optimistic: it says retrieval-based generation can quickly locate the original references so that users can verify an answer. Whether a given product shows those references depends on how it was built.

An Indian example, with limits on what is known

India’s Ministry of Agriculture and Farmers Welfare launched Phase 1 of Bharat-VISTAAR on 17 February 2026. Its release of 16 February says the platform delivers information to farmers by connecting government and scientific sources, with weather from the India Meteorological Department, mandi prices from AgMarkNet, pest and disease advice from the National Pest Surveillance System, AgriStack data and 10 central schemes. A farmer can call 155261 in Hindi or English, and the release calls the platform “based on verified and secure information”.

Neither that release nor the Ministry’s reply in Rajya Sabha on 13 March 2026 describes the software underneath or says the system uses retrieval-augmented generation, so this article does not claim it does. What the pages describe is the design RAG exists to support: answers meant to come from named official records rather than from a model’s memory. Another tuput article sets Bharat-VISTAAR beside other documented farm deployments.

The 13 March reply adds that crop practices from the Indian Council of Agricultural Research are part of the platform, and that four more languages are planned within three months of launch and five more within six.

Share
Copied!

Sources & further reading

  1. Lewis et al.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv:2005.11401, submitted 22 May 2020)
  2. Lewis et al.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, full paper (PDF)
  3. NeurIPS 2020 Proceedings: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  4. Meta AI blog: Retrieval Augmented Generation, streamlining the creation of intelligent natural language processing models (28 September 2020)
  5. Karpukhin et al.: Dense Passage Retrieval for Open-Domain Question Answering (arXiv:2004.04906)
  6. Karpukhin et al.: Dense Passage Retrieval for Open-Domain Question Answering, full paper (PDF)
  7. Guu et al.: REALM, Retrieval-Augmented Language Model Pre-Training (arXiv:2002.08909, 10 February 2020)
  8. Hugging Face Transformers documentation: RAG
  9. Anthropic: Introducing Contextual Retrieval (19 September 2024)
  10. Gao et al.: Retrieval-Augmented Generation for Large Language Models, A Survey (arXiv:2312.10997)
  11. Gao et al.: Retrieval-Augmented Generation for Large Language Models, A Survey, full paper (PDF)
  12. Magesh et al.: Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools (arXiv:2405.20362)
  13. Shuster et al.: Retrieval Augmentation Reduces Hallucination in Conversation (arXiv:2104.07567)
  14. PIB, Ministry of Agriculture and Farmers Welfare: Bharat-VISTAAR to be launched on 17 February (16 February 2026)
  15. PIB, Ministry of Agriculture and Farmers Welfare: Rollout of BHARAT-VISTAAR Platform, Rajya Sabha reply (13 March 2026)

Researched and written with the help of AI tools and edited for accuracy. Provided for general information and discussion only, not professional advice. See our editorial standards and disclaimer. Spotted an error? Tell us.

#retrieval-augmented generation#rag#large language models#dense passage retrieval#contextual retrieval#ai hallucination#bharat-vistaar#embeddings

Enjoyed this? Get the next one.

One good read at a time, straight to your inbox. No spam, unsubscribe anytime.

More in AI & Technology
Two New York Lawyers and Their Firm Paid $5,000 in 2023 for Six Court Cases ChatGPT Invented, and by October 2026 a Public Tracker Listed 2,202 Court Decisions Involving AI Hallucinations
Orders from New York, Alabama and the Supreme Court of India, set beside the research papers on why chatbots invent facts. They show how a made-up citation reaches a judge, what has cut the error rate, and how courts have chosen to respond.
Eight Google Authors Posted the Transformer Paper on 12 June 2017, and Its Big Model Scored 28.4 BLEU in English to German After 3.5 Days on Eight GPUs
A walk through the design behind today's chatbots, using the paper's own tables and Google's own example of a word that has to look at a river. It also covers the parts of the 2017 design that later models kept, changed or dropped.
Bharat-VISTAAR, Launched in Jaipur on 17 February 2026, Lets a Farmer Ask an AI Adviser Questions by Phone Call on 155261 and Draws on ICAR, IMD, Agmarknet and AgriStack Data
One call to 155261 reaches an AI adviser fed by ICAR practice guides, weather and mandi data. The launch releases, the language plan and the usage figures published so far, set out with dates and sources.
← all articles