Retrieval Augmented Generation (RAG)
General Purpose language model can be fine tuned to achieve several common tasks such as sentiment analysis and Named Entity Recognition (NER) . These tasks don’t require any additional information.
For more complex and knowledge intensive tasks, it’s possible to build a Large Language Model (LLM) that has access to external knowledge sources to complete tasks. This enable more factual consistency, improve reliability of generated responses, this helps to mitigate the problem of “hallucination”
Meta AI researchers introduced a method called Retrieval Augmented Generation (RAG) to address such knowledge intensive tasks. RAG combines an information retrieval component such as a Vector Stores with a text generation component. RAG models can be fine tuned in an efficient manner without retraining the entire model.
- RAG takes an input and retrieves a set of information from a source.
- the information is then concatenated and passed as input to the Large Language Model (LLM)
This helps us bypass the learning stage and also have access to the latest information at any point of time, as the LLM’s parametric knowledge is static, using this the model can give us reliable information with retrieval-based generation and this can be further enhanced with Prompt Engineering
The below is an illustration of how RAG can be integrated with a parametric model with non parametric input data.
RAG performs strong on several benchmarks such as Natural Questions, WebQuestions, and CuratedTrec. RAG generates responses that are more factual, specific, and diverse when tested on MS-MARCO and Jeopardy questions. RAG also improves results on FEVER fact verification.
Source: promptingguide.ai
This shows the potential for RAG based applications as a viable option for enhancing outputs of language model in knowledge-intensive tasks
more recently, these retriever based approaches have become more and more popular and are combined with LLMs to improve the capabilities and factual consistency
References