{"id":8219,"date":"2023-11-30T06:18:54","date_gmt":"2023-11-30T14:18:54","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=8219"},"modified":"2025-04-24T17:04:06","modified_gmt":"2025-04-24T17:04:06","slug":"evaluating-rag-pipelines-with-ragas","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/evaluating-rag-pipelines-with-ragas\/","title":{"rendered":"Evaluating RAG Pipelines With\u00a0ragas"},"content":{"rendered":"\n
\n

A Guide to Metrics and Stuffing Strategy Assessment<\/span><\/h2>\n
\n
\n
\n<\/figure><\/div><\/div><\/section>\n\n\n\n
\"Evaluating
Photo by Austin Distel<\/a> on\u00a0Unsplash<\/a><\/figcaption><\/figure>\n\n\n\n

In this post, you will learn how to set up and evaluate Retrieval-Augmented Generation (RAG<\/a>) pipelines using LangChain.<\/p>\n\n\n\n

You will explore the impact of different chain types\u200a\u2014\u200aMap Reduce, Stuff, Refine, and Re-rank\u200a\u2014\u200aon the performance of your RAG pipeline. This guide is a practical introduction to using the ragas<\/strong><\/a> library for RAG pipeline evaluation. Starting with fundamental concepts, you\u2019ll learn how different configurations affect your results.<\/p>\n\n\n\n

This post is designed for those with a technical background in natural language processing and AI, offering detailed guidance on optimizing and evaluating RAG pipelines for improved performance.<\/p>\n\n\n\n

It\u2019s a bare-bones blog, and my thoughts are not yet fully fleshed out. But this is an excellent introduction to using the ragas<\/code> library for evaluating your RAG pipelines. In the future, I would like to look at assessing the chunk_size<\/code> for the RAG metrics. If you\u2019re interested in working with me on a more extensive study of the concepts introduced in this blog, please reach out!<\/strong><\/p>\n\n\n\n

Start by getting some preliminaries out of the way:<\/p>\n\n\n\n

%%capture\n!pip install langchain openai tiktoken faiss-cpu ragas\n\nimport<\/span> os\nimport<\/span> getpass\n\nos.environ[\"OPENAI_API_KEY\"<\/span>] = getpass.getpass(\"Enter Your OpenAI API Key:\"<\/span>)\n\nfrom<\/span> langchain.embeddings.openai import<\/span> OpenAIEmbeddings\nfrom<\/span> langchain.vectorstores import<\/span> FAISS\nfrom<\/span> langchain.text_splitter import<\/span> RecursiveCharacterTextSplitter\nfrom<\/span> langchain.chat_models import<\/span> ChatOpenAI\nfrom<\/span> langchain.chains import<\/span> RetrievalQA\nfrom<\/span> langchain.document_loaders import<\/span> WebBaseLoader<\/span><\/pre>\n\n\n\n
Now, download some text files that will be put into a vector database. I\u2019ve written about this in other blogs. Feel free to check them out if you do not understand what\u2019s happening below.<\/p>\n\n\n\n
In a nutshell, the code loads some websites into document objects, splits them into chunks of 1000 characters each, and puts those chunks into a vector database.<\/p>\n\n\n\n
# The websites for the text you want to load<\/span>\nwebsites = [\n    \"https:\/\/www.gutenberg.org\/files\/56075\/56075-h\/56075-h.htm#SENECA_OF_A_HAPPY_LIFE\"<\/span>,\n    \"https:\/\/www.gutenberg.org\/files\/56075\/56075-h\/56075-h.htm#SENECA_OF_ANGER\"<\/span>,\n    \"https:\/\/www.gutenberg.org\/files\/10661\/10661-h\/10661-h.htm\"<\/span>,\n    \"https:\/\/www.gutenberg.org\/cache\/epub\/17183\/pg17183-images.html\"<\/span>\n    ]\n\n# Use the WebBaseLoader to create Document objects for each webside<\/span>\nweb_loader = WebBaseLoader(websites)\nweb_docs = web_loader.load()\n\n# Instantiate a text splitter<\/span>\ntext_splitter = RecursiveCharacterTextSplitter(\n    chunk_size=1000<\/span>,\n    chunk_overlap=50<\/span>\n    )\n\n\nweb_texts = text_splitter.split_documents(web_docs)\n\nweb_db = FAISS.from_documents(\n    web_texts,\n    embeddings,\n)<\/span><\/pre>\n\n\n\n
Here\u2019s an example of what one chunk of the split text looks like:<\/p>\n\n\n\n
web_texts[42<\/span>]<\/span><\/pre>\n\n\n\n
Document(page_content=\u201929\\nCHAPTER II.\\nSEVERAL SORTS OF BENEFITS.\\nWe shall divide benefits into absolute and vulgar;\\r\\nthe one appertaining to good life, the other is only\\r\\nmatter of commerce. The former are the more excellent,\\r\\nbecause they can never be made void; whereas\\r\\nall material benefits are tossed back and forward,\\r\\nand change their master. There are some offices\\r\\nthat look like benefits, but are only desirable conveniences,\\r\\nas wealth, etc., and these a wicked man\\r\\nmay receive from a good, or a good man from an\\r\\nevil. Others, again, that bear the face of injuries,\\r\\nwhich are only benefits ill taken; as cutting, lancing,\\r\\nburning, under the hand of a surgeon. The greatest\\r\\nbenefits of all are those of good education, which\\r\\nwe receive from our parents, either in the state of\\r\\nignorance or perverseness; as, their care and tenderness\\r\\nin our infancy; their discipline in our childhood,\\r\\nto keep us to our duties by fear; and, if fair means\\r\\nwill not do, their proceeding afterwards to severity\u2019, metadata={\u2018source\u2019: \u2018https:\/\/www.gutenberg.org\/files\/56075\/56075-h\/56075-h.htm#SENECA_OF_A_HAPPY_LIFE<\/a>', \u2018title\u2019: \u201cThe Project Gutenberg eBook of Seneca\u2019s Morals of a Happy Life, Benefits, Anger and Clemency, by Lucius Annaeus Seneca\u201d, \u2018language\u2019: \u2018No language found.\u2019})<\/pre>\n\n\n\n
The code below sets up a RetrievalQA chain, which will be used to retrieve documents using one of four strategies:<\/h4>\n\n\n\n
    \n
  1. Map reduce<\/li>\n\n\n\n
  2. Stuff<\/li>\n\n\n\n
  3. Refine<\/li>\n\n\n\n
  4. Re-rank<\/li>\n<\/ol>\n\n\n\n
    For this blog post, you\u2019ll use the following question to retrieve relevant documents from the vector database.<\/p>\n\n\n\n
    question = \"What does Seneca say are the qualities of a happy life and how can happiness be achieved?\"<\/span><\/span><\/pre>\n\n\n\n
    def<\/span> get_chain_result<\/span>(chain_type, llm, retriever, question<\/span>):\n    \"\"\"\n    Initialize a chain of the specified type and invoke it with the given question.\n\n    Parameters:\n    - chain_type (str): The type of the chain (e.g., \"map_reduce\", \"stuff\", \"refine\", \"map_rerank\").\n    - llm: The language model.\n    - retriever: The retriever object.\n    - question (str): The question to be asked.\n\n    Returns:\n    - dict: The result of invoking the chain with the question.\n    \"\"\"<\/span>\n    chain = RetrievalQA.from_chain_type(\n        llm=llm,\n        chain_type=chain_type,\n        retriever=retriever,\n        verbose=True<\/span>,\n        return_source_documents=True<\/span>\n    )\n    result = chain.invoke(question)\n    return<\/span> result\n\nretriever = web_db.as_retriever()<\/span><\/pre>\n\n\n\n
    \n
    \n
    \n<\/div>\n
    \n
    \n
    Want to learn how to build modern software with LLMs using the newest tools and techniques in the field? Check out this free LLMOps course<\/a> from industry expert Elvis Saravia of DAIR.AI!<\/p><\/blockquote>\n<\/div>\n<\/div>\n<\/section>\n\n\n\n
    \n
    \n
    \n<\/div>\n
    \n
    \n
    Map Reduce<\/h3>\n
    Consists of a map step, where each document is individually summarized, and a reduce step where these mini-summaries are combined. An optional compression step can be added.<\/p>\n
    \n<\/figure><\/div><\/div><\/section>\n\n\n\n
    \"Evaluating
    Photo by British Library<\/a> on\u00a0Unsplash<\/a><\/figcaption><\/figure>\n\n\n\n
    This method runs an initial prompt on each chunk of data and then uses a different prompt to combine all the initial outputs.<\/p>\n\n\n\n
    map_reduce<\/code> separates texts into batches, where you can define the batch size (i.e. llm=OpenAI(batch_size=5)<\/code>).<\/p>\n\n\n\n
    It feeds each batch with the question to LLM separately and comes up with the final answer based on the answers from each batch.<\/p>\n\n\n\n
    Pros:<\/strong> It can scale to more documents and documents of larger length. Since the calls to the LLM are on independent, individual documents they can be parallelized.<\/p>\n\n\n\n
    Cons:<\/strong> This requires more calls to the LLM. You can also get some information during the final combined call.<\/p>\n\n\n\n
    The code below will run a<\/p>\n\n\n\n
    map_reduce_result = get_chain_result(\"map_reduce\"<\/span>, llm, retriever, question)\nmap_reduce_result['result'<\/span>]<\/span><\/pre>\n\n\n\n
    According to the provided text, Seneca views wisdom and virtue as the qualities of a happy life. Happiness can be achieved by first understanding what one ought to do (wisdom) and then living in accordance with that knowledge (virtue).<\/p>\n\n\n\n
    Stuff<\/h3>\n\n\n\n
    Takes multiple small documents and combines them into a single prompt for the LLM. It is cost-efficient because it involves only one request to the LLM.<\/p>\n\n\n\n
    \n<\/figure>\n\n\n\n
    \"Evaluating
    Photo by Luca Laurence<\/a> on\u00a0Unsplash<\/a><\/figcaption><\/figure>\n\n\n\n
    The chain_type<\/code> “stuff” is the simplest method. It uses all related text from the documents as the context in the prompt to the LLM.<\/p>\n\n\n\n
    Pros: You only make a single call to the LLM. And when it generates text, it has access to all the data simultaneously. It\u2019s cost-efficient since you only make one call to the LLM<\/p>\n\n\n\n
    Cons: This can sometimes exceed the token limit for a LLM.<\/p>\n\n\n\n
    Use this when you have a bunch of small documents and not too many of them.<\/p>\n\n\n\n
    stuff_result = get_chain_result(\"stuff\"<\/span>, llm, retriever, question)\nstuff_result['result'<\/span>]<\/span><\/pre>\n\n\n\n
    Seneca states that happiness is founded upon wisdom and virtue. To achieve happiness, one must first know what they ought to do and then live according to that knowledge. He emphasizes the importance of philosophy and precept as aids toward a happy life, as well as the blessing of a good conscience. According to Seneca, a good man can never be miserable, nor a wicked man happy, and no man is unfortunate if he cheerfully submits to Providence.<\/p>\n\n\n\n
    Refine<\/h3>\n\n\n\n
    Looks at each document individually and updates its answer with each new document.<\/em><\/p>\n\n\n\n
    \n<\/figure>\n\n\n\n
    \"\"\/
    Photo by John Jennings<\/a> on\u00a0Unsplash<\/a><\/figcaption><\/figure>\n\n\n\n
    It is useful when there are too many documents, but it can be slow and confusing if they reference each other.<\/p>\n\n\n\n
    \u201crefine\u201d also separates texts into batches, but it feeds the first batch to LLM, the answer, and the second to LLM.<\/p>\n\n\n\n
    It refines the answer by going through all the batches.<\/p>\n\n\n\n
    refine_results = get_chain_result(\"refine\"<\/span>, llm, retriever, question)\nrefine_results['result'<\/span>]<\/span><\/pre>\n\n\n\n
    With the additional context from Seneca\u2019s Chapter IX, \u201cOf Levity of Mind, and Other Impediments of a Happy Life,\u201d we gain a clearer understanding of Seneca\u2019s perspective on the qualities of a happy life and how happiness can be achieved. The refined answer, incorporating the new information, would be as follows:<\/pre>\n\n\n\n
    Seneca, the Stoic philosopher, posits that the foundation of a happy life is built upon wisdom and virtue. To achieve happiness, one must first comprehend what is right and proper (wisdom), and then live in accordance with that understanding (virtue). Happiness, in Seneca\u2019s view, is not a matter of external circumstances but is intrinsically linked to one\u2019s inner moral state and the quality of one\u2019s character.<\/pre>\n\n\n\n
    The guidance of philosophy and the instruction of precepts are instrumental in leading one towards a life of happiness. These help individuals to cultivate the wisdom and virtue necessary to live in harmony with nature and their own rational nature.<\/pre>\n\n\n\n
    A critical quality of a happy life, according to Seneca, is the possession of a good conscience. A person who acts rightly will maintain inner peace and contentment, knowing that their actions align with their values and understanding of what is good.<\/pre>\n\n\n\n
    Another quality is the cheerful submission to Providence. Seneca emphasizes that a person who accepts their fate and understands the role of Providence in their life will not be disturbed by misfortunes. In Stoic thought, Providence refers to the rational order of the universe, and aligning oneself with this order is key to tranquility.<\/pre>\n\n\n\n
    Seneca also warns against a life lived without purpose or design\u200a\u2014\u200alikening it to straws carried aimlessly by the current of a river. Instead, he advocates for a life lived with intention and reflection, considering not just the parts of life but its entirety.<\/pre>\n\n\n\n
    In summary, Seneca\u2019s conception of a happy life arises from a deep engagement with philosophy, the cultivation of virtue, living with intention, and maintaining a good conscience. Moreover, he stresses the importance of accepting one\u2019s role in the greater scheme of things (Providence) and understanding that a virtuous person cannot truly be unhappy, just as a wicked person cannot be truly happy. Living with wisdom and virtue ensures that one is not carried haphazardly through life but instead navigates it with purpose and moral clarity.<\/pre>\n\n\n\n
    Map Re-rank<\/h3>\n\n\n\n
    Tries to get an answer for each document and assigns it a confidence score, picking the highest confidence answer in the end.<\/p>\n\n\n\n
    \n<\/figure>\n\n\n\n
    \"\"\/
    Photo by Martin Sanchez<\/a> on\u00a0Unsplash<\/a><\/figcaption><\/figure>\n\n\n\n
    \u201cmap-rerank\u201d separates texts into batches, feeds each batch to LLM, returns a score of how fully it answers the question, and comes up with the final answer based on the high-scored answers from each batch.<\/p>\n\n\n\n
    rerank_results = get_chain_result(\"map_rerank\"<\/span>, llm, retriever, question)\nrerank_results['result'<\/span>]<\/span><\/pre>\n\n\n\n
    Happiness is founded upon wisdom and virtue. One must first know what to do and then live according to that knowledge. Philosophy and precept are helpful toward a happy life, as is the blessing of a good conscience. A good man can never be miserable, nor a wicked man happy. Happiness is also achieved by cheerfully submitting to Providence.<\/pre>\n\n\n\n
    Introduction to ragas<\/h3>\n\n\n\n
    The ragas library evaluates Retrieval-Augmented Generation (RAG) pipelines, particularly within the LangChain framework.<\/p>\n\n\n\n
    It utilizes Large Language Models (LLMs) to conduct evaluations across various metrics, each addressing a specific aspect of the RAG pipeline\u2019s performance:<\/p>\n\n\n\n
    1. Faithfulness: Assesses the factual accuracy of the generated answer about the provided context.
    \n2. Answer Relevancy: Measures the relevance of the generated answer to the posed question.
    \n3. Context Relevancy: Evaluate the signal-to-noise ratio in retrieved contexts.<\/p>\n\n\n\n
    Ragas uses different methods to leverage LLMs, effectively measuring these aspects while overcoming inherent biases. The library stands out for its ability to provide nuanced insights into the effectiveness of RAG pipelines. It is a valuable tool for developers and researchers working on enhancing the accuracy and relevance of AI-generated content.<\/p>\n\n\n\n
    In the code snippet below, the Ragas library evaluates different strategies in a RAG pipeline.<\/p>\n\n\n\n
    The code imports metrics such as faithfulness, answer relevancy, and context relevancy and then sets up evaluator chains for each metric using RagasEvaluatorChain<\/code>. These chains are stored in a dictionary named eval_chains<\/code>. The function evaluate_strategy<\/code> takes a strategy, and the evaluator chains as inputs and computes scores for each metric. It iterates over each evaluation chain, applies it to the given strategy, and stores the resulting scores in a dictionary, which it returns.<\/p>\n\n\n\n
    This allows for a comprehensive evaluation of a strategy across multiple relevant metrics.<\/p>\n\n\n\n
    from<\/span> ragas.metrics import<\/span> faithfulness, answer_relevancy, context_relevancy\nfrom<\/span> ragas.langchain import<\/span> RagasEvaluatorChain\n\neval_chains = {\n    m.name: RagasEvaluatorChain(metric=m)\n    for<\/span> m in<\/span> [faithfulness, answer_relevancy, context_relevancy]\n}\n\ndef<\/span> evaluate_strategy<\/span>(strategy, eval_chains<\/span>):\n    \"\"\"\n    Evaluate a given strategy using the provided evaluation chains.\n\n    Parameters:\n    - strategy (dict): The strategy to be evaluated.\n    - eval_chains (dict): A dictionary of evaluation chains.\n\n    Returns:\n    - dict: A dictionary containing scores for the given strategy.\n    \"\"\"<\/span>\n    scores = {}\n    for<\/span> name, eval_chain in<\/span> eval_chains.items():\n        score_name = f\"{name}<\/span>_score\"<\/span>\n        #Evaluate the strategy using the eval_chain<\/span>\n        evaluation_result = eval_chain(strategy)\n        #Retrieve the specific score from the evaluation result<\/span>\n        specific_score = evaluation_result[score_name]\n        #Store this score in the scores dictionary<\/span>\n        scores[score_name] = specific_score\n    return<\/span> scores\n\n# List of strategies<\/span>\nstrategies = [stuff_result, refine_results, rerank_results, map_reduce_result]\nstrategy_names = ['stuff_result'<\/span>, 'refine_results'<\/span>, 'rerank_results'<\/span>, 'map_reduce_result'<\/span>]\n\n# Collect scores for each strategy<\/span>\nresults = []\nfor<\/span> strategy, strategy_name in<\/span> zip<\/span>(strategies, strategy_names):\n    scores = evaluate_strategy(strategy, eval_chains)\n    scores['result_type'<\/span>] = strategy_name  # Add the strategy name as 'result_type'<\/span>\n    results.append(scores)<\/span><\/pre>\n\n\n\n
    The following code is a helper function to plot the resulting metrics:<\/p>\n\n\n\n
    def<\/span> plot_evaluation_score<\/span>(results, metric<\/span>):\n    \"\"\"\n    Plot evaluation scores for a specific metric from a list of dictionaries.\n\n    Parameters:\n    - results (list): A list of dictionaries containing evaluation scores.\n    - metric (str): The specific metric to plot.\n    \"\"\"<\/span>\n    # Extract scores for the given metric<\/span>\n    scores = [result[metric] for<\/span> result in<\/span> results]\n\n    # Plotting<\/span>\n    plt.figure(figsize=(10<\/span>, 6<\/span>))\n    plt.bar(range<\/span>(len<\/span>(results)), scores, tick_label=[r['result_type'<\/span>] for<\/span> r in<\/span> results])\n\n    # Customizing the chart<\/span>\n    plt.title(f'Evaluation Scores for {metric}<\/span>'<\/span>)\n    plt.ylabel('Score'<\/span>)\n    plt.xlabel('Strategy'<\/span>)\n    plt.xticks(rotation=45<\/span>)\n    plt.tight_layout()\n\n    # Display the chart<\/span>\n    plt.show()<\/span><\/pre>\n\n\n\n
    Now, it\u2019s time to assess the performance of each strategy.<\/p>\n\n\n\n
    Faithfulness Metric<\/h3>\n\n\n\n
    The faithfulness metric is integral to evaluating the reliability and accuracy of language model outputs in question-answering scenarios.<\/p>\n\n\n\n
    It ensures that the information provided by AI systems is contextually relevant and factually accurate, which is paramount in applications where decision-making relies on the model\u2019s outputs.<\/p>\n\n\n\n
    Definition and Purpose<\/h4>\n\n\n\n
    Faithfulness<\/strong> in the context of language models, especially in question-answering systems, measures how accurately and reliably the model\u2019s generated answer adheres to the given context or source material. Ensuring the model\u2019s outputs are relevant, factually correct and consistent with the provided information is crucial.<\/p>\n\n\n\n
    Calculation Process<\/h4>\n\n\n\n
      \n
    1. Statement Generation and Identification<\/strong>: The process begins using a Large Language Model (LLM) to analyze the generated answer. This step involves identifying or extracting key statements or assertions in the response. These statements are the crux of the answer and are what the model claims to be true in response to the given question.<\/li>\n\n\n\n
    2. Statement Verification<\/strong>: Another critical step involves verifying these extracted statements against the provided context or source material. This verification is usually done using another LLM or advanced NLP technique. The purpose is to check each statement for accuracy and alignment with the context. This step can involve cross-referencing the statements with facts or data in the context, ensuring that the model\u2019s responses are plausible and factually grounded.<\/li>\n<\/ol>\n\n\n\n
      Scoring Mechanism<\/h4>\n\n\n\n
      The faithfulness score is quantified on a scale from 0 to 1, where 0 indicates complete unfaithfulness (none of the statements align with the context), and 1 indicates total faithfulness (the context supports all statements).<\/p>\n\n\n\n
      The calculation involves counting the number of statements from the generated answer verified as accurate and consistent with the context and dividing this count by the total number of statements made. This ratio gives a precise, quantifiable measure of how faithful the answer is.<\/p>\n\n\n\n
      Application in ragas<\/code><\/h4>\n\n\n\n
      In the ragas<\/code> framework, as seen from the _faithfulness.py<\/a><\/code> source code, the Faithfulness<\/code> class incorporates these steps as part of its evaluation chain.<\/p>\n\n\n\n
      # Plots the 'faithfulness_score' metric<\/span>\nplot_evaluation_score(results, 'faithfulness_score'<\/span>) <\/span><\/pre>\n\n\n\n
      \n<\/figure>\n\n\n\n
      \"\"\/
      Image by author<\/figcaption><\/figure>\n\n\n\n
      The \u2018stuff_result\u2019 and \u2018rerank_results\u2019 strategies achieved perfect scores (1.00), indicating a high level of factual accuracy in the context provided.<\/p>\n\n\n\n
      However, the \u2018refine_results\u2019 strategy scored slightly lower, suggesting room for improvement in maintaining factual consistency.<\/p>\n\n\n\n
      Answer Relevancy<\/h3>\n\n\n\n
      The Answer Relevancy metric is critical in evaluating language models, particularly in question-answering systems.<\/p>\n\n\n\n
      Focusing on the semantic alignment between the question and the answer ensures that the generated responses are contextually relevant and specifically address the query. This metric is crucial for applications where the accuracy and relevance of LLM-generated responses are paramount.<\/p>\n\n\n\n
      Definition and Purpose<\/h4>\n\n\n\n
      Answer Relevancy<\/strong> is a metric to evaluate how well a generated answer corresponds to a given question or prompt. It assesses the relevance of the content of the answer in the context of the question, ensuring that the response is accurate and directly addresses the query.<\/p>\n\n\n\n
      Key Attributes<\/h4>\n\n\n\n