Evaluating RAG Applications - Future AGI Documentation

Retreival Augmented Generation Evaluation using Future AGI

Step 1 - Install necessary packages and making necessary imports

!pip install --ignore-installed blinker
!pip install futureagi datasets

import json
import requests
from fi.evals import Evaluator

from fi.evals import (
    ContextAdherence,
    ContextRetrieval,
    ContextSufficiency,
    RagasAnswerCorrectness,
    RagasCoherence,
    RagasHarmfulness
)
from fi.testcases import TestCase, LLMTestCase

from datasets import load_dataset

Step 2 - Load the dataset and select an instance of the dataset

# Load the dataset
dataset = load_dataset("explodinggradients/ragas-wikiqa")
sample_data = dataset["train"]
df = sample_data.to_pandas()
df = df.head(10)
df.head()

question	correct_answer	incorrect_answer	question_id	generated_with_rag	context	generated_without_rag
HOW AFRICAN AMERICANS WERE IMMIGRATED TO THE US	As such, African immigrants are to be distinguished…	From the Immigration and Nationality Act of 19…	Q0	African Americans were immigrated to the United…	[African immigration to the United States refers…	African Americans were immigrated to the US in…
what are points on a mortgage	Points, sometimes also called a “discount point”…	Discount points may be different from originating…	Q1012	Points on a mortgage are a form of pre-paid…	[Discount points, also called mortgage points…	A mortgage point is a fee equal to 1% of the l…
how does interlibrary loan work	The user makes a request with their local library…	Although books and journal articles are the most…	Q102	Interlibrary loan works by allowing patrons…	[Interlibrary loan (abbreviated ILL, and sometimes…	Interlibrary loan is a service that allows lib…
WHAT IS A FY QUARTER	A fiscal year (or financial year, or sometimes…	Fiscal years vary between businesses and countries…	Q1027	A FY quarter is a three-month period within…	[April.\n\n\n=== United States ===\n\n\n==== F…	A FY Quarter is a three-month period in the fi…
who wrote a rose is a rose is a rose	The sentence “Rose is a rose is a rose is a rose”…	I know that in daily life we don’t go around saying…	Q1032	Gertrude Stein wrote the sentence “A rose is…	[The sentence “Rose is a rose is a rose is a rose”…	Gertrude Stein wrote “A Rose is a Rose is a Rose…”

Step 3 - Choose the evaluations you want to perform

Available RAG evaluations in Future AGI :

Context Adherence

Description: Ensures that responses remain within the provided context, avoiding information not present in the retrieved data.
Key Points: Focuses on detecting hallucinations and ensuring factual consistency.

Context Relevance

Description: Assesses how well the retrieved context aligns with the query.
Key Points: Determines sufficiency of context to address the input.

Completeness

Description: Evaluates whether the response fully answers the query.
Key Points: Focuses on providing comprehensive and accurate answers.

Chunk Attribution

Description: Tracks which context chunks are used in generating responses.
Key Points: Highlights which parts of the context contribute to the response.

Chunk Utilization

Description: Measures the effective usage of context chunks in generating responses.
Key Points: Indicates the level of relevance and reliance on the provided context.

Context Similarity

Description: Compares the provided context with expected context using similarity metrics.
Key Points: Uses techniques like cosine similarity and Jaccard index for comparison.

Groundedness

Description: Ensures that the response is strictly grounded in the provided context.
Key Points: Verifies factual reliance on retrieved information.

Summarization Accuracy

Description: Evaluates the accuracy of a summary against the original document.
Key Points: Ensures faithfulness to the source material.

Eval Context Retrieval Quality

Description: Assesses the quality and adequacy of the retrieved context.
Key Points: Measures sufficiency and relevance of the retrieved information.

Eval Ranking

Description: Provides ranking scores for contexts based on relevance and criteria.
Key Points: Prioritizes contexts that best align with the query.

Step 5 - Create an object of the chosen evaluator(s)

# Create an object of the chosen evaluator(s)
#FutureAGI Metrics

context_adherence = ContextAdherence(config={"check_internet": False})
context_retrieval = ContextRetrieval(config={
    "check_internet": False,
    "criteria": "Is context retrieved align with the input"
})
context_sufficiency = ContextSufficiency(config={
    "check_internet": False,
    "model": "gpt-4o-mini"})

metrics = {
    "context_adherence": context_adherence,
    "context_retrieval": context_retrieval,
    "context_sufficiency": context_sufficiency,
}

Step 6 - Initialize the Evaluator and run evaluations

# Initialize the Evaluator
evaluator = Evaluator(fi_api_key="your_api_key", fi_secret_key="your_secret_key", fi_base_url="https://api.futureagi.com")

for column in metrics:
    df[column] = None

for index, datapoint in df.iterrows():
    datapoint = datapoint.to_dict()
    ragas_test_case = TestCase(
        context=datapoint['context'],
        query=datapoint['question'],
        input=datapoint['question'],
        output=datapoint['generated_with_rag']
    )
    for metric in metrics:
        results = evaluator.evaluate(metrics[metric], ragas_test_case)
        df.at[index, metric] = results.eval_results[0]

Step 7 - Aggregate the results

sum_context_adherence = 0
sum_context_retrieval = 0
sum_context_sufficiency = 0

for index, datapoint in df.iterrows():
    sum_context_adherence += datapoint['context_adherence'].metrics[0].value
    sum_context_retrieval += datapoint['context_retrieval'].metrics[0].value
    sum_context_sufficiency += datapoint['context_sufficiency'].metrics[0].value

print(f"Average Context Adherence: {sum_context_adherence/len(df)}")
print(f"Average Context Retrieval: {sum_context_retrieval/len(df)}")
print(f"Average Context Sufficiency: {sum_context_sufficiency/len(df)}")

Average Context Adherence: 0.9399999999999998
Average Context Retrieval: 0.9
Average Context Sufficiency: 1.0

​Retreival Augmented Generation Evaluation using Future AGI

​Available RAG evaluations in Future AGI :

​Context Adherence

​Context Relevance

​Completeness

​Chunk Attribution

​Chunk Utilization

​Context Similarity

​Groundedness

​Summarization Accuracy

​Eval Context Retrieval Quality

​Eval Ranking