> ## Documentation Index
> Fetch the complete documentation index at: https://futureagi.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluating RAG Applications

### Retreival Augmented Generation Evaluation using Future AGI

**Step 1 - Install necessary packages and making necessary imports**

```python theme={null}
!pip install --ignore-installed blinker
!pip install futureagi datasets
```

```python theme={null}
import json
import requests
from fi.evals import Evaluator

from fi.evals import (
    ContextAdherence,
    ContextRetrieval,
    ContextSufficiency,
    RagasAnswerCorrectness,
    RagasCoherence,
    RagasHarmfulness
)
from fi.testcases import TestCase, LLMTestCase

from datasets import load_dataset
```

**Step 2 - Load the dataset and select an instance of the dataset**

```python theme={null}
# Load the dataset
dataset = load_dataset("explodinggradients/ragas-wikiqa")
sample_data = dataset["train"]
df = sample_data.to_pandas()
df = df.head(10)
df.head()
```

| question                                        | correct\_answer                                        | incorrect\_answer                                      | question\_id | generated\_with\_rag                               | context                                                | generated\_without\_rag                              |
| ----------------------------------------------- | ------------------------------------------------------ | ------------------------------------------------------ | ------------ | -------------------------------------------------- | ------------------------------------------------------ | ---------------------------------------------------- |
| HOW AFRICAN AMERICANS WERE IMMIGRATED TO THE US | As such, African immigrants are to be distinguished... | From the Immigration and Nationality Act of 19...      | Q0           | African Americans were immigrated to the United... | \[African immigration to the United States refers...   | African Americans were immigrated to the US in...    |
| what are points on a mortgage                   | Points, sometimes also called a "discount point"...    | Discount points may be different from originating...   | Q1012        | Points on a mortgage are a form of pre-paid...     | \[Discount points, also called mortgage points...      | A mortgage point is a fee equal to 1% of the l...    |
| how does interlibrary loan work                 | The user makes a request with their local library...   | Although books and journal articles are the most...    | Q102         | Interlibrary loan works by allowing patrons...     | \[Interlibrary loan (abbreviated ILL, and sometimes... | Interlibrary loan is a service that allows lib...    |
| WHAT IS A FY QUARTER                            | A fiscal year (or financial year, or sometimes...      | Fiscal years vary between businesses and countries...  | Q1027        | A FY quarter is a three-month period within...     | \[April.\n\n\n=== United States ===\n\n\n==== F...     | A FY Quarter is a three-month period in the fi...    |
| who wrote a rose is a rose is a rose            | The sentence "Rose is a rose is a rose is a rose"...   | I know that in daily life we don't go around saying... | Q1032        | Gertrude Stein wrote the sentence "A rose is...    | \[The sentence "Rose is a rose is a rose is a rose"... | Gertrude Stein wrote "A Rose is a Rose is a Rose..." |

**Step 3 - Choose the evaluations you want to perform**

### Available RAG evaluations in Future AGI :

#### Context Adherence

* **Description:** Ensures that responses remain within the provided context, avoiding information not present in the retrieved data.
* **Key Points:** Focuses on detecting hallucinations and ensuring factual consistency.

#### Context Relevance

* **Description:** Assesses how well the retrieved context aligns with the query.
* **Key Points:** Determines sufficiency of context to address the input.

#### Completeness

* **Description:** Evaluates whether the response fully answers the query.
* **Key Points:** Focuses on providing comprehensive and accurate answers.

#### Chunk Attribution

* **Description:** Tracks which context chunks are used in generating responses.
* **Key Points:** Highlights which parts of the context contribute to the response.

#### Chunk Utilization

* **Description:** Measures the effective usage of context chunks in generating responses.
* **Key Points:** Indicates the level of relevance and reliance on the provided context.

#### Context Similarity

* **Description:** Compares the provided context with expected context using similarity metrics.
* **Key Points:** Uses techniques like cosine similarity and Jaccard index for comparison.

#### Groundedness

* **Description:** Ensures that the response is strictly grounded in the provided context.
* **Key Points:** Verifies factual reliance on retrieved information.

#### Summarization Accuracy

* **Description:** Evaluates the accuracy of a summary against the original document.
* **Key Points:** Ensures faithfulness to the source material.

#### Eval Context Retrieval Quality

* **Description:** Assesses the quality and adequacy of the retrieved context.
* **Key Points:** Measures sufficiency and relevance of the retrieved information.

#### Eval Ranking

* **Description:** Provides ranking scores for contexts based on relevance and criteria.
* **Key Points:** Prioritizes contexts that best align with the query.

**Step 5 - Create an object of the chosen evaluator(s)**

```python theme={null}
# Create an object of the chosen evaluator(s)
#FutureAGI Metrics

context_adherence = ContextAdherence(config={"check_internet": False})
context_retrieval = ContextRetrieval(config={
    "check_internet": False,
    "criteria": "Is context retrieved align with the input"
})
context_sufficiency = ContextSufficiency(config={
    "check_internet": False,
    "model": "gpt-4o-mini"})

metrics = {
    "context_adherence": context_adherence,
    "context_retrieval": context_retrieval,
    "context_sufficiency": context_sufficiency,
}
```

**Step 6 - Initialize the Evaluator and run evaluations**

```python theme={null}
# Initialize the Evaluator
evaluator = Evaluator(fi_api_key="your_api_key", fi_secret_key="your_secret_key", fi_base_url="https://api.futureagi.com")

for column in metrics:
    df[column] = None

for index, datapoint in df.iterrows():
    datapoint = datapoint.to_dict()
    ragas_test_case = TestCase(
        context=datapoint['context'],
        query=datapoint['question'],
        input=datapoint['question'],
        output=datapoint['generated_with_rag']
    )
    for metric in metrics:
        results = evaluator.evaluate(metrics[metric], ragas_test_case)
        df.at[index, metric] = results.eval_results[0]
```

**Step 7 - Aggregate the results**

```python theme={null}
sum_context_adherence = 0
sum_context_retrieval = 0
sum_context_sufficiency = 0

for index, datapoint in df.iterrows():
    sum_context_adherence += datapoint['context_adherence'].metrics[0].value
    sum_context_retrieval += datapoint['context_retrieval'].metrics[0].value
    sum_context_sufficiency += datapoint['context_sufficiency'].metrics[0].value

print(f"Average Context Adherence: {sum_context_adherence/len(df)}")
print(f"Average Context Retrieval: {sum_context_retrieval/len(df)}")
print(f"Average Context Sufficiency: {sum_context_sufficiency/len(df)}")
```

```
Average Context Adherence: 0.9399999999999998
Average Context Retrieval: 0.9
Average Context Sufficiency: 1.0
```