MRR (Mean Reciprocal Rank)

import json
from fi.evals import Evaluator

evaluator = Evaluator()

result = evaluator.evaluate(
    eval_templates="mrr",
    inputs={
        "hypothesis": json.dumps([
            "France is in Europe.",
            "Napoleon was born in Corsica.",
            "Paris is the capital of France.",
            "The Eiffel Tower was built in 1889.",
            "The Louvre is in Paris."
        ]),
        "reference": json.dumps([
            "Paris is the capital of France.",
            "The Eiffel Tower was built in 1889.",
            "The Louvre is in Paris."
        ])
    }
)

print(result.eval_results[0].output)   # 0.333
print(result.eval_results[0].reason)

import { Evaluator } from "@future-agi/ai-evaluation";

const evaluator = new Evaluator();

const result = await evaluator.evaluate(
  "mrr",
  {
    hypothesis: JSON.stringify([
      "France is in Europe.",
      "Napoleon was born in Corsica.",
      "Paris is the capital of France.",
      "The Eiffel Tower was built in 1889.",
      "The Louvre is in Paris."
    ]),
    reference: JSON.stringify([
      "Paris is the capital of France.",
      "The Eiffel Tower was built in 1889.",
      "The Louvre is in Paris."
    ])
  }
);

console.log(result.eval_results[0]?.output);   // 0.333
console.log(result.eval_results[0]?.reason);

In this example, the first relevant chunk (“Paris is the capital of France.”) appears at position 3, so the reciprocal rank is 1/3 = 0.333.


Required Input	Type	Description
`hypothesis`	`string`	JSON-serialized list of retrieved chunks in ranked order
`reference`	`string`	JSON-serialized list of ground-truth relevant chunks

Output
	Field	Description
	Result	Returns a score between 0 and 1, where 1 means the first relevant chunk is at position 1
	Reason	Short summary string of the score, e.g. `MRR: 0.333`

MRR does not take a k parameter. It scans the entire retrieved list to find the first relevant item.

Batch evaluation

To evaluate multiple queries in a single call, pass a list of JSON-serialized inputs. Each element represents one retrieval evaluation:

Python

results = evaluator.evaluate(
    eval_templates="mrr",
    inputs={
        "hypothesis": [
            json.dumps(["Paris is the capital of France.", "France is in Europe.", "Napoleon was born in Corsica."]),
            json.dumps(["The sky is blue.", "Water is wet."]),
            json.dumps(["Unrelated 1.", "Unrelated 2.", "Unrelated 3.", "The Louvre is in Paris."]),
        ],
        "reference": [
            json.dumps(["Paris is the capital of France.", "The Eiffel Tower was built in 1889."]),
            json.dumps(["The sky is blue.", "Water is wet."]),
            json.dumps(["The Louvre is in Paris."]),
        ],
    },
)

for i, r in enumerate(results.eval_results):
    print(f"Query {i+1}: {r.output}")
# Query 1: 1.0    (first relevant at position 1)
# Query 2: 1.0    (first relevant at position 1)
# Query 3: 0.25   (first relevant at position 4)

How it works

MRR (Mean Reciprocal Rank) measures how quickly the retriever surfaces the first relevant result. The score is the reciprocal of the rank position where the first relevant chunk appears. Formula:

MRR = 1 / (position of the first relevant item)

If the first relevant chunk is at position 1, the score is 1.0. At position 2, it’s 0.5. At position 3, it’s 0.333. If no relevant chunk is found, the score is 0.0. MRR is particularly useful for question-answering RAG systems where the first relevant chunk often contains the answer. It directly measures the user experience of finding information quickly. Matching is based on exact string equality between retrieved chunks and ground-truth chunks.

What to do when MRR is Low

If MRR is low, the first relevant chunk is appearing too far down in results:

Apply a re-ranking step to push the most relevant chunk to the top position
Check if irrelevant but semantically similar chunks are outranking the correct answer
Ensure query formatting matches the style of your indexed documents
For short queries, consider query expansion to add context that helps the retriever identify the best match
Verify that the first relevant chunk in your ground truth is actually the most directly relevant one

Differentiating MRR with Similar Evals

NDCG@K: NDCG@K evaluates the ranking quality across all relevant chunks, while MRR only looks at where the first relevant chunk appears.
Hit Rate: Hit Rate is binary (was any relevant chunk retrieved?), while MRR measures exactly how high the first relevant chunk ranks.
Recall@K: Recall@K measures how many relevant chunks were found regardless of position, while MRR focuses solely on the position of the first relevant chunk.

Get Started

Guides

Batch evaluation

How it works

What to do when MRR is Low

Differentiating MRR with Similar Evals

​Batch evaluation

​How it works

​What to do when MRR is Low

​Differentiating MRR with Similar Evals

Batch evaluation

How it works

What to do when MRR is Low

Differentiating MRR with Similar Evals