Recall Score (Deprecated)

result = evaluator.evaluate(
    eval_templates="recall_score",
    inputs={
        "reference": "The Eiffel Tower is a famous landmark in Paris, built in 1889 for the World's Fair. It stands 324 meters tall.",
        "hypothesis": "The Eiffel Tower, located in Paris, was built in 1889 and is 324 meters high."
    },
    model_name="turing_flash"
)

print(result.eval_results[0].output)
print(result.eval_results[0].reason)

import { Evaluator, Templates } from "@future-agi/ai-evaluation";

const evaluator = new Evaluator();

const result = await evaluator.evaluate(
  "recall_score",
  {
    reference: "The Eiffel Tower is a famous landmark in Paris, built in 1889 for the World's Fair. It stands 324 meters tall.",
    hypothesis: "The Eiffel Tower, located in Paris, was built in 1889 and is 324 meters high."
  },
  {
    modelName: "turing_flash",
  }
);

console.log(result);

​

Overview

​

What to do If you get Undesired Results

• Ensure that all key facts, entities, and relationships from the reference are included in the hypothesis
• Check for missing details, numbers, dates, or proper nouns that might be important
• Verify that important contextual information isn’t omitted
• Consider that paraphrasing may preserve recall as long as the core information is included
• For summaries, focus on including the most critical information from the reference
• Be aware that recall doesn’t penalize for additional information in the hypothesis (that’s measured by precision)

​

Comparing Recall Score with Similar Evals

• ROUGE Score: While Recall Score focuses on information coverage, ROUGE Score uses n-gram overlap to evaluate text similarity.
• BLEU Score: Recall Score measures how much reference information is captured, while BLEU Score emphasizes precision by measuring how much of the hypothesis matches the reference.
• Completeness: Recall Score measures information coverage from a reference text, whereas Completeness evaluates whether a response fully answers a given query.