Skip to main content
result = evaluator.evaluate(
    eval_templates="chunk_utilization",
    inputs={
        "context": [
            "Paris is the capital and largest city of France.",
            "France is a country in Western Europe.",
            "Paris is known for its art museums and fashion districts."
        ],
        "output": "According to the provided information, Paris is the capital city of France. It is a major European city and a global center for art, fashion, and culture.",
        "input": "What is the capital of France?"
    },
    model_name="turing_flash"
)

print(result.eval_results[0].output)
print(result.eval_results[0].reason)

import { Evaluator, Templates } from "@future-agi/ai-evaluation";

const evaluator = new Evaluator();

const result = await evaluator.evaluate(
  "chunk_utilization",
  {
    context: [
      "Paris is the capital and largest city of France.",
      "France is a country in Western Europe.",
      "Paris is known for its art museums and fashion districts."
    ],
    output: "According to the provided information, Paris is the capital city of France. It is a major European city and a global center for art, fashion, and culture.",
    input: "What is the capital of France?"
  },
  {
    modelName: "turing_flash",
  }
);

console.log(result);
Input
Required InputTypeDescription
contextstring or list[string]The contextual information provided to the model
outputstringThe response generated by the language model
Output
FieldDescription
ScoreReturns a numeric score, where higher values indicate more effective utilization of context
ReasonProvides a detailed explanation of the evaluation

​
What to Do When Chunk Utilization Score is Low

  • Ensure that the context provided is relevant and sufficiently detailed for the model to utilise effectively.
  • Modify the input prompt to better guide the model in using the context. Clearer instructions may help the model understand how to incorporate the context into its response.
  • If the model consistently fails to use context, it may require retraining or fine-tuning with more examples that emphasise the importance of context utilization.

​
Differentiating Chunk Utilization with Chunk Attribution

Chunk Attribution assesses whether the model acknowledges and references the provided context at all, yielding a binary result: Pass if the context is used, or Fail if it is not. In contrast, Chunk Utilization evaluates how effectively the model incorporates that context into its response, producing a score that reflects the depth of its reliance on the information. While Attribution checks if the context was used, Utilization measures how well it was used to generate a meaningful and informed output.