Documentation Index
Fetch the complete documentation index at: https://futureagi.mintlify.app/llms.txt
Use this file to discover all available pages before exploring further.
result = evaluator.evaluate(
eval_templates="chunk_utilization",
inputs={
"context": [
"Paris is the capital and largest city of France.",
"France is a country in Western Europe.",
"Paris is known for its art museums and fashion districts."
],
"output": "According to the provided information, Paris is the capital city of France. It is a major European city and a global center for art, fashion, and culture.",
"input": "What is the capital of France?"
},
model_name="turing_flash"
)
print(result.eval_results[0].output)
print(result.eval_results[0].reason)
| Input | | | |
|---|
| Required Input | Type | Description |
| context | string or list[string] | The contextual information provided to the model |
| output | string | The response generated by the language model |
| Output | | |
|---|
| Field | Description |
| Score | Returns a numeric score, where higher values indicate more effective utilization of context |
| Reason | Provides a detailed explanation of the evaluation |
What to Do When Chunk Utilization Score is Low
- Ensure that the context provided is relevant and sufficiently detailed for the model to utilise effectively.
- Modify the input prompt to better guide the model in using the context. Clearer instructions may help the model understand how to incorporate the context into its response.
- If the model consistently fails to use context, it may require retraining or fine-tuning with more examples that emphasise the importance of context utilization.
Chunk Attribution assesses whether the model acknowledges and references the provided context at all, yielding a binary result: Pass if the context is used, or Fail if it is not. In contrast, Chunk Utilization evaluates how effectively the model incorporates that context into its response, producing a score that reflects the depth of its reliance on the information. While Attribution checks if the context was used, Utilization measures how well it was used to generate a meaningful and informed output.
