> ## Documentation Index
> Fetch the complete documentation index at: https://futureagi.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Evals for Prototype

To configure evaluations for your prototype, define a list of EvalTag objects that specify which evals should be run against your model outputs.

<CodeGroup>
  ```python Python theme={null}
  eval_tags = [
      EvalTag(
          eval_name=EvalName.CONTEXT_ADHERENCE,
          type=EvalTagType.OBSERVATION_SPAN,
          value=EvalSpanKind.LLM,
          mapping={
              "output": "llm.output_messages.0.message.content",
              "context": "llm.input_messages.1.message.content"
          },
          custom_eval_name="context_check",
          model=ModelChoices.TURING_LARGE
      )
  ]
  ```

  ```typescript JS/TS theme={null}
  const evalTags = [
        EvalTag.create({
          type: EvalTagType.OBSERVATION_SPAN,
          value: EvalSpanKind.LLM,
          eval_name: EvalName.CHUNK_ATTRIBUTION,
          custom_eval_name: "Chunk_Attribution",
          mapping: {
            "context": "raw.input",
            "output": "raw.output"
          },
          model: ModelChoices.TURING_SMALL
        }),
      ]
  ```
</CodeGroup>

* `eval_name`: The evaluations to run on the  spans
* `type`: Specifies where to apply the evaluation
* `value`: Identifies the kind of span to evaluate
* `mapping`: Contains mapping of the required inputs of the eval [Learn more →](/future-agi/get-started/prototype/evals#understanding-the-mapping-attribute)
* `custom_eval_name`: Custom name to assign the eval tag
* `model`: Model name to be assigned especially incase of future-agi evals

### Adding Custom Evals

For **custom\_built** evals, the name of custom-eval should be entered as string.

<CodeGroup>
  ```python Python theme={null}
  eval_tags = [
      EvalTag(
          eval_name='custom_eval_name_entered',
          value=EvalSpanKind.LLM,
          type=EvalTagType.OBSERVATION_SPAN,
          mapping={
              'input' : 'input.value'
          },
          custom_eval_name="<custom_eval_name2>",
      ),
  ]
  ```

  ```typescript JS/TS theme={null}
  const evalTags = [
        EvalTag.create({
          type: EvalTagType.OBSERVATION_SPAN,
          value: EvalSpanKind.LLM,
          eval_name: "Custom_eval_name_entered",
          custom_eval_name: "Chunk_Attribution",
          mapping: {
            "context": "raw.input",
            "output": "raw.output"
          }
        }),
      ]
  ```
</CodeGroup>

### Understanding the Mapping Attribute

The `mapping` attribute is a crucial component that connects eval requirements with your data. Here's how it works:

1. **Each eval has some required keys**: Different evaluations require different inputs. For example, the Context Adherence eval requires both `context` and `output` keys.

2. **Spans contain attributes**: Your spans (like LLM spans, retriever spans, etc.) have attributes that store information as key-value pairs also known as span attributes.

3. **Mapping connects them**: The mapping object specifies which span attribute should be used for each required key.

For example, in this mapping:

```python theme={null}
mapping={
    "output": "llm.output_messages.0.message.content",
    "context": "llm.input_messages.1.message.content"
}
```

* The `output` key required by the eval will use data from this span attribute `llm.output_messages.0.message.content`
* The `context` input will use data from this span attribute `llm.input_messages.1.message.content`

This allows evaluations to be flexible and work with different data while maintaining consistent evaluation logic.

***

Below are the list of evals Future AGI provides and their corresponding mappings and configuration parameters.

### 1. Conversation Coherence

Assesses whether a dialogue maintains logical flow and contextual consistency throughout all exchanges. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/conversation-coherence)

**Mapping:**

* **messages** : Data that contains the complete conversation, represented as an array of user and assistant messages.

**Output:** Returns an output score. A higher score reflects a logically consistent and contextually relevant conversation. A lower score indicates issues like abrupt topic shifts, irrelevant responses, or loss of context.

***

### 2. Conversation Resolution

Checks if a conversation reaches a satisfactory conclusion that addresses the user's initial query or intent. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/conversation-resolution)

**Mapping:**

* **messages** : Data that contains the complete conversation, represented as an array of user and assistant messages.

**Output:** Returns an output score. A higher score indicates that the conversation was resolved effectively. A lower score points to incomplete, unclear, or unresolved conversations.

***

### 3. Content Moderation

Identifies and flags potentially harmful, unsafe, or prohibited content in text outputs. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/content-moderation)

**Mapping:**

* **text**: `string` - The text to be evaluated for content moderation

**Output:** Returns a score where higher values indicate safer content, lower values indicate potentially inappropriate content

***

### 4. Context Adherence

Checks if a response stays strictly within the bounds of provided context without introducing external information. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/context-adherence)

**Mapping:**

* **context**: `string` - The context provided to the AI system.
* **output**: `string` - The output generated by the AI system.

**Output:** Returns a score where a higher score indicates stronger adherence to the context.

***

### 5. Context Relevance

Verifies that content is meaningfully related to the provided context and addresses the query appropriately. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/context-relevance)

**Mapping:**

* **context**: `string` - The context provided to the AI system.
* **input**: `string` - The input to the AI system.

**Output:** Returns a score where higher values indicate more relevant context.

***

### 6. Completeness

Analyzes whether an output fully addresses all aspects of the input request or task. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/completeness)

**Mapping:**

* **input**: `string` - The input to the AI system.
* **output**: `string` - The output generated by the AI system.

**Output:** Returns a score where the higher values indicate more complete content

***

### 7. PII

Detects Personally Identifiable Information in the response. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/pii)

**Mapping:**

* **input**: `string` - The input to be evaluated for PII

**Output:** Returns a 'Passed' if the response does not contains any PII, else returns 'Failed'

***

### 8. Toxicity

Detects presence of personally identifiable information to protect user privacy and ensure compliance. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/toxicity)

**Mapping:**

* **input**: `string` - the input to be evaluated for toxicity

**Output:** Returns either "Passed" or "Failed", where "Passed" indicates non-toxic content, "Failed" indicates presence of harmful or aggressive language

***

### 9. Tone

Evaluates the emotional quality and overall sentiment expressed in the content. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/tone)

**Mapping:**

* **input**: `string` - The input to be evaluated for tone

**Output:** Returns tone labels such as "neutral", "joy", etc whatever tag that indicates the dominant emotional tone detected in the content

***

### 10. Sexist

Detects gender-biased or discriminatory language in the text. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/sexist)

**Mapping:**

* **input**: `string` - The input to be evaluated for sexism

**Output:** Returns either "Passed" or "Failed", where "Passed" indicates no sexist content detected, "Failed" indicates presence of gender bias or discriminatory language

***

### 11. Prompt Injection

Identifies attempts to manipulate the model through crafted inputs that try to override instructions. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/prompt-injection)

**Mapping:**

* **input**: `string` - The input to the AI system

**Output:** Returns a 'Passed' if the input is not a prompt injection, else returns 'Failed'

***

### 12. Prompt Instruction Adherence

Checks if outputs follow specific instructions provided in the prompt. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/instruction-adherence)

**Mapping:**

* **output**: `string` - The output generated by the AI system.

**Output:** Returns a score between 0 and 1. A high score reflects strong adherence, where all prompt requirements are met, tasks are fully addressed, specified formats and constraints are followed, and both explicit and implicit instructions are properly handled. Conversely, a low score indicates significant deviations from the prompt instructions.

***

### 13. Data Privacy Compliance

Ensures content adheres to data protection standards and privacy regulations. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/data-privacy)

**Mapping:**

* **input**: `string` - The input to be evaluated

**Output:** Returns a 'Passed' if the content adheres to data protection standards and privacy regulations, else returns 'Failed'

***

### 14. Is Json

Validates whether text output is properly formatted as valid JSON. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/is-json)

**Mapping:**

* **text**: `string` - The text to be evaluated

**Output:** Returns a 'Passed' if the text is valid JSON, else returns 'Failed'

***

### 15. One Line

Checks that the entire response is contained in a single line. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/length-evals)

**Mapping:**

* **text**: `string` - The text to be evaluated

**Output:** Returns a 'Passed' if the text is a single line, else returns 'Failed'

***

### 16. Contains Valid Link

Confirms if the response contains at least one valid URL. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/valid-links)

**Mapping:**

* **text**

***

### 17. No Valid Links

Checks that no valid links are present in the response. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/valid-links)

**Mapping:**

* **text**: `string` - The text to be evaluated

**Output:** Returns a 'Passed' if the text does not contain any valid hyperlinks, else returns 'Failed'

***

### 18. Is Email

Checks if the response is a properly formatted email address. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/is-email)

**Mapping:**

* **text**: `string` - The text to be evaluated

**Output:** Returns a 'Passed' if the text is a properly formatted email address, else returns 'Failed'

***

### 19. Summary Quality

Checks if a summary captures the main points accurately and succinctly. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/summary-quality)

**Mapping:**

* **input**: `string` - The input to the AI system
* **output**: `string` - The output generated by the AI system
* **context**: `string` - The context provided to the AI system

**Output:** Returns a score where a higher score indicates better summary quality.

***

### 20. Factual Accuracy

Determines if the response is factually correct based on the context provided. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/factual-accuracy)

**Mapping:**

* **input**: `string` - The input provided to the AI system
* **output**: `string` - The output generated by the AI system
* **context**: `string` - The context provided to the AI system

**Output:** Returns a score where a higher score indicates greater factual accuracy.

***

### 21. Translation Accuracy

Evaluates the accuracy of translated content. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/translation-accuracy)

**Mapping:**

* **input**: `string` - The input to the AI system
* **output**: `string` - The translated output generated by the AI system

**Output:** Returns a score where a higher score indicates greater translation accuracy.

***

### 22. Cultural Sensitivity

Assesses given text for inclusivity and cultural awareness. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/cultural-sensitivity)

**Mapping:**

* **input**: `string` - The input to be evaluated for cultural sensitivity

**Output:** Returns either "Passed" or "Failed", where "Passed" indicates culturally appropriate content, "Failed" indicates potential cultural insensitivity

***

### 23. Bias Detection

Detects presence of bias or unfairness in the text. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/bias-detection)

**Mapping:**

* **input**: `string` - The input to be evaluated for bias

**Output:** Returns either "Passed" or "Failed", where "Passed" indicates neutral content, "Failed" indicates presence of bias.

***

### 24. LLM Function Calling

Checks if the output properly uses a function/tool call with correct parameters. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/llm-function-calling)

**Mapping:**

* **input**: `string` - The input to the AI system
* **output**: `string` - The output generated by the AI system

**Output:** Returns a 'Passed' if the output properly uses a function/tool call with correct parameters, else returns 'Failed'

***

### 25. Groundedness

Evaluates if the response is grounded in the provided context. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/groundedness)

**Mapping:**

* **input**: `string` - The input to the AI system
* **output**: `string` - The output generated by the AI system

**Output:** Returns a 'Passed' if the output is grounded according to the input provided, else returns 'Failed'

***

### 26. Audio Transcription

Analyzes the transcription accuracy of the given audio and its transcription. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/audio-transcription)

**Mapping:**

* **input audio**: `string` - The URL of the audio to be evaluated
* **input transcription**: `string` - The output generated by the AI system

**Output:** Returns a score based on the criteria provided.

***

### 27. Audio Quality

Evaluates the quality of the given audio. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/audio-quality)

**Mapping:**

* **input audio**: `string` - The URL of the audio to be evaluated

**Output:** Returns a score based on the criteria provided.

***

### 28. Chunk Attribution

Tracks if the context chunk is used in generating the response. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/chunk-attribution)

**Mapping:**

* **input**: `string` - The input to the AI system
* **output**: `string` - The output generated by the AI system
* **context**: `string` - The context provided to the AI system

**Output:** Returns a Passed or Failed based on the input, output and context.

***

### 29. Chunk Utilization

Measures how effectively context chunks are used in responses. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/chunk-utilization)

**Mapping:**

* **input**: `string` - The input to the AI system
* **output**: `string` - The output generated by the AI system
* **context**: `string` - The context provided to the AI system

**Output:** Returns a score based on the criteria provided.

***

### 30. Eval Ranking

Provides ranking score for each context based on specified criteria. [Learn more →](/future-agi/get-started/evaluation/builtin-evals/eval-ranking)

**Mapping:**

* **input**: `string` - The input to the AI system
* **context**: `string` - The output generated by the AI system

**Output:** Returns a score based on the criteria provided.

***

### 31. No Racial Bias

Ensures that the output does not contain or imply racial bias, stereotypes, or preferential treatment.

**Mapping**

* **input**: `string` - The input provided

**Output:** Returns a Passed or Failed based on the code provided.

***

### 32. No Gender Bias

Checks that the response does not reinforce gender stereotypes or exhibit bias based on gender identity.

**Mapping**

* **input**: `string` - The input provided

**Output:** Returns a Passed or Failed based on the code provided.

***

### 33. No Age Bias

Evaluates if the content is free from stereotypes, discrimination, or assumptions based on age.

**Mapping**

* **input**: `string` - The input provided

**Output:** Returns a Passed or Failed based on the code provided.

***

### 34. No OpenAI Reference

Ensures that the model response does not mention being an OpenAI model or reference its training data or providers.

**Mapping**

* **input**: `string` - The input provided

**Output:** Returns a Passed or Failed based on the code provided.

***

### 35. No Appologies

Checks if the model unnecessarily apologizes, e.g., 'I'm sorry, but…'

**Mapping**

* **input**: `string` - The input provided

**Output:** Returns a Passed or Failed based on the code provided.

***

### 36. Is Polite

Ensures that the output maintains a respectful, kind, and non-aggressive tone.

**Mapping**

* **input**: `string` - The input provided

**Output:** Returns a Passed or Failed based on the code provided.

***

### 37. Is Concise

Measures whether the answer is brief and to the point, avoiding redundancy.

**Mapping:**

* **input**: `string` - The input to be evaluated for conciseness

**Output:** Returns 'Passed' if the text is appropriately concise, 'Failed' if it contains unnecessary verbosity.

***

### 38. Is Helpful

Evaluates whether the response answers the user's question effectively.

**Mapping:**

* **input**: `string` - The user's question
* **output**: `string` - The response to be evaluated

**Output:** Returns 'Passed' if the response effectively answers the question, 'Failed' if it fails to address the query adequately.

***

### 39. Is Code

Checks whether the output is valid code or contains expected code snippets.

**Mapping:**

* **input**: `string` - The input to be evaluated for code validity

**Output:** Returns 'Passed' if the output contains valid, well-structured code, 'Failed' if it contains invalid code or syntax errors.

***

### 40. Fuzzy Match

Compares model output with an expected answer using approximate (not exact) matching.

**Mapping:**

* **input**: `string` - The output to be evaluated
* **output**: `string` - The expected answer to compare against

**Output:** Returns 'Passed' if the output semantically matches the expected answer, 'Failed' if there are significant differences in meaning or intent.

***

### 41. Answer Refusal

Checks if the model correctly refuses to answer when prompted with harmful, sensitive, or restricted queries.

**Mapping:**

* **input**: `string` - The harmful/sensitive query
* **output**: `string` - The model's response

**Output:** Returns 'Passed' if the model correctly refuses inappropriate requests, 'Failed' if it provides harmful or sensitive information.

***

### 42. Detect Hallucination

Identifies if the model fabricated facts or added information that was not present in the input or reference.

**Mapping:**

* **input**: `string` - The original input/reference
* **output**: `string` - The response to be evaluated for hallucinations

**Output:** Returns 'Passed' if the output contains only information present in or logically derived from the input, 'Failed' if it contains fabricated content.

***

### 43. No Harmful Therapeutic Guidance

Ensures that the model does not provide potentially harmful psychological or therapeutic advice.

**Mapping:**

* **input**: `string` - The input to be evaluated for harmful therapeutic guidance

**Output:** Returns 'Passed' if the output appropriately avoids harmful therapeutic guidance, 'Failed' if it contains such content.

***

### 44. Clinically Inappropriate Tone

Evaluates whether the model's tone is unsuitable for clinical or mental health contexts.

**Mapping:**

* **input**: `string` - The input to be evaluated for clinical tone appropriateness

**Output:** Returns 'Passed' if the tone is clinically appropriate, 'Failed' if it contains inappropriate elements.

***

### 45. Is Harmful Advice

Detects whether the model gives advice that could be physically, emotionally, legally, or financially harmful.

**Mapping:**

* **input**: `string` - The input to be evaluated for harmful advice

**Output:** Returns 'Passed' if the output appropriately avoids harmful advice, 'Failed' if it contains such content.

***

### 46. Content Safety Violation

A broad check for content that violates safety or usage policies—this includes toxicity, hate speech, explicit content, violence, etc.

**Mapping:**

* **input**: `string` - The input to be evaluated for content moderation

**Output:** Returns 'Passed' if the content adheres to safety guidelines, 'Failed' if it contains safety violations.

***

### 47. Is Good Summary

Evaluates if a summary is clear, well-structured, and includes the most important points from the source material.

**Mapping:**

* **input**: `string` - The source material
* **output**: `string` - The summary to be evaluated

**Output:** Returns 'Passed' if the summary effectively captures the main points and is well-structured, 'Failed' if it lacks clarity or misses important information.

***

### 48. Is Factually Consistent

Checks if the generated output is factually consistent with the source/context (e.g., input text or documents).

**Mapping:**

* **input**: `string` - The source/context material
* **output**: `string` - The output to be evaluated for factual consistency

**Output:** Returns 'Passed' if the output is factually consistent with the source, 'Failed' if it contains factual inconsistencies.

***

### 49. Is Compliant

Ensures that the output adheres to legal, regulatory, or organizational policies (e.g., HIPAA, GDPR, company rules).

**Mapping:**

* **input**: `string` - The input to be evaluated for compliance

**Output:** Returns 'Passed' if the output adheres to all relevant policies, 'Failed' if it contains compliance violations.

***

### 50. Is Informal Tone

Detects whether the tone is informal or casual (e.g., use of slang, contractions, emoji).

**Mapping:**

* **input**: `string` - The input to be evaluated for tone formality

**Output:** Returns 'Passed' if the tone is informal, 'Failed' if it is formal, neutral, or lacks any informal indicators.

***

### 51. Evaluate Function Calling

Tests if the model correctly identifies when to trigger a tool/function and includes the right arguments in the function call.

**Mapping:**

* **input**: `string` - The user's request
* **output**: `string` - The function call to be evaluated

**Output:** Returns 'Passed' if the function calling is correct and appropriate, 'Failed' if there are errors in function selection or argument usage.

***

### 52. Task Completion

Measures whether the model fulfilled the user's request accurately and completely.

**Mapping:**

* **input**: `string` - The user's request
* **output**: `string` - The model's response to be evaluated

**Output:** Returns 'Passed' if the task is completed successfully and accurately, 'Failed' if the response is incomplete or inaccurate.

***

### 53. Caption Hallucination

Evaluates whether image captions or descriptions contain factual inaccuracies or hallucinated details that are not present in the instruction.

**Mapping:**

* **input**: `string` - The user's request
* **output**: `string` - The model's response to be evaluated

**Output:** Returns 'Passed' if the description accurately reflects the instruction without adding unverified details, 'Failed' if it contains hallucinated elements.

***

### 54. Bleu Score

Computes a bleu score between the expected gold answer and the model output.

**Mapping:**

* **reference**: `string` - The reference answer
* **hypothesis**: `string` - The model output

**Output:** Returns a score between 0 and 1. Higher values indicate greater lexical overlap.

***

### 55. Rouge Score

Computes a rouge score between the expected gold answer and the model output.

**Mapping:**

* **reference**: `string` - The reference answer
* **hypothesis**: `string` - The model output

**Output:** Returns a score between 0 and 1. Higher values indicate greater recall-oriented overlap.

***

### 56. Text to SQL

Evaluates if the model correctly converts natural language text into valid and accurate SQL queries.

**Mapping:**

* **input**: `string` - The input text to be evaluated
* **output**: `string` - The output to be evaluated

**Output:** Returns 'Passed' if the SQL query is correct and valid, 'Failed' if it is incorrect, invalid, or doesn't match the input requirements.

***

### 57. Recall Score

Calculates Recall = (# relevant retrieved) / (# relevant total)

**Mapping:**

* **reference**: `string` - The reference set
* **hypothesis**: `string` - The retrieved set

**Output:** Returns a recall score between 0 and 1.

***

### 58. Levenshtein Similarity

Measures the number of edits (insertions, deletions, or substitutions) to transform generated text to reference text. It is case-insensitive, punctuation-insensitive, and returns a normalized similarity.

**Mapping:**

* **response**: `string` - Model-generated output to be evaluated
* **expected\_text**: `string` - Reference string against which the output is compared

**Output:** Returns a normalized Levenshtein distance between 0 and 1. A score of 1.0 means perfect match.

***

### 59. Numeric Similarity

Extracts numeric values from generated text and computes the normalized difference from the reference number. Returns the normalized numeric similarity.

**Mapping:**

* **response**: `string` - Model-generated output to be evaluated
* **expected\_text**: `string` - Reference string against which the output is compared

**Output:** Returns a normalized numeric similarity score between 0 and 1.

***

### 60. Embedding Similarity

Measures the cosine semantic similarity between the generated text and the reference text.

**Mapping:**

* **response**: `string` - Model-generated output to be evaluated
* **expected\_text**: `string` - Reference string against which the output is compared

**Output:** Returns a score between 0 and 1 representing semantic similarity. Higher values indicate stronger similarity.

***

### 61. Semantic List Contains

Checks if the generated response semantically contains one or more phrases from a reference list.

**Mapping:**

* **response**: `string` - Model-generated output to be evaluated
* **expected\_text**: `string` or `List[string]` - Reference phrases or keywords

**Output:** Returns a score between 0 and 1. Closer to 1.0 if match criteria are more satisfied, or closer to 0.0 otherwise.

***

### 62. Is AI Generated Image

Evaluates if the given image is generated by AI or not.

**Mapping:**

* **input\_image**: `string` - The input image to be evaluated

**Output:** Returns a score indicating the likelihood that the image is AI-generated.