> ## Documentation Index
> Fetch the complete documentation index at: https://futureagi.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluator (Cloud API)

> Using the Future AGI Python SDK Evaluator class for cloud-based evaluations with Turing models.

<Warning>
  **New API Available**: The [`evaluate()` function](/sdk-reference/evaluate) is the recommended way to run evaluations. It supports local metrics, LLM-as-Judge, multimodal (images/audio), augmentation, and auto-generated criteria — all through a single function call.

  The `Evaluator` class below is for cloud-only evaluations using Future AGI's Turing models. For new projects, start with [`evaluate()`](/sdk-reference/evaluate).
</Warning>

The Future AGI Python SDK provides an `Evaluator` class to programmatically run evaluations on your data and language model outputs. This document details its usage based on the provided SDK snippets.

## `Evaluator`

```
class Evaluator(APIKeyAuth):

```

An evaluator is an abstraction used for running evaluations on your data and model outputs.

### Initialization

Initializes the `Evaluator` client. API keys and base URL can be provided directly or will be read from environment variables (`FI_API_KEY`, `FI_SECRET_KEY`, `FI_BASE_URL`) if not specified.

```python theme={null}
def __init__(
        self,
        fi_api_key: Optional[str] = None,
        fi_secret_key: Optional[str] = None,
        fi_base_url: Optional[str] = None,
        **kwargs,
    ) -> None:
```

**Arguments:**

* `fi_api_key` (Optional\[str], optional): API key. Defaults to None.
* `fi_secret_key` (Optional\[str], optional): Secret key. Defaults to None.
* `fi_base_url` (Optional\[str], optional): Base URL. Defaults to None.
* `**kwargs`:
  * `timeout` (Optional\[int]): Timeout value in seconds. Default: `200`.
  * `max_queue_bound` (Optional\[int]): Maximum queue size. Default: `5000`.
  * `max_workers` (Optional\[int]): Maximum number of workers. Default: `8`.

***

### `evaluate`

Runs a single evaluation or a batch of evaluations independently.

```python theme={null}
def evaluate(
        self,
        eval_templates: Union[str, type[EvalTemplate]],
        inputs: Union[
            TestCase,
            List[TestCase],
            Dict[str, Any],
            List[Dict[str, Any]],
        ],
        timeout: Optional[int] = None,
        model_name: Optional[str] = None
    ) -> BatchRunResult:
```

**Arguments:**

* `eval_templates` (Union\[str, EvalTemplate, List\[EvalTemplate]]): A single evaluation template or a list of evaluation templates.
* `inputs` (Union\[TestCase, List\[TestCase], Dict\[str, Any], List\[Dict\[str, Any]]): A single test case or a list of test cases. Supports various `TestCase` types.
* `timeout` (Optional\[int], optional): Timeout value in seconds for the evaluation. Defaults to None (uses the client's default timeout).
* `model_name` (Optional\[str], optional): Model name to use for the evaluation while using Future AGI Built Evals. Defaults to None.

<Note>
  When running Future AGI Built Evals, you have to specify the model name to use for the evaluation, otherwise the SDK will throw an error.
</Note>

**Returns:**

* `BatchRunResult`: An object containing the results of the evaluation(s).

**Raises:**

* `ValidationError`: If the inputs do not match the evaluation templates.
* `Exception`: If the API request fails or other errors occur during evaluation.

***

### `list_evaluations`

Fetches information about all available evaluation templates.

```python theme={null}
def list_evaluations(self) -> List[Dict[str, Any]]:
```

**Returns:**

* `List[Dict[str, Any]]`: A list of dictionaries, where each dictionary contains information about an available evaluation template. This typically includes details like the template's `id`, `name`, `description`, and expected parameters.

### `eval_templates`

The list of templates that can be used to evaluate your data.

<Warning>
  We are deprecating the following templates: `content_moderation`, `factual_accuracy`, `content_safety_violation`, `is_factually_consistent`, and `is_compliant`.

  For new integrations, please use:

  * `toxicity` instead of `content_moderation` and `content_safety_violation`
  * `groundedness` instead of `factual_accuracy` and `is_factually_consistent`
  * `data_privacy_compliance` instead of `is_compliant`
</Warning>

#### **Conversation Coherence**

Evaluates if a conversation flows logically and maintains context throughout

```python theme={null}
class ConversationCoherence():
```

#### **Conversation Resolution**

Checks if the conversation reaches a satisfactory conclusion or resolution. The conversation must have atleast two users

```python theme={null}
class ConversationResolution():
```

#### **Content Moderation (Deprecated)**

Uses OpenAI's content moderation to evaluate text safety

```python theme={null}
class ContentModeration():
```

#### **Context Adherence**

Measures how well responses stay within the provided context

```python theme={null}
class ContextAdherence():
```

#### **Context Relevance**

Evaluates the relevancy of the context to the query

```python theme={null}
class ContextRelevance():
```

#### **Completeness**

Evaluates if the response completely answers the query

```python theme={null}
class Completeness():
```

#### **Chunk Attribution**

Tracks if the context chunk is used in generating the response.

```python theme={null}
class ChunkAttribution():
```

#### **Chunk Utilization**

Measures how effectively context chunks are used in responses

```python theme={null}
class ChunkUtilization():
```

#### **PII**

Detects personally identifiable information (PII) in text.

```python theme={null}
class PII():
```

#### **Toxicity**

Evaluates content for toxic or harmful language

```python theme={null}
class Toxicity():
```

#### **Tone**

Analyzes the tone and sentiment of content

```python theme={null}
class Tone():
```

#### **Sexist**

Detects sexist content and gender bias

```python theme={null}
class Sexist():
```

#### **Prompt Injection**

Evaluates text for potential prompt injection attempts

```python theme={null}
class PromptInjection():
```

#### **Not Gibberish Text**

Checks if the text is not gibberish

```python theme={null}
class NotGibberish():
```

#### **Safe for Work text**

Evaluates if the text is safe for work.

```python theme={null}
class SafeForWork():
```

#### **Prompt Instruction Adherence**

Assesses how closely the output follows the given prompt instructions, checking for completion of all requested tasks and adherence to specified constraints or formats. Evaluates both explicit and implicit requirements in the prompt.

```python theme={null}
class PromptAdherence():
```

#### **Data Privacy Compliance**

Checks output for compliance with data privacy regulations (GDPR, HIPAA, etc.). Identifies potential privacy violations, sensitive data exposure, and adherence to privacy principles.

```python theme={null}
class DataPrivacyCompliance():
```

#### **Is Json**

Validates if content is proper JSON format

```python theme={null}
class IsJson():
```

#### **One Line**

Checks if the text is a single line

```python theme={null}
class OneLine():
```

#### **Contains Valid Link**

Checks for presence of valid URLs

```python theme={null}
class ContainsValidLink():
```

#### **Is Email**

Validates email address format

```python theme={null}
class IsEmail():
```

#### **No Valid Links**

Checks if the text contains no invalid URLs

```python theme={null}
class NoValidLinks():
```

#### **Eval Ranking**

Provides ranking score for each context based on specified criteria.

```python theme={null}
class EvalRanking():
```

#### **Summary Quality**

Evaluates if a summary effectively captures the main points, maintains factual accuracy, and achieves appropriate length while preserving the original meaning. Checks for both inclusion of key information and exclusion of unnecessary details.

```python theme={null}
class SummaryQuality(config={
    "check_internet": {"type": "boolean", "default": False}
}):
```

#### **Factual Accuracy (Deprecated)**

Verifies if the provided output is factually correct or not.

```python theme={null}
class FactualAccuracy(config={
    "check_internet": {"type": "boolean", "default": False}
}):
```

#### **Translation Accuracy**

Evaluates the quality of translation by checking semantic accuracy, cultural appropriateness, and preservation of original meaning. Considers both literal accuracy and natural expression in the target language.

```python theme={null}
class TranslationAccuracy():
```

#### **Cultural Sensitivity**

Analyzes output for cultural appropriateness, inclusive language, and awareness of cultural nuances. Identifies potential cultural biases or insensitive content.

```python theme={null}
class CulturalSensitivity():
```

#### **Bias Detection**

Identifies various forms of bias including gender, racial, cultural, or ideological bias in the output. Evaluates for balanced perspective and neutral language use.

```python theme={null}
class BiasDetection():
```

#### **Evaluate LLM Function calling**

Assesses accuracy and effectiveness of LLM function calls.

```python theme={null}
class EvaluateLLMFunctionCalling():
```

#### **Audio Transcription**

Analyzes the transcription accuracy of the given audio and its transcription.

```python theme={null}
class AudioTranscription():
```

#### **Audio Quality**

Evaluates the quality of the given audio.

```python theme={null}
class AudioQuality():
```

#### **Protect Flash**

FutureAGI's proprietary evaluator to check if the content is harmful

```python theme={null}
class ProtectFlash():
```

#### **No Racial Bias**

Ensures that the output does not contain or imply racial bias, stereotypes, or preferential treatment.

```python theme={null}
class NoRacialBias():
```

#### **No Gender Bias**

Checks that the response does not reinforce gender stereotypes or exhibit bias based on gender identity.

```python theme={null}
class NoGenderBias():
```

#### **No Age Bias**

Evaluates if the content is free from stereotypes, discrimination, or assumptions based on age.

```python theme={null}
class NoAgeBias():
```

#### **No Openai Reference**

Ensures that the model response does not mention being an OpenAI model or reference its training data or providers.

```python theme={null}
class NoOpenaiReference():
```

#### **No Apologies**

Checks if the model unnecessarily apologizes, e.g., 'I'm sorry, but…'

```python theme={null}
class NoApologies():
```

#### **Is Polite**

Ensures that the output maintains a respectful, kind, and non-aggressive tone.

```python theme={null}
class IsPolite():
```

#### **Is Concise**

Measures whether the answer is brief and to the point, avoiding redundancy.

```python theme={null}
class IsConcise():
```

#### **Is Helpful**

Evaluates whether the response answers the user's question effectively.

```python theme={null}
class IsHelpful():
```

#### **Is Code**

Checks whether the output is valid code or contains expected code snippets.

```python theme={null}
class IsCode():
```

#### **Fuzzy Match**

Compares model output with an expected answer using approximate (not exact) matching.

```python theme={null}
class FuzzyMatch():
```

#### **Answer Refusal**

Checks if the model correctly refuses to answer when prompted with harmful, sensitive, or restricted queries.

```python theme={null}
class AnswerRefusal():
```

#### **Detect Hallucination**

Identifies if the model fabricated facts or added information that was not present in the input or reference.

```python theme={null}
class DetectHallucination():
```

#### **No Harmful Therapeutic Guidance**

Ensures that the model does not provide potentially harmful psychological or therapeutic advice.

```python theme={null}
class NoHarmfulTherapeuticGuidance():
```

#### **Clinically Inappropriate Tone**

Evaluates whether the model's tone is unsuitable for clinical or mental health contexts.

```python theme={null}
class ClinicallyInappropriateTone():
```

#### **Is Harmful Advice**

Detects whether the model gives advice that could be physically, emotionally, legally, or financially harmful.

```python theme={null}
class IsHarmfulAdvice():
```

#### **Content Safety Violation (Deprecated)**

A broad check for content that violates safety or usage policies—this includes toxicity, hate speech, explicit content, violence, etc.

```python theme={null}
class ContentSafetyViolation():
```

#### **Is Good Summary**

Evaluates if a summary is clear, well-structured, and includes the most important points from the source material.

```python theme={null}
class IsGoodSummary():
```

#### **Is Factually Consistent (Deprecated)**

Checks if the generated output is factually consistent with the source/context (e.g., input text or documents).

```python theme={null}
class IsFactuallyConsistent():
```

#### **Is Compliant (Deprecated)**

Ensures that the output adheres to legal, regulatory, or organizational policies (e.g., HIPAA, GDPR, company rules).

```python theme={null}
class IsCompliant():
```

#### **Is Informal Tone**

Detects whether the tone is informal or casual (e.g., use of slang, contractions, emoji).

```python theme={null}
class IsInformalTone():
```

#### **Evaluate Function Calling**

Tests if the model correctly identifies when to trigger a tool/function and includes the right arguments in the function call.

```python theme={null}
class EvaluateFunctionCalling():
```

#### **Task Completion**

Measures whether the model fulfilled the user's request accurately and completely.

```python theme={null}
class TaskCompletion():
```

#### **Caption Hallucination**

Evaluates whether image captions or descriptions contain factual inaccuracies or hallucinated details that are not present in the instruction. This metric helps ensure that AI-generated image descriptions remain faithful to the instruction content.

```python theme={null}
class CaptionHallucination():
```

#### **Bleu Score**

Computes a bleu score between the expected gold answer and the model output.

```python theme={null}
class BleuScore():
```

#### Aggregated Metric

Combines multiple evaluation metrics into a single normalised score.

```python theme={null}
class AggregatedMetric(config={
    "metrics": {"type": "list", "default": []},
    "metric_names": {"type": "list", "default": []},
    "aggregator": {"type": "option", "default": "average"},
    "weights": {"type": "list", "default": None},
}):
```

#### ROUGE Score

Calculate ROUGE score between generated text and reference text

```python theme={null}
class ROUGEScore(config={
    "rouge_type": {"type": "option", "default": "rouge1", "options": ["rouge1", "rouge2", "rougeL"]},
    "use_stemmer": {"type": "boolean", "default": True}
}):
```

#### Numerical Difference

Calculate numerical difference between generated value and reference value

```python theme={null}
class NumericDiff(config={
    "extract_numeric": {"type": "boolean", "default": True},
    "normalized_result": {"type": "boolean", "default": True}
}):
```

#### Levenshtein Distance

Calculate edit distance between generated text and reference text

```python theme={null}
class LevenshteinDistance(config={
    "case_insensitive": {"type": "boolean", "default": False},
    "remove_punctuation": {"type": "boolean", "default": False}
}):
```

#### Embedding Similarity

Calculate semantic similarity between generated text and reference text

```python theme={null}
class EmbeddingSimilarity(config={
    "similarity_method": {"type": "option", "default": "cosine", "options": ["cosine", "euclidean", "manhattan"]},
    "normalize": {"type": "boolean", "default": True}
}):
```

#### Semantic List Contains

Check if text contains phrases semantically similar to reference phrases

```python theme={null}
class SemanticListContains(config={
    "case_insensitive": {"type": "boolean", "default": True},
    "remove_punctuation": {"type": "boolean", "default": True},
    "match_all": {"type": "boolean", "default": False},
    "similarity_threshold": {"type": "float", "default": 0.7}
}):
```

***

### Example Usage

```python theme={null}
from fi.evals import Evaluator, Tone
from fi.testcases import TestCase

evaluator = Evaluator()

test_case = TestCase(
    input="Write a professional email",
    output="Dear Sir/Madam, I hope this email finds you well. I am writing to inquire about...",
    context="Maintain formal business communication tone",
)

template = Tone()

response = evaluator.evaluate(eval_templates=[template], inputs=[test_case], model_name="turing_flash")
```