Skip to main content

Documentation Index

Fetch the complete documentation index at: https://futureagi.mintlify.app/llms.txt

Use this file to discover all available pages before exploring further.

Following evals help in assessing whether the text aligns with specific requirements, such as containing necessary information, adhering to expected formats, or avoiding unwanted terms:

​
1. Contains

Definition: Evaluates whether the input text contains a specific keyword. This is useful for ensuring that essential terms are present in the text. Evaluation Using Interface Input:
  • Required Inputs:
    • text: The content column to search within.
  • Configuration Parameters:
    • keyword: String - The text to search for in the input text.
    • case_sensitive: Boolean (optional) - Whether the search should match case (defaults to False if omitted).
Output:
  • Score: Passed or Failed
Interpretation:
  • Passed: The specified keyword is present in the text.
  • Failed: The specified keyword is not present in the text.
Evaluation using Python SDK
Click here to learn how to setup evaluation using the Python SDK.
Input TypeParameterTypeDescriptionUI Component
Required InputstextstringThe content column to search within.Column Select
Configuration ParameterskeywordstringThe keyword to search for in the text.Text Input
case_sensitiveboolOptional: Whether the keyword search should be case-sensitive.Checkbox
from fi.evals import Evaluator
from fi.testcases import TestCase
from fi.evals.templates import Contains

evaluator = Evaluator(
    fi_api_key="your_api_key",
    fi_secret_key="your_secret_key",
    fi_base_url="<https://api.futureagi.com>"
)

contains_eval = Contains(config={
    "keywords": "Hello",
    "case_sensitive": True
    }
)

test_case = TestCase(
    text="Hello world! How are you?"
)

result = evaluator.evaluate(eval_templates=[contains_eval], inputs=[test_case], model_name="turing_flash")
contains_text = result.eval_results[0].metrics[0].value
What to Do When Contains Evaluation Fails: If the evaluation fails, consider revising the text to include the necessary keyword. Providing clearer instructions regarding required terms can help prevent this issue in future evaluations.

​
2. Contains Any

Definition: Checks if the input text contains any of the specified keywords. This evaluation is useful for scenarios where the presence of at least one keyword is required. Evaluation Using InterfaceInput:
  • Required Inputs:
    • text: The content column to search within.
  • Configuration Parameters:
    • keywords: List[String] - A list of possible strings to search for. (Enter as a comma-separated string in UI).
    • case_sensitive: Boolean (optional) - Whether the search should match case (defaults to False if omitted).
Output:
  • Score: Passed or Failed
Interpretation:
  • Passed: At least one of the specified keywords is present in the text.
  • Failed: None of the specified keywords are present in the text.
Evaluation using Python SDK
Click here to learn how to setup evaluation using the Python SDK.
Input TypeParameterTypeDescriptionUI Component
Required InputstextstringThe content column to search within.Column Select
Configuration Parameterskeywordslist[string]List of keywords to search for in the text. (Enter as a comma-separated string in UI).Text Input
case_sensitiveboolOptional: Whether the keyword search should be case-sensitive.Checkbox
from fi.evals import Evaluator
from fi.testcases import TestCase
from fi.evals.templates import ContainsAny

evaluator = Evaluator(
    fi_api_key="your_api_key",
    fi_secret_key="your_secret_key",
    fi_base_url="<https://api.futureagi.com>"
)

contains_eval = ContainsAny(config={
    "keywords": ["Hello", "world"],
    "case_sensitive": True
    }
)

test_case = TestCase(
    text="Hello world! How are you?"
)

result = evaluator.evaluate(eval_templates=[contains_eval], inputs=[test_case], model_name="turing_flash")
contains_text = result.eval_results[0].metrics[0].value  # 1.0 or 0.0
What to Do When Contains Any Evaluation Fails: If the evaluation fails, ensure that at least one of the required keywords is included in the text. Adjusting the content to meet this requirement can improve compliance in future evaluations.

​
3. Contains All

Definition: Verifies that the input text contains all specified keywords. This evaluation is critical for ensuring comprehensive coverage of necessary terms. Evaluation Using InterfaceInput:
  • Required Inputs:
    • text: The content column to search within.
  • Configuration Parameters:
    • keywords: List[String] - The list of keywords that must all be present. (Enter as a comma-separated string in UI).
    • case_sensitive: Boolean (optional) - Whether the search should match case (defaults to False if omitted).
Output:
  • Score: Passed or Failed
Interpretation:
  • Passed: All of the specified keywords are present in the text.
  • Failed: At least one of the specified keywords is missing from the text.
Evaluation using Python SDK
Click here to learn how to setup evaluation using the Python SDK.
Input TypeParameterTypeDescriptionUI Component
Required InputstextstringThe content column to search within.Column Select
Configuration Parameterskeywordslist[string]List of keywords that must all be present in the text. (Enter as a comma-separated string in UI).Text Input
case_sensitiveboolOptional: Whether the keyword search should be case-sensitive.Checkbox
from fi.evals import Evaluator
from fi.testcases import TestCase
from fi.evals.templates import ContainsAll

evaluator = Evaluator(
    fi_api_key="your_api_key",
    fi_secret_key="your_secret_key",
    fi_base_url="<https://api.futureagi.com>"
)

contains_all_eval = ContainsAll(config={
    "keywords": ["hello", "world"],
    "case_sensitive": False})

test_case = TestCase(
    text="Hello world! How are you?"
)

result = evaluator.evaluate(eval_templates=[contains_all_eval], inputs=[test_case], model_name="turing_flash")
contains_all = result.eval_results[0].metrics[0].value
What to Do When Contains All Evaluation Fails: If the evaluation fails, review the text to identify which keywords are missing. Revise the content to include all required keywords to meet the evaluation criteria.

​
4. Contains None

Definition: Verifies that the input text contains none of the specified terms. This evaluation is important for filtering out unwanted or prohibited content. Evaluation Using InterfaceInput:
  • Required Inputs:
    • text: The content column to search within.
  • Configuration Parameters:
    • keywords: List[String] - The list of keywords that should not be present. (Enter as a comma-separated string in UI).
    • case_sensitive: Boolean (optional) - Whether the search should match case (defaults to False if omitted).
Output:
  • Score: Passed or Failed
Interpretation:
  • Passed: None of the specified forbidden keywords are present in the text.
  • Failed: At least one of the specified forbidden keywords is present in the text.
Evaluation using Python SDK
Click here to learn how to setup evaluation using the Python SDK.
Input TypeParameterTypeDescriptionUI Component
Required InputstextstringThe content column to search within.Column Select
Configuration Parameterskeywordslist[string]List of keywords that should not be present in the text. (Enter as a comma-separated string in UI).Text Input
case_sensitiveboolOptional: Whether the keyword search should be case-sensitive.Checkbox
from fi.evals import Evaluator
from fi.testcases import TestCase
from fi.evals.templates import ContainsNone

evaluator = Evaluator(
    fi_api_key="your_api_key",
    fi_secret_key="your_secret_key",
    fi_base_url="<https://api.futureagi.com>"
)

contains_none_eval = ContainsNone(config={
    "keywords": ["hello", "world"],
    "case_sensitive": False})

test_case = TestCase(
    text="This is a good and clean text"
)

result = evaluator.evaluate(eval_templates=[contains_none_eval], inputs=[test_case], model_name="turing_flash")
contains_none = result.eval_results[0].metrics[0].value
What to Do When Contains None Evaluation Fails: If the evaluation fails, identify which unwanted terms are present in the text. Revise the content to remove these terms to ensure compliance with the evaluation criteria.

​
5. Starts With

Definition: Checks if the input text begins with a specific substring. This evaluation is useful for ensuring that text adheres to expected formats or structures. Evaluation Using InterfaceInput:
  • Required Inputs:
    • text: The content column to check.
  • Configuration Parameters:
    • substring: String - The required starting text (prefix).
    • case_sensitive: Boolean (optional) - Whether the comparison should match case (defaults to False if omitted).
Output:
  • Score: Passed or Failed
Interpretation:
  • Passed: The text begins with the specified substring.
  • Failed: The text does not begin with the specified substring.
Evaluation using Python SDK
Click here to learn how to setup evaluation using the Python SDK.
Input TypeParameterTypeDescriptionUI Component
Required InputstextstringThe content column to check.Column Select
Configuration ParameterssubstringstringThe substring to check for at the start of the text.Text Input
case_sensitiveboolOptional: Whether the comparison should be case-sensitive.Checkbox
from fi.evals import Evaluator
from fi.testcases import TestCase
from fi.evals.templates import StartsWith

evaluator = Evaluator(
    fi_api_key="your_api_key",
    fi_secret_key="your_secret_key",
    fi_base_url="<https://api.futureagi.com>"
)

starts_with_eval = StartsWith(config={
    "substring": "Dear",
    "case_sensitive": True})

test_case = TestCase(
    text="Dear Sir/Madam,"
)

result = evaluator.evaluate(eval_templates=[starts_with_eval], inputs=[test_case], model_name="turing_flash")
starts_with = result.eval_results[0].metrics[0].value  # 1.0 or 0.0
What to Do When Starts With Evaluation Fails: If the evaluation fails, consider revising the text to ensure it begins with the required substring. Providing clearer formatting guidelines can help prevent this issue in future evaluations.

​
6. Ends With

Definition: Checks if the input text ends with a specific substring. This evaluation is important for validating the conclusion of the text. Evaluation Using InterfaceInput:
  • Required Inputs:
    • text: The content column to check.
  • Configuration Parameters:
    • substring: String - The required ending text (suffix).
    • case_sensitive: Boolean (optional) - Whether the comparison should match case (defaults to False if omitted).
Output:
  • Score: Passed or Failed
Interpretation:
  • Passed: The text ends with the specified substring.
  • Failed: The text does not end with the specified substring.
Evaluation using Python SDK
Click here to learn how to setup evaluation using the Python SDK.
Input TypeParameterTypeDescriptionUI Component
Required InputstextstringThe content column to check.Column Select
Configuration ParameterssubstringstringThe substring to check for at the end of the text.Text Input
case_sensitiveboolOptional: Whether the comparison should be case-sensitive.Checkbox
from fi.evals import Evaluator
from fi.testcases import TestCase
from fi.evals.templates import EndsWith

evaluator = Evaluator(
    fi_api_key="your_api_key",
    fi_secret_key="your_secret_key",
    fi_base_url="<https://api.futureagi.com>"
)

starts_with_eval = EndsWith(config={
    "substring": "you",
    "case_sensitive": True})

test_case = TestCase(
    text="thank you"
)

result = evaluator.evaluate(eval_templates=[starts_with_eval], inputs=[test_case], model_name="turing_flash")
ends_with = result.eval_results[0].metrics[0].value
What to Do When Ends With Evaluation Fails: If the evaluation fails, revise the text to ensure it concludes with the required substring. Clear guidelines on expected endings can help improve compliance in future evaluations.

​
7. Equals

Definition: Compares if the input text is exactly equal to a specified expected text. This evaluation is crucial for scenarios where precise matching is required. Evaluation Using Interface Input:
  • Required Inputs:
    • text: The content column to check.
    • expected_text: The column containing the exact string to match against.
  • Configuration Parameters:
    • case_sensitive: Boolean (optional) - Whether the comparison should match case (defaults to False if omitted).
Output:
  • Score: Passed or Failed
Interpretation:
  • Passed: The text is identical to the expected_text (considering case sensitivity).
  • Failed: The text differs from the expected_text (considering case sensitivity).
Evaluation using Python SDK
Click here to learn how to setup evaluation using the Python SDK.
Input TypeParameterTypeDescriptionUI Component
Required InputstextstringThe content column to check.Column Select
expected_textstringThe column containing the exact string to match against.Column Select
Configuration Parameterscase_sensitiveboolOptional: Whether the comparison should be case-sensitive.Checkbox
from fi.evals import Evaluator
from fi.testcases import TestCase
from fi.evals.templates import Equals

evaluator = Evaluator(
    fi_api_key="your_api_key",
    fi_secret_key="your_secret_key",
    fi_base_url="<https://api.futureagi.com>"
)

equals_eval = Equals(config={"case_sensitive": False})

test_case = TestCase(
    text="Hello, World!",
    expected_text="Hello"
)

result = evaluator.evaluate(eval_templates=[equals_eval], inputs=[test_case], model_name="turing_flash")
is_equal = result.eval_results[0].metrics[0].value
What to Do When Equals Evaluation Fails: If the evaluation fails, review the text for discrepancies. Adjusting the content to match the expected text precisely can help meet the evaluation criteria.