Test Case Classes

The Test Case classes define the structure for test cases used in evaluations, including support for text, conversational, LLM, multimodal (image/audio), and more.

`TestCase` Class

Represents a general test case for evaluation.

class TestCase(BaseModel):
    text: Optional[str] = None
    document: Optional[str] = None
    input: Optional[str] = None
    output: Optional[str] = None
    prompt: Optional[str] = None
    criteria: Optional[str] = None
    actual_json: Optional[dict] = None
    expected_json: Optional[dict] = None
    expected_text: Optional[str] = None
    query: Optional[str] = None
    response: Optional[str] = None
    context: Union[List[str], str] = None

Attributes:

text (Optional[str]): Text for the test case.
document (Optional[str]): Document content.
input (Optional[str]): Input string.
output (Optional[str]): Output string.
prompt (Optional[str]): Prompt used.
criteria (Optional[str]): Evaluation criteria.
actual_json (Optional[dict]): Actual JSON object.
expected_json (Optional[dict]): Expected JSON object.
expected_text (Optional[str]): Expected text output.
query (Optional[str]): Query string.
response (Optional[str]): Response string.
context (Union[List[str], str]): Context for the test case.

`ConversationalTestCase` Class

Represents a conversational test case, consisting of a list of LLM test cases (messages).

class ConversationalTestCase(BaseModel):
    messages: List[LLMTestCase]

Attributes:

messages (List[LLMTestCase]): List of LLM test case messages.

`LLMTestCase` Class

Represents a test case for LLM (Language Model) evaluation.

class LLMTestCase(BaseModel):
    query: str
    response: str
    context: Optional[Union[str, List[str]]] = None
    expected_response: Optional[str] = None

Attributes:

query (str): The input query.
response (str): The model’s response.
context (Optional[Union[str, List[str]]]): Context for the test case.
expected_response (Optional[str]): The expected response.

`MLLMImage` Class

Represents an image input for multimodal LLM test cases.

class MLLMImage(BaseModel):
    url: str
    local: Optional[bool] = None

Attributes:

url (str): URL or local path to the image.
local (Optional[bool]): Whether the image is local.

`MLLMAudio` Class

Represents an audio input for multimodal LLM test cases.

class MLLMAudio(BaseModel):
    url: str
    local: Optional[bool] = None
    is_plain_text: bool = False

Attributes:

url (str): URL or local path to the audio file.
local (Optional[bool]): Whether the audio is local.
is_plain_text (bool): Whether the input is plain text (not audio).

`MLLMTestCase` Class

Represents a multimodal LLM test case, supporting image and audio inputs.

class MLLMTestCase(TestCase):
    image_url: Optional[Union[str, MLLMImage]] = None
    input_image_url: Optional[Union[str, MLLMImage]] = None
    output_image_url: Optional[Union[str, MLLMImage]] = None
    input_audio: Optional[Union[str, MLLMAudio]] = None
    call_type: Optional[str] = None

Attributes:

image_url (Optional[Union[str, MLLMImage]]): Image input.
input_image_url (Optional[Union[str, MLLMImage]]): Input image.
output_image_url (Optional[Union[str, MLLMImage]]): Output image.
input_audio (Optional[Union[str, MLLMAudio]]): Input audio.
call_type (Optional[str]): Type of call (if applicable).

Example Usage

from fi.testcase import TestCase, LLMTestCase, ConversationalTestCase, MLLMTestCase

# Simple test case
tc = TestCase(input="What is the capital of France?", output="Paris")

# LLM test case
llm_tc = LLMTestCase(query="Who wrote 1984?", response="George Orwell")

# Conversational test case
conv_tc = ConversationalTestCase(messages=[llm_tc])

# Multimodal test case
mllm_tc = MLLMTestCase(
    input="Describe this image.",
    image_url="path/to/image.jpg"
)

​Test Case Classes

​TestCase Class

​ConversationalTestCase Class

​LLMTestCase Class

​MLLMImage Class

​MLLMAudio Class

​MLLMTestCase Class

​Example Usage

Test Case Classes

`TestCase` Class

`ConversationalTestCase` Class

`LLMTestCase` Class

`MLLMImage` Class

`MLLMAudio` Class

`MLLMTestCase` Class

Example Usage