Skip to main content

Testing a Voice AI Agent with Agent Simulate SDK

This notebook demonstrates how to use the agent-simulate SDK to test a conversational voice AI agent. We will:
  1. Install the necessary libraries.
  2. Start a local LiveKit development server.
  3. Set up environment variables.
  4. Define a simple, local support agent to act as the agent-under-test.
  5. Define a test scenario with a simulated customer persona.
  6. Run the simulation and record the conversation.
  7. Display the transcript and play back the recorded audio.
  8. Run evaluations on the conversation.
Open In Colab

1. Installation

First, let’s install the agent-simulate SDK and other required Python packages.

Download VAD Model

The livekit-agents SDK uses the Silero VAD (Voice Activity Detection) plugin. We need to download its model weights before we can start the simulation.

2. Start LiveKit Server

For this demo, we’ll run a local LiveKit development server. Open a new terminal and run the following commands to download and start the server:
The server will keep running in that terminal.

3. Set Environment Variables

We need to configure our API keys and LiveKit server details. The livekit-server --dev command prints the key, secret, and URL you need. Important:
  • Copy the API Key, API Secret, and URL from the livekit-server output.
  • You will also need an OPENAI_API_KEY for the simulated customer’s LLM.
  • If you want to run evaluations, you’ll also need your FI_API_KEY and FI_SECRET_KEY.

4. Define the Agent-Under-Test

Instead of connecting to a remote, deployed agent, we’ll define and run a simple SupportAgent locally. The TestRunner will manage spawning this agent for each test case.

5. Define Test Scenario & Persona

Now we’ll use the agent-simulate SDK to define the test case. We need two main components:
  1. AgentDefinition: Tells the TestRunner how to spawn our local SupportAgent.
  2. Scenario: Contains one or more Persona objects that define the simulated customer’s details.

6. Run the Simulation

Now we’ll instantiate the TestRunner and call run_test. This will:
  1. Create a new, unique LiveKit room for this test.
  2. Spawn our SupportAgent and connect it to the room.
  3. Connect the simulated customer (“Fubar”) to the room.
  4. Record the full conversation.
  5. Return a TestReport containing the results.

7. View Results

The TestReport object contains the full transcript and paths to the recorded audio files. Let’s display the transcript. In an interactive notebook, you could use IPython.display.Audio to play back the combined conversation.

8. Run Evaluations

The agent-simulate SDK includes a helper function, evaluate_report, to easily run evaluations on your test results using the ai-evaluation library. You define a list of eval_specs, which map fields from the TestReport (like transcript or audio_combined_path) to the inputs required by your chosen evaluation templates.