Test Case Best Practice

How to write AI agent test cases that simulate realistically and grade fairly.

A test case has four parts: Test Case (name and tags), Contact Prompt (how the simulated contact behaves), Success Criteria (what the judge grades), and Test Variables (contact data passed into the agent).

1. Test Case

  • One scenario per test. Each test should differ in persona, scenario, or path. Don't duplicate an existing test.
  • Ground every test in something real: a common situation from real calls, or a specific section of the agent prompt (Task Steps, Q&A/Objections, Guardrails, an action). Don't make up scenarios.
  • Test the agent's job, not its ability to resist tricks. Skip adversarial "break the agent" tests. Put guardrails into the success criteria of real scenarios instead.
  • Name: under 8 words, scenario-focused, no brand names, no "Test Case 1".
  • Tags: one scenario tag plus Regression for tests you'll maintain over time, for example Happy Path, Regression. Scenario tags: Happy Path, Unhappy Path, Compliance, Guardrails, Actions, Data Capture, Transfers, Knowledge Base. Use Stress Test for one-off scenarios.
  • Mix (single-state agent): about 65% end-to-end paths, 25% objections, 10% edge cases.
  • Every agent's suite should include:
    • No AI / speak to a human: the contact refuses to talk to AI. The agent hands off and stops qualifying.
    • Do Not Call (outbound agents): the contact asks to be removed. The agent follows its own prompt's soft-attempt rule, then confirms and ends the call.
    • Identity allowance (only when the prompt allows it): a spouse or other authorized person answers. The agent tries the named contact first, then continues with the other person.
NameTagsScenarioExpected outcome
Fee Question Mid-IntakeHappy Path, RegressionPartway through the call, the caller asks what it costsAgent answers from the prompt, then picks up the intake where it left off
Commercial Vehicle VIPTransfers, RegressionCaller was hit by a delivery truckAgent transfers to the VIP team without asking intake questions

2. Contact Prompt

Written from the contact's point of view. Under 600 words. Contains only contact instructions: no success criteria and no variables.

Structure

## Goal
[one sentence: what the contact wants]

## Call Scenario
[one sentence: why this call is happening]

## Contact Persona & Behavior
* [2–3 words, e.g. "Skeptical and busy."]

## Conversation Flow Instructions
* When asked [X]: "[exact reply]"
* If the agent asks [Y] again: "[reply]"
* ...one bullet per agent turn, through the agent's close...
* [end condition — see below]

What matters

  • One bullet per agent turn, and each one has a trigger and a quoted reply. Start each bullet with the agent's turn (When asked…, If the agent…), then give a quoted reply. A vague flow like "Confirms good time" doesn't give the simulator enough to work with.
  • Script the whole call, not just the interesting part. Keep going after the key moment, through the agent's wrap-up and any closing question. If the flow stops early, the contact hangs up before the agent can close.
  • Add If… branches for questions that get repeated, skipped, or asked out of order.
  • Talk like a real caller. Replies are 3–12 words, sometimes just one ("yeah", "no"). Use filler and hesitation ("uh", "sorry, say that again?"). Contacts don't volunteer information. On outbound calls they start out skeptical ("who is this?").
  • Silent call start: if the agent waits for the contact, the first bullet must have the contact speak first (* When the call connects: "Hello?").
  • End with the standard end condition, copied word for word. Never write "stay on the line" or "don't hang up first", because the simulated contact will repeat it and the call won't end.
    • Agent ends the call:
      * Goodbye loop: After the agent says goodbye, say a single goodbye or thanks line. Wait for the agent to call end_call or transfer_call. If the agent speaks again without calling either function, then end the call. Never say you will stay on the line, wait to disconnect, or remain until the agent hangs up.
      * Contact end call condition: Do not end on the agent's first goodbye. After one thanks or goodbye, wait for the agent to call end_call or transfer_call. End only if the agent speaks again without calling either function.
    • Agent transfers:
      * Contact end call condition: Do not end before the agent transfers the call to [destination]. When the agent transfers, end immediately and say nothing else.

Example: cooperative happy path with a transfer

## Goal
Get a cheaper auto insurance quote.

## Call Scenario
The agent is calling about a quote you requested online.

## Contact Persona & Behavior
* Cooperative and easy.

## Conversation Flow Instructions
* If asked whether you are Walter: "Yes, this is he."
* When the agent confirms your state: "Yes, that's correct."
* When asked whether this is auto, home, or both: "Just auto for now."
* When asked whether you're insured right now: "Yeah, I have coverage now."
* When asked why you're looking to switch: "Rates went up again."
* If asked how much you pay: "About nine hundred a month."
* When told you'll be connected to a licensed agent: "Okay, thank you."
* Contact end call condition: Do not end before the agent transfers the call to a licensed agent. When the agent transfers, end immediately and say nothing else.

Don't write flows like this (no agent-turn triggers, no quoted replies):

- Contact answers with a greeting first.
- Confirms good time.
- Says they have no heating right now.

3. Success Criteria

A full list the judge can grade on its own. Never write "see contact prompt".

Structure

- The agent [outcome 1]
- The agent [outcome 2, continuing through the close]
- The agent says something similar to: "[goodbye / transfer line]" and calls function [end_call / transfer_call / named function]
- The agent does NOT [forbidden behavior for this scenario]
- Do not grade [what's out of scope for this test]

## General Success Criteria
- Agent must not say robotic or system language: no JSON, brackets, or verbalized action inputs/payloads.
- Agent must not repeat phrases, say the same thing twice, or loop at any point in the call.
- When an action has to say a message or prompt message, the order of the function call and the spoken message does not matter as long as both happen in the same turn.

What matters

  • Cover every step through the close, ending with the final function call. Stopping at the lookup or the transfer offer leaves the criteria incomplete.
  • Use "says something similar to", never "says exactly". A valid paraphrase should pass.
  • Pair spoken lines with their function using "and", never "before" or "after". For example: says something similar to: "Have a great day" and calls function end_call. Putting an order between the agent's own speech and its function call fails correct calls. Ordering relative to the contact is fine ("…after the caller confirms").
  • Include at least one "does NOT" that's specific to this scenario, for example "does NOT transfer on a do-not-call request".
  • Add one judge-scoping bullet that says what not to grade, so wording differences don't fail the test.
  • Name functions specifically. Skip subjective words ("politely", "friendly") unless tone is the point of the test.
  • Always include the three General Success Criteria shown above, word for word.

Example: Do Not Call

- The agent acknowledges the request to be taken off the call list
- The agent follows this prompt's soft-attempt rule (e.g. at most one soft persuasion, or zero if the prompt forbids it)
- The agent says something similar to the prompt's removal confirmation and calls function end_call
- The agent does NOT transfer the call or keep asking qualification questions
- Do not grade how the soft attempt is phrased, only whether the count matches the prompt

## General Success Criteria
...

Example: No AI / human requested

- The agent acknowledges the request for a human
- The agent says something similar to the prompt's human-transfer message and calls the transfer function named in the prompt
- The agent does NOT keep qualifying after the caller refuses to speak to AI
- Do not grade small wording differences in the acknowledgement

## General Success Criteria
...

4. Test Variables

Contact data (custom vars) passed into the agent for the simulated call.

  • Use exact Regal dot notation, for example contact.firstName or contact.customProperties.leadSource.
  • Only include variables the test's path uses. If the agent greets the contact by name, set contact.firstName. Leave out anything the path never reads.
  • Use realistic values: shapes seen on real calls (e.g. actual lead-source values), not placeholder text. Vary values across tests to exercise different branches.
  • Testing missing data: include the key with an empty value rather than leaving it out, so the agent takes the "collect it" branch.
  • Prefill values that trigger behavior, for example contact.state when a custom action needs it to look up coverage.
  • Match the version under test. If the agent no longer reads a field, drop it, even if older tests still have it.
VariableExampleNote
contact.firstNameJordanNeeded when the agent confirms identity
contact.customProperties.leadSourceReferralUse real values from your data
contact.emailAddress(empty)Tests the path where the agent collects a missing email

Did this page help you?