Generate a Test Spec File

A test spec is a YAML file in your DX project that lists the test cases for a particular agent. The test spec file is local to your DX project and is an easy-to-read equivalent of the AiEvaluationDefinition metadata component that defines an agent test in your development org.

Run this interactive command from the VS Code integrated terminal to generate a test spec:

The agent generate test-spec command first asks you what type of test you want to make; currently, only AGENT is supported. It then asks you to select the agent you want to test from a list of agent API names. The command compiles this list by searching your local Salesforce DX project, it doesn't search your org. After you select an agent, the command continues asking for basic information, such as whether you want to overwrite the default file of the test spec if it already exists, and the name and description of the test.

The command then prompts you to create one or more test cases by asking you for this information for each test case:

• Utterance (Required) - A natural-language statement, question, or command to test the agent. Enter utterances that you expect the users of your agent to use. For example: “Who is working the front desk today at noon?” Or “Send me the list of customer complaints from yesterday.”

• Expected topic (Required) - The API name of the topic that you expect the agent to use when it responds to the utterance. The command gets the list of topics from the local metadata components in your Salesforce DX project that are associated with your agent, specifically GenAiPlugin.

• Expected actions (Required) - The API name of one or more actions associated with the topic that you expect the agent to use. The command gets the list of actions from the local metadata components in your Salesforce DX project associated with the agent topic, specifically GenAiFunction.

• Expected outcome (Required) - A natural-language description of the expected outcome of the test case. For example, “Anna is working the front desk.” Or “The list of customer complaints is provided.”

• Custom evaluation (Optional) - A way to test an agent response for specific strings or numbers. See Add Custom Evaluation Criteria to an Agent Test Spec if you opt to add a custom evaluation to your test.

• Conversation history (Optional) - Sample conversation history, which adds context to the test. See Add Conversation History to Enhance the Test Context if you opt to add conversation history to your test.

This is an example of a generated test spec file for the Resort_Manager agent. By default, the test spec is generated in the specs directory and its name is {Agent_API_name}-testSpec.yaml. This sample spec file defines one test case:

If you opt to not add custom evaluations or conversation history to the agent test, the customEvaluations and conversationHistory sections still show up but they're empty.

The agent generate test-spec command automatically adds the metrics: section; see Use Metrics to Assess Agent Responses for more information about these useful metrics.

Now that you understand the structure of the agent spec YAML file, you can manually edit it in the VS Code editor by adding new test cases.

If you have an AiEvaluationDefinition metadata XML file in your DX project, you can make a YAML test spec file from it. You can do this by passing the file path of the XML file to the --from-definition flag. This example also shows how to use the --output-file flag to specify the name of the spec file:

• Salesforce CLI Command Reference: agent Commands
• Salesforce Extensions for Visual Studio Code
• Salesforce Help: Agentforce Testing Center
• Reference: AiEvaluationDefinition
• Build Tests in Metadata API