Generate a Test Spec File

A test spec is a YAML file in your DX project that lists the test cases for a particular agent. ‌The test spec file is local to your DX project and is an easy-to-read equivalent of the AiEvaluationDefinition metadata component that defines an agent test in your development org.

Run this command from VS Code’s integrated terminal to generate a test spec:

As with all Salesforce CLI commands, you can also generate a test spec by opening a terminal (macOS, Linux) or command prompt (Windows), changing to your DX project directory, and running the agent generate test-spec CLI command.

The command asks you what test you want to make (currently only AGENT), the API name of the agent you want to test, and a name and description of your new test. The command doesn’t require an org; rather, it looks in your local DX project to compile the list of agent API names, topics, actions, and so on. The command then prompts you to create one or more test cases by asking you for this information for each one:

  • Utterance: Natural language statement, question, or command to test the agent. Enter utterances that you expect the users of your agent to use. For example: “Who is working the front desk today at noon?” Or “Send me the list of customer complaints from yesterday.”
  • Expected topic: The API name of the topic that you expect the agent to use when it responds to the utterance. The command gets the list of topics from the local metadata associated with your agent, specifically GenAiPlugin.
  • Expected actions: The API name of one or more actions associated with the topic that you expect the agent to use. ‌The command gets the list of actions from the local metadata associated with the agent topic, specifically GenAiFunction.
  • Expected outcome: Natural language description of the expected outcome of the test case. For example, “Anna is working the front desk.” Or “The list of customer complaints is provided.”

This is an example of a generated test spec file for the Resort_Manager agent; by default, the test spec is generated in the specs directory and its name is <Agent_API_name>-testSpec.yaml. This sample spec file defines just one test case:

You can keep running the agent generate test-spec command to refine your test spec, such as adding new test cases. Or you can simply edit the generated YAML file in the VS Code editor once you understand its simple structure.

If you have an AiEvaluationDefinition metadata XML file in your DX project, you can make a YAML test spec file from it. You can do this by passing the filepath of the XML file to the --from-definition flag. This example also shows how to use the --output-file flag to specify the name of the spec file: