agent generate test-spec
Generate an agent test spec, which is a YAML file that lists the test cases for testing a specific agent.
The first step when using Salesforce CLI to create an agent test in your org is to use this interactive command to generate a local YAML-formatted test spec file. The test spec YAML file contains information about the agent being tested, such as its API name, and then one or more test cases. This command uses the metadata components in your DX project when prompting for information, such as the agent API name; it doesn’t look in your org.
To generate a specific agent test case, this command prompts you for this information; when possible, the command provides a list of options for you to choose from:
- Utterance: Natural language statement, question, or command used to test the agent.
- Expected topic: API name of the topic you expect the agent to use when responding to the utterance.
- Expected actions: One or more API names of the expection actions the agent takes.
- Expected outcome: Natural language description of the outcome you expect.
- (Optional) Custom evaluation: Test an agent's response for specific strings or numbers.
- (Optional) Conversation history: Boilerplate for additional context you can add to the test in the form of a conversation history.
You can manually add contextVariables to test cases in the generated YAML file to inject contextual data (such as CaseId or RoutableId) into agent sessions. This is useful for testing agent behavior with different contextual information.
When your test spec is ready, you then run the “agent test create” command to actually create the test in your org and synchronize the metadata with your DX project. The metadata type for an agent test is AiEvaluationDefinition.
If you have an existing AiEvaluationDefinition metadata XML file in your DX project, you can generate its equivalent YAML test spec file with the –from-definition flag.
| Flag Name (Long) | Flag Name (Short) | Description |
|---|---|---|
‑‑flags‑dir | N/A | Type: Value Import flag values from a directory. |
‑‑force‑overwrite | N/A | Type: Boolean Don’t prompt for confirmation when overwriting an existing test spec YAML file. |
‑‑from‑definition | ‑d | Type: Value Filepath to the AIEvaluationDefinition metadata XML file in your DX project that you want to convert to a test spec YAML file. |
‑‑output‑file | ‑f | Type: Value Name of the generated test spec YAML file. Default value is “specs/<AGENT_API_NAME>-testSpec.yaml”. |
Generate an agent test spec YAML file interactively:
Generate an agent test spec YAML file and specify a name for the new file; if the file exists, overwrite it without confirmation:
Generate an agent test spec YAML file from an existing AiEvaluationDefinition metadata XML file in your DX project: