Metadata API Reference for Testing API

To define tests, use the AiEvaluationDefinition Metadata API type.

Represents an evaluation's purpose, the agent being tested, and the test sets to be executed.

In Metadata API, you can create test definitions, including specifying inputs and expected outcomes, and deploy them to different orgs. In Connect API, you can execute test scenarios, poll for results, and retrieve test outcomes.
This type extends the Metadata metadata type and inherits its fullName field.

AiEvaluationDefinition components have the suffix .aiEvaluationDefinition and are stored in the aiEvaluationDefinitions folder.

AiEvaluationDefinition components are available in API version 63.0 and later.

AiEvaluationDefinition is available only if Agentforce is enabled. See Set Up Agents in Salesforce Help.

Field Name	Type	Description
`description`	string	The purpose of the test.
`name`	string	Required. The API name of the test. Can contain only underscores and alphanumeric characters and must be unique in your org. It must begin with a letter, not include spaces, not end with an underscore, and not contain two consecutive underscores.
`subjectName`	string	Required. A unique identifier for the agent being tested. Make sure that this identifier matches the API name of the agent, which you can find on the agent details page in Setup.
`subjectType`	string	Required. The type of subject being tested. The only currently supported value is `AGENT`.
`subjectVersion`	string	The agent version to test. If not provided, the latest active version is used by default. You can find the version in the BotVersion metadata type.
`testCase`	AiEvaluationTestCase[]	A list of test cases.

Represents a test case.

Field Name	Type	Description
`expectation`	AiEvaluationExpectation[]	The criteria used to test the artifact's responses.
`inputs`	AiEvaluationAgentTestCaseInput[]	The specific input provided to the artifact being tested.
`number`	int	The unique number for the test case. If not provided, the value is automatically calculated.

Represents the expected outcome for a test case.

Field Name	Type	Description
`expectedValue`	string	Required. The expected outcome of the test. The format of this field depends on the value of the `name` field. The expected outcome is compared against the response generated when you run the test using Connect REST API.
`name`	string	Required. The expectation name. Valid values are: `topic_sequence_match`, For this value, the `expectedValue` field value is a string representing the topic that the agent is expected to use, such as `OOTBSingleRecordSummary`. For a list of agent topics, see Standard Agent Topic Reference in Salesforce Help. `action_sequence_match`, For this value, the `expectedValue` field value is a string[] representing a list of actions that you expect the artifact to take during the test, such as `['IdentifyRecordByName', 'action2']`. For a list of agent actions, see Standard Agent Action Reference in Salesforce Help. `bot_response_rating`, For this value, the `expectedValue` field value is a string representing the expected response generated by the artifact, such as `Summarization of the Global Media account`.

Represents the input for a test case.

Field Name	Type	Description
`utterance`	string	Required. The request sent to the agent.

Here's an example of an AiEvaluationDefinition component.

This metadata type supports the wildcard character * (asterisk) in the package.xml manifest file. For information about using the manifest file, see Deploying and Retrieving Metadata with the Zip File.