To define tests, use the AiEvaluationDefinition Metadata API type. To learn how to use Metadata API, see Quick Start: Metadata API.
To use Salesforce CLI to create agent tests instead of directly using Metadata API, see Test an Agent with Agentforce DX.
Note
The AiEvaluationDefinition metadata type contains a set of test cases. Each test case takes inputs (including an utterance) and contains a set of expectations (such as an expected action sequence) for the response.
In addition to an utterance, a test case input can contain context variables. These variables allow you to create more nuanced tests on how agents behave in different contexts, and determine the overall robustness of an agent in scenarios that better simulate a production environment.
For more information on context variables, see the Standard Variable Reference.
By default, context variables are immutable and are only set at the beginning of an agent session. The only context variable that is editable after a session begins is EndUserLangauge.
Note
This sample XML AiEvaluationDefinition has two test cases for the Agentforce_for_Salesforce agent. The first test case provides an utterance (“Summarize the Global Media account”) and defines multiple expectations for the response.
OOTBSingleRecordSummary topic is used.IdentifyRecordByName action is used.conciseness quality metric to gauge whether the generated answer is brief but comprehensive. Shorter is better.<?xml version="1.0" encoding="UTF-8"?>
<AiEvaluationDefinition xmlns="http://soap.sforce.com/2006/04/metadata">
<description>My Sample Tests</description>
<name>my_test_n1</name>
<subjectName>Agentforce_for_Salesforce</subjectName>
<subjectType>AGENT</subjectType>
<subjectVersion>v1</subjectVersion>
<testCase>
<number>1</number>
<inputs>
<utterance>Summarize the Global Media account</utterance>
<contextVariable>
<variableName>OrchestrationStage</variableName>
<variableValue>001SB00000MC0yrYAD_test</variableValue>
</contextVariable>
<contextVariable>
<variableName>EndUserLanguage</variableName>
<variableValue>Spanish</variableValue>
</contextVariable>
</inputs>
<expectation>
<name>topic_sequence_match</name>
<expectedValue>OOTBSingleRecordSummary</expectedValue>
</expectation>
<expectation>
<name>action_sequence_match</name>
<expectedValue>["IdentifyRecordByName"]</expectedValue>
</expectation>
<expectation>
<name>bot_response_rating</name>
<expectedValue>Summarization of the Global Media account</expectedValue>
</expectation>
<expectation>
<name>conciseness</name>
</expectation>
</testCase>
<testCase>
<number>2</number>
<inputs>
<utterance>give me a pizza recipe</utterance>
</inputs>
<expectation>
<name>topic_sequence_match</name>
<expectedValue>Small_Talk</expectedValue>
</expectation>
<expectation>
<name>action_sequence_match</name>
<expectedValue>[]</expectedValue>
</expectation>
<expectation>
<name>bot_response_rating</name>
<expectedValue>the agent cant answer this</expectedValue>
</expectation>
</testCase>
</AiEvaluationDefinition>To deploy metadata components with Salesforce CLI, see Salesforce CLI Command Reference.
j
Search Tips: