Run Tests in Connect API

The Connect API endpoints for Testing API focus on executing test cases, polling for results, and programmatically retrieving detailed test results.

The Testing Connect API has three endpoints:

  • Start Test: Starts an asynchronous test on an agent. The test evaluates predesigned test cases that are deployed via Metadata API or the Testing Center.
  • Get Test Status: Retrieves the operational status of a specific test. We designed this endpoint to poll to monitor the progress of a test.
  • Get Test Results: Retrieves a detailed report of a test, including information on each test case and the results of all predetermined expectations.

To use Salesforce CLI to run agent tests instead of directly using Connect API, see Run the Agent Tests with Agentforce DX.

To successfully test your agents, you need predefined test cases in metadata files or generated tests from the Testing Center. Review Metadata API test definitions in Metadata API Reference. To call Testing API endpoints in Connect API, you first need to create a connected app and create a token.

To use Testing API, create a connected app to support the client credential flow. See these topics in Salesforce Help:

These instructions walk you through a basic connected-app implementation that supports Testing API communication.

  1. From Setup, find and select App Manager.
  2. From the App Manager page, click New Connected App and then select Create a Connected App.
  3. For Connected App Name, enter an app name.
  4. For Contact Email, enter your admin email address.
  5. In the API section, select Enable OAuth Settings. This action displays additional OAuth-related settings.
  6. For Callback URL, enter https://login.salesforce.com.
  7. From Selected OAuth Scopes, add these scopes to the connected app:
    • Access Connect REST API Resources (chatter_api)
    • Manage user data via APIs (api)
    • Manage user data via Web browsers (web)
    • Perform requests at any time (refresh_token, offline_access)
  8. Deselect Require Proof Key for Code Exchange (PKCE) Extension for Support Authorization Flows.
  9. Select these items:
    • Enable Client Credentials Flow
    • Issue JSON Web Token (JWT)-based access tokens for named users
  10. Save the app. Then click Continue. After saving the app, you see the Manage Connected Apps page.
  11. Click Manage.
    If you need to get this to this same screen later, select Manage Connected Apps from Setup.
  12. Click Edit Policies.
  13. In the Client Credentials Flow section, set Run As to the execution user for your org. For Enterprise Edition orgs, we recommend that you select an execution user who has the API Only User permission.
  14. Check Issue JSON Web Token (JWT)-based access tokens. Leave the Token Timeout value as 30 minutes.
  15. Save the app.

To create a token, you need the consumer key and the consumer secret from your connected app.

  1. From Setup, find and select App Manager.
  2. Find your connected app, click the dropdown arrow, and then select View.
  3. Click Manage Consumer Details.
  4. Copy the Consumer Key and Consumer Secret values.

All calls to Testing API require a token. Create a token by using the consumer key, the consumer secret, and your domain name.

  • MY_DOMAIN_URL: Get the domain from Setup by searching for My Domain. Copy the value shown in the Current My Domain URL field.
  • CONSUMER_KEY, CONSUMER_SECRET: Get the consumer key and secret by following the instructions in Obtain Credentials.

The previous cURL request returns a JSON payload similar to this response.

Copy the access token specified in the access_token property. This token is required when you make requests to the API. Congratulations! The Testing Connect API is now ready for use. Review Connect API Reference to familiarize yourself with required headers, parameters, and response objects.

After you complete the setup steps and use Metadata API to deploy your tests, run your tests by using these three Connect API endpoints:

  • Start Test: POST …/einstein/ai-evaluations/runs
  • Get Test Status: GET …/einstein/ai-evaluations/runs/:runId
  • Get Test Results: GET …/einstein/ai-evaluations/runs/:runId/results

To learn how to use Connect API, see Connect REST API Quick Start.

To start a test, make a POST request to /services/data/v63.0/einstein/ai-evaluations/runs. Provide the name of the test definition (specified in the metadata component) in the body of the request. See Start Test.

  • INSTANCE_NAME: The instance of your Salesforce org.
  • TOKEN: The access token obtained from Create a Token.
  • TEST_NAME: The name of the test to start.

The response returns an evaluation ID that you can use to check the status of the test and the test results.

To check the test status, make a GET request to /services/data/v63.0/einstein/ai-evaluations/runs/{runId}

The response provides information about the test. See Get Test Status.

After the test is completed, make a GET request to /services/data/v63.0/einstein/ai-evaluations/runs/{runId}/results

The response contains the results of the test. See Get Test Results.

If your tests all pass, congratulations! However, if one or more tests fail, you have some work to do. For more information, look at the errorMessage field in a failed test result. Use the conversation preview panel in the Agent Builder UI to talk to the active agent and test the words and responses in a conversational way. Then use the information to fine tune your agent instructions, actions, or topics.

  • A topic test checks if the agent responded with the expected topic when it received the utterance. A topic test is defined by an expectation name of topic_sequence_match in the AiEvaluationDefinition metadata component. If the test fails, check the topic's expectedValue defined in AiEvaluationDefinition versus the topic that the agent actually used.

  • An action test verifies if the agent used the expected action or actions. An action test is defined by an expectation name of action_sequence_match in the AiEvaluationDefinition metadata component. If the test fails, check the action's expectedValue defined in AiEvaluationDefinition versus the action that the agent actually used.

  • An outcome test uses a semantic comparison between the expected and actual values using natural language. Even if the text of the actual outcome differs from the expected outcome, the test can still pass if the core meaning is the same. However, if the actual outcome is significantly different, the test fails. An outcome test is defined by an expectation name of bot_response_rating in the AiEvaluationDefinition metadata component. If the test fails, check the action's expectedValue defined in AiEvaluationDefinition versus the actual agent response.