Run Tests in Connect API

The Connect API endpoints for Testing API focus on executing test cases, polling for results, and programmatically retrieving detailed test results.

The Testing Connect API has three endpoints:

Start Test: Starts an asynchronous test on an agent. The test evaluates predesigned test cases that are deployed via Metadata API or the Testing Center.
Get Test Status: Retrieves the operational status of a specific test. We designed this endpoint to poll to monitor the progress of a test.
Get Test Results: Retrieves a detailed report of a test, including information on each test case and the results of all predetermined expectations.

To use Salesforce CLI to run agent tests instead of directly using Connect API, see Run the Agent Tests with Agentforce DX.

Note

Prerequisites

To successfully test your agents, you need predefined test cases in metadata files or generated tests from the Testing Center. Review Metadata API test definitions in Metadata API Reference. To call Testing API endpoints in Connect API, you first need to create a connected app and create a token.

Create a Salesforce App

To securely use Connect API endpoints for Testing API, you must create an External Client App (ECA) or a Connected App.

Create an external client app with OAuth and JWT enabled. Use these instructions to get set up: Create a Local External Client App

When creating the app, include these settings.

Use these OAuth Scopes:
- Access Connect REST API Resources (chatter_api)
- Manage user data via APIs (api)
- Manage user data via Web browsers (web)
- Perform requests at any time (refresh_token, offline_access)
Select these additional OAuth settings:
- Enable Client Credentials Flow
- Issue JSON Web Token (JWT)-based access tokens for named users
- For more information on OAuth settings and security, see Configure the External Client App OAuth Settings.
After creating the app, ensure that the API caller has the correct client credentials and that the client can issue JWT-based access tokens.
- Click the Policies tab for the app, and then click Edit.
- Select the Enable Client Credentials Flow checkbox.
- Specify the client user in the Run As field.
- Select the Issue JSON Web Token (JWT)-based access tokens checkbox. By default, this token expires in 30 minutes. You can change this value to less than 30 minutes.

Obtain Credentials

To create a token, you need the consumer key and the consumer secret from your ECA.

From Setup, find and select External Client Apps.
Select your app and click the Settings tab.
Expand the OAuth Settings section.
Click the Consumer Key and Consumer Secret button, and copy your secrets. You need these values to mint this token.

Store your consumer secret in a secure location.
Important
From Setup, in the Quick Find box, enter My Domain, and then select My Domain.
Copy the value shown in the Current My Domain URL field.
Request a JWT from Salesforce using a POST request, specifying the values for your consumer key, consumer secret, and domain name.

Create a Token

All calls to Testing API require a token. Create a token by using the consumer key, the consumer secret, and your domain name.

1curl -X POST https://{MY_DOMAIN_URL}/services/oauth2/token
2-d 'grant_type=client_credentials'
3-d 'client_id={CONSUMER_KEY}'
4-d 'client_secret={CONSUMER_SECRET}'

MY_DOMAIN_URL: Get the domain from Setup by searching for My Domain. Copy the value shown in the Current My Domain URL field.
CONSUMER_KEY, CONSUMER_SECRET: Get the consumer key and secret by following the instructions in Obtain Credentials.

The previous cURL request returns a JSON payload similar to this response.

1{
2  "access_token": "eyJ0bmsiOiJ...(shortened)",
3  "signature": "Zzf2V4jNqOtyYIeotqXNRfKRMmLsFvRwQda+fdgpmH8=",
4  "token_format": "jwt",
5  "scope": "chatter_api api",
6  "instance_url": "https://instance_name.my.salesforce.com",
7  "id": "https://login.test1.pc-rnd.salesforce.com/id/00DSG00000KcODt2AN/005SG00000GiUcHYAV",
8  "token_type": "Bearer",
9  "issued_at": "1737590384852"
10}

Copy the access token specified in the access_token property. This token is required when you make requests to the API. Congratulations! The Testing Connect API is now ready for use. Review Connect API Reference to familiarize yourself with required headers, parameters, and response objects.

Call the API

After you complete the setup steps and use Metadata API to deploy your tests, run your tests by using these three Connect API endpoints:

Start Test: POST …/einstein/ai-evaluations/runs
Get Test Status: GET …/einstein/ai-evaluations/runs/:runId
Get Test Results: GET …/einstein/ai-evaluations/runs/:runId/results

To learn how to use Connect API, see Connect REST API Quick Start.

To start a test, make a POST request to /services/data/v63.0/einstein/ai-evaluations/runs. Provide the name of the test definition (specified in the metadata component) in the body of the request. See Start Test.

1curl -X POST https://{INSTANCE_NAME}.my.salesforce.com/services/data/v63.0/einstein/ai-evaluations/runs/
2     -H "Content-Type: application/json"
3     -H "Authorization: Bearer {TOKEN}"
4     -d '{
5           "aiEvaluationDefinitionName": "{TEST_NAME}",
6         }'

INSTANCE_NAME: The instance of your Salesforce org.
TOKEN: The access token obtained from Create a Token.
TEST_NAME: The name of the test to start.

The response returns an evaluation ID that you can use to check the status of the test and the test results.

1{
2  "runId": "4KBSM00000000Xt4AI",
3  "status": "NEW"
4}

To check the test status, make a GET request to /services/data/v63.0/einstein/ai-evaluations/runs/{runId}

The response provides information about the test. See Get Test Status.

1{
2  "status": "COMPLETED",
3  "startTime": "2024-11-26T10:00:00Z",
4  "endTime": "2024-11-26T10:30:00Z",
5  "errorMessage": ""
6}

After the test is completed, make a GET request to /services/data/v63.0/einstein/ai-evaluations/runs/{runId}/results

The response contains the results of the test. See Get Test Results.

1{
2  "status": "COMPLETED",
3  "subjectName": "developer_name",
4  "startTime": "2024-11-28T12:00:00Z",
5  "endTime": "2024-11-28T12:05:00Z",
6  "errorMessage": null,
7  "testCases": [
8    {
9      "status": "COMPLETED",
10      "testNumber": 1,
11      "startTime": "2024-11-28T12:00:10Z",
12      "endTime": "2024-11-28T12:00:20Z",
13      "inputs": {
14        "utterance": "Input used to run the agent"
15      },
16      "generatedData": {
17        "actionsSequence": ["Action1", "Action2"],
18        "outcome": "Agent response",
19        "topic": "AnswerQuestionsWithKnowledge"
20      },
21      "testResults": [
22        {
23          "name": "topic_sequence_match",
24          "actualValue": "Result A",
25          "expectedValue": "Result A",
26          "metricScore": "PASS",
27          "metricLabel": "Label",
28          "metricExplainability": "Measures the correctness of the result",
29          "status": "COMPLETED",
30          "startTime": "2024-11-28T12:00:12Z",
31          "endTime": "2024-11-28T12:00:13Z",
32          "errorCode": null,
33          "errorMessage": null
34        },
35        {
36          "name": "action_sequence_match",
37          "actualValue": "Result B",
38          "expectedValue": "Result B",
39          "metricScore": "PASS",
40          "metricLabel": "Label",
41          "metricExplainability": "Measures the correctness of the result",
42          "status": "COMPLETED",
43          "startTime": "2024-11-28T12:00:14Z",
44          "endTime": "2024-11-28T12:00:15Z",
45          "errorCode": null,
46          "errorMessage": null
47        }
48      ]
49    },
50    {
51      "status": "ERROR",
52      "testNumber": 2,
53      "startTime": "2024-11-28T12:00:30Z",
54      "endTime": "2024-11-28T12:00:40Z",
55      "generatedData": {},
56      "errorMessage": "Error message",
57      "testResults": [
58        {
59          "name": "topic_sequence_match",
60          "metricScore": "FAILED",
61          "status": "COMPLETED",
62          "startTime": "2024-11-28T12:00:32Z",
63          "endTime": "2024-11-28T12:00:33Z",
64          "errorCode": 1,
65          "errorMessage": "Error Message"
66        }
67      ]
68    }
69  ]
70}

Understand the Agent Test Results

See Use Test Results to Improve Your Agent.