Run the Agent Tests with Agentforce DX

Once you have agent tests in your development org, either by creating them with a CLI command, deploying them with Metadata API, or using the Testing Center UI in your org, you have multiple pro-code options to run the tests and view the results:

  • Use CLI commands. This method is particularly useful in headless continuous integration (CI) environments.
  • Use VS Code’s Test view in the Side Bar, similar to how you run Apex and LWC tests.
  • Use Testing API, see Run Tests in Connect API.

You can, of course, also run them from the low-code Testing Center UI if you prefer.

Use the agent test run command to run an agent test. The command runs the test in your development org and requires the test’s API name. If you created the test using the agent test create CLI command, you provided the API name for the test.

As always, run the command from VS Code’s integrated terminal, or use a terminal (macOS, Linux) or a command prompt (Windows) from your DX project. In this example, the API name of the test is Resort_Manager_Test and the development org has the alias my-dev-org:

If you’ve set a default org in your DX project, such as your sandbox, you don’t need to specify --target-org.

Run this command to get a list of all the available agent tests in your development org; the first column lists the test API name:

The tests are run asynchronously by default, which means the agent test run command simply outputs the agent test resume command that you then run to resume and view the test results. The resume command uses a job ID, for example:

To run the tests synchronously, use the --wait flag and specify the number of minutes to wait for the command to finish:

By default, the agent test run command outputs human-readable results to the terminal. Use the --result-format flag to get JSON, TAP (Test Anything Protocol), or Junit test results, and the --output-dir to specify where to put the resulting files. For example:

To view the results of a completed agent test run, use agent test results. The command requires a job ID, which was outputted when you originally ran the test. For example:

When the agent test run command completes, it outputs overall test results and details about each test case. Here’s a screenshot of the default human-readable output after running the command in VS Code’s integrated terminal.

Human-readable output of the “agent test run” results, including expected and actual results for each agent test case.

The top section (1) of the command output gives basic information, such as the overall status of the test, the number of test cases run, and the number of tests that succeeded or failed.

The middle section (2) details each test case, starting with the utterance specified in the agent test spec YAML file. The table breaks down the different components of the test case and indicates whether each part succeeded or failed. For example:

  • The Topic test checks if the agent responded with the expected topic when it received the utterance. If the test fails, the Actual column shows which topic the agent actually used.
  • Similarly, the Action test verifies if the agent used the expected action(s); if not, the Actual column displays the action(s) the agent actually used.
  • The Outcome test is particularly interesting because it evaluates expected and actual values using natural language. Even if the text of the actual outcome differs from the expected outcome, the test can still pass if the core meaning or "gist" of both texts is the same. However, if the actual outcome is significantly different, the test fails.

The bottom section (3) summarizes the overall test results, such as the duration of the test and the total percentage pass rate for topic, action, and outcome tests.

View and run agent tests from VS Code’s Agent Test panel, similar to how you can already view and run Apex or LWC tests.

When you run agent tests from VS Code, they run in your development org, the same as when you run tests using the CLI agent test run command. But VS Code also requires that your DX project contain the metadata component associated with the agent test (AiEvaluationDefinition) in a package directory.

VS Code showing file explorer of "agentforcedx" DX project with an AiEvaluationDefinition XML file highlighted in the package directory.

So be sure the metadata in your development org and DX project are synchronized. To retrieve an agent test that’s in your org but not yet in your DX project, run this command from VS Code’s integrated terminal:

To explore the agent tests, click the beaker icon in the Activity Bar on the left side of the VS Code editor, then open Agent Tests. The panel lists the tests it found, with the numbered test cases listed below. To run a test, click the green arrow to the right of its name.

VS Code showing beaker icon in left Activity Bar highlighted and the green arrow next to an agent test highlighted.

The Output panel displays the running results. Once the test completes, click on a test case in the testing panel for detailed results.

VS Code showing agent test results in the output tab

The test case results, while in a slightly different format, show the same information as running the test with the CLI agent test run command. See Understand the Agent Test Results for details.