Customize the Agent Test Spec

Running the agent generate test-spec command results in a good agent test spec. But there are other items you can add to the file to create even better tests. Because the agent test spec file is in YAML format, it's easy to update manually.

Service agents can connect to customer channels. Specifically, you can configure your Service agent to route conversations to it with enhanced messaging channels. See Connect a Service Agent to Messaging.

You can also personalize your Service agent conversations with context variables. Context variables let your agent use record fields of the Messaging Session standard objects as action inputs. See Use Context Variables in Service Agent Conversations.

If you've configured your Service agent with context variables, you can test how they work at the same time you test your agent. Do this by adding a contextVariables: section to your test cases in the agent test spec. For each context variable you want to test, add a name|value pair. The name property is the API name of the context variable and value is a specific value you want to test.

For example, let's say that you added the EndUserLanguage context variable to your Service agent. This variable specifies the preferred language of the user participating in the messaging session. To run a test case as if the user were talking in Spanish, manually update your YAML agent test spec like this.

To find the API name of a context variable, open your agent in your development org's Agent Builder UI and go to the Context tab.

Agent Builder showing the Context tab and EndUserLanguage variable highlighted.

Context variable API names correspond to field names of the MessagingSession standard object.

When you run agent generate test-spec, the command automatically adds out-of-the-box metrics in a metrics: section for each test case in the generated spec YAML file. For example:

Use these metrics to assess the quality and latency of agent responses during testing. Specifically:

  • Coherence - Measures whether the response is easy to understand and has no grammatical errors.
  • Completeness - Measures whether the response includes all the essential information.
  • Conciseness - Measures whether the response is brief but comprehensive. Shorter is better.
  • Latency - Returns the latency in milliseconds from sending a request until a response is received.

After you run an agent test that has these metrics in the test YAML file, you get a Metrics section in the test case output similar to this:

See Use Test Results to Improve Your Agent for information on the returned scores for these metrics.

If you don't want to include these metrics in an agent test run, simply delete the specific metric, or the entire metrics: section, from the test case in the spec YAML file.