Headless Example: Optimize an Agent by Analyzing Production Data

In this headless example, you analyze production data to identify incorrect or suboptimal agent responses and underperforming conversations. Your AI assistant identifies root causes, turns them into durable test cases, and suggests fixes, which you validate against a full test suite in your testing environment, such as a sandbox, before promoting to production.

Use your preferred AI assistant, such as Claude Code or Agentforce Vibes.

Complete all the prerequisites in Headless Examples.

In your terminal, run sf org list. If you see the headless org alias in the list, you’re successfully authenticated.

This example also requires:

  • Data 360
  • The Audit and Feedback and Agentforce Session Tracing settings must be enabled on the Einstein Audit, Analytics, and Monitoring Setup page. You also need the Salesforce Standard Data Model managed package. See Set Up Agent Optimization in Salesforce Help.
  • An agent in production. We use the QnA agent created in Ground an Agent with Agentforce Data Libraries but the same techniques work for any agent.

Copy and paste this prompt into your AI assistant:

Your AI assistant analyzes session traces to look for potential optimizations, grouped by root cause rather than as a flat list of individual suboptimal responses. If the agent doesn’t have any production session traces or a test suite, your AI assistant synthesizes utterances based on the agent structure and analyzes the responses instead.

Your AI assistant returns a ranked list of recurring issues and root causes, but you won’t act on the suggestions yet.

Before you start to optimize the agent, convert what you learned into durable regression tests. We ensure that any Personally Identifiable Information (PII) is removed from the test cases.

Copy and paste this prompt into your AI assistant:

Never copy raw production records or transcripts into a test case, even temporarily. Generalize the failure condition and scrub or synthesize any supporting data first. This step keeps your test suite safe to store in source control and safe to review like code.

Here’s an example row in qna-agent-eval-cases.csv:

This format matches what Testing Center expects for a CSV-based test suite upload, and the same fields map to the test suite creation payload if you’re calling the Testing Center API directly instead of using the UI.

Copy and paste this prompt into your AI assistant:

Some of the initial test cases may fail because the data library files that ground the agent have gaps. The test case failures are the intended regression signal. For a production agent, you would address the test case failures by improving the quality of the grounding data for the agent.

Now that you have test cases covering the issues that you found, you can implement and validate fixes in a test environment. You don’t want to make agent changes in a production environment. Depending on your preferred development lifecycle environment, you might test in a sandbox.

  1. Run this terminal command to log in to your sandbox. Substitute your sandbox name for MyDomainName--SandboxName.
  1. After you’re authenticated in the sandbox, copy and paste this prompt into your AI assistant:

Review the diff before accepting the changes. Don’t silently apply changes to agent metadata. It’s important to keep a human in the loop!

Don’t publish the agent yet because we have one more quality gate to pass.

Copy and paste this prompt into your AI assistant:

Treat this step as an important quality gate, not a suggestion. Only promote the new agent version to production if the full suite passes, including any pre-existing regression test cases. A fix that resolves one issue but silently breaks another is worse than no fix at all.

After the full test suite passes, copy and paste this prompt into your AI assistant:

Closing the loop is what makes this process repeatable rather than a one-off optimization.

Copy and paste this prompt into your AI assistant, 24–48 hours after deployment or after a meaningful volume of sessions:

If any issues persist or a new regression appears, roll back to the previous agent version and repeat the earlier steps with the updated findings. As you repeat the steps in this optimization cycle, you will build up a richer test suite and your agent response quality will continue to improve.