In this headless example, you analyze production data to identify incorrect or suboptimal agent responses and underperforming conversations. Your AI assistant identifies root causes, turns them into durable test cases, and suggests fixes, which you validate against a full test suite in your testing environment, such as a sandbox, before promoting to production.
Use your preferred AI assistant, such as Claude Code or Agentforce Vibes.
Complete all the prerequisites in Headless Examples.
In your terminal, run sf org list. If you see the headless org alias in the list, you’re successfully authenticated.
This example also requires:
QnA agent created in Ground an Agent with Agentforce Data Libraries but the same techniques work for any agent.Copy and paste this prompt into your AI assistant:
1Analyze the last seven days of session traces for the QnA agent. This agent answers simple questions by retrieving answers from our knowledge base (RAG). Find the questions it got wrong or where it said "I don't know" or gave a low-confidence answer, at least three times each. For each one, tell me: the question customers asked, what the agent answered, what the correct answer should have been, and why it likely failed — for example, the answer wasn't in the knowledge base, the wrong article was retrieved, or the question was phrased in a way the agent didn't recognize.Your AI assistant analyzes session traces to look for potential optimizations, grouped by root cause rather than as a flat list of individual suboptimal responses. If the agent doesn’t have any production session traces or a test suite, your AI assistant synthesizes utterances based on the agent structure and analyzes the responses instead.
Your AI assistant returns a ranked list of recurring issues and root causes, but you won’t act on the suggestions yet.
Before you start to optimize the agent, convert what you learned into durable regression tests. We ensure that any Personally Identifiable Information (PII) is removed from the test cases.
Copy and paste this prompt into your AI assistant:
1For each recurring question failure you found, create a Testing Center test case in CSV format with these columns: Utterance, Expected Topic, Expected Action, Expected Outcome. Use the customer's actual question (or a close paraphrase) as the Utterance, the QnA topic as Expected Topic, the knowledge retrieval action as Expected Action, and the correct answer as the Expected Outcome. Replace any customer name, ID, or personal detail in the utterance with a generic placeholder (like "the customer" or a made-up example) so no real customer data appears in the file. Save this as a CSV file named qna-agent-test-cases.csv, formatted so it can be uploaded directly to Testing Center or submitted through the Testing Center API's test suite creation endpoint.Never copy raw production records or transcripts into a test case, even temporarily. Generalize the failure condition and scrub or synthesize any supporting data first. This step keeps your test suite safe to store in source control and safe to review like code.
Important
Here’s an example row in qna-agent-eval-cases.csv:
1Utterance: "What's your return policy for online orders?" | Expected Topic: QnA | Expected Action: Knowledge Retrieval | Expected Outcome: "You have 30 days from the delivery date to return your product for a full refund, provided it's in original condition with all packaging, remote, and accessories."This format matches what Testing Center expects for a CSV-based test suite upload, and the same fields map to the test suite creation payload if you’re calling the Testing Center API directly instead of using the UI.
Copy and paste this prompt into your AI assistant:
1Create a test suite and add the test cases from qna-agent-test-cases.csv.Some of the initial test cases may fail because the data library files that ground the agent have gaps. The test case failures are the intended regression signal. For a production agent, you would address the test case failures by improving the quality of the grounding data for the agent.
Note
Now that you have test cases covering the issues that you found, you can implement and validate fixes in a test environment. You don’t want to make agent changes in a production environment. Depending on your preferred development lifecycle environment, you might test in a sandbox.
MyDomainName--SandboxName.1sf org login web --alias headless-sandbox --instance-url https://MyDomainName--SandboxName.sandbox.my.salesforce.com1Apply the suggested fixes for the identified issues to a new version of the agent. Show me a diff of the proposed changes before finalizing them.Review the diff before accepting the changes. Don’t silently apply changes to agent metadata. It’s important to keep a human in the loop!
Don’t publish the agent yet because we have one more quality gate to pass.
Copy and paste this prompt into your AI assistant:
1Run the full QnA agent test suite in Testing Center against the new agent version — any existing test cases plus the test cases generated from qna-agent-test-cases.csv. Report pass or fail per test case, and flag any question that now gets a wrong or low-confidence answer that previously passed.Treat this step as an important quality gate, not a suggestion. Only promote the new agent version to production if the full suite passes, including any pre-existing regression test cases. A fix that resolves one issue but silently breaks another is worse than no fix at all.
After the full test suite passes, copy and paste this prompt into your AI assistant:
1Deploy the validated agent version to production. Tag this deployment with the test suite version used to validate it.Closing the loop is what makes this process repeatable rather than a one-off optimization.
Copy and paste this prompt into your AI assistant, 24–48 hours after deployment or after a meaningful volume of sessions:
1Re-run trace analysis for this agent covering the period since the last deployment. Confirm whether the originally identified issue rate has decreased. Flag any new recurring issues introduced since the deployment.If any issues persist or a new regression appears, roll back to the previous agent version and repeat the earlier steps with the updated findings. As you repeat the steps in this optimization cycle, you will build up a richer test suite and your agent response quality will continue to improve.
j