Create Custom Scorers for Agent Testing

Custom scorers allow you to define evaluation logic for agent testing. Standard Expectations cover common test scenarios like topic matching and response coherence, but custom scorers create expectations tailored to your business requirements.

Custom scorers evaluate agent behavior at different levels of granularity. A custom scorer can test an entire conversation, a single interaction, or a specific moment within an interaction. Scorers use a prompt template to evaluate agent behavior automatically with an LLM.

Use the AiAgentScorerDefinition Metadata API type to define custom scorers and deploy them to your org.

A custom scorer evaluates agent behavior and produces a result that maps to an outcome: pass, fail, or not applicable. Each scorer has two key components:

Engine: The evaluation logic. Use a PromptTemplate engine to assess the agent’s behavior with an LLM.
Output mapping: Rules that translate the engine’s result into a pass, fail, or not-applicable outcome.

In this guide, we define a custom scorer that uses a prompt template to detect whether a customer dropped off before the conversation resolved.

Agentforce is enabled in your org with at least one active agent. See Set up Agents in Salesforce Help.
If your scorer uses the PromptTemplate engine type, the prompt template must exist in your org or get deployed alongside the scorer. See Deploy a Scorer with a Prompt Template for details on deploying both together.

Create an AiAgentScorerDefinition metadata component to define your scorer. The component exists in the aiAgentScorerDefinitions folder with the .aiAgentScorerDefinition file suffix.

The scorer defines two inputScope fields that work together to control what data the scorer evaluates:

Field	Valid values	Description
Top-level `inputScope`	`Session`	The data the scorer’s evaluation logic (its prompt template or manual scores) operates on.
`agentAssociation.inputScope`	`Session`, `Intent`	The unit of agent activity that the scorer runs against. Use `Intent` to score each intent in a session individually.

To reference the latest interaction inside a session-scoped scorer, use the getLastInteraction invocable action within your prompt template.

Field	Type	Description
`inputScope`	string	Required. The scope of agent data the scorer’s evaluation logic operates on. Valid value: `Session`.
`scorerType`	string	Required. The scorer’s output shape. Valid values: `Predefined`, `OpenEnded`. Use `Predefined` when the scorer returns one of a fixed set of values; use `OpenEnded` for free-form output typed by a Lightning Type.
`dataType`	string	Required. The data type of the scorer’s output. Valid values: `Text`, `Number`, `LightningType`. Use `Text` or `Number` with `scorerType: Predefined`; use `LightningType` with `scorerType: OpenEnded`.
`lightningType`	string	Required when `dataType` is `LightningType`. The Lightning Type that describes the scorer’s output — for example, `lightning__numberType`, `lightning__textType`, or `lightning__booleanType`.
`semanticType`	string	Optional. Describes how to interpret the scorer’s output for reporting. Valid values: `Dimension`, `Measurement`.
`scorerVersion`	scorerVersion[]	Required. The version configuration for the scorer. Scorers support multiple versions.

Version numbers must be sequential starting from 1, and each scorer supports a maximum of 100 versions.

Field	Type	Description
`versionNumber`	integer	Required. The version number. Must be sequential starting from `1`.
`status`	string	Required. The lifecycle status. Valid values: `Draft`, `Available`, `Archived`. See Scorer Status.
`description`	string	Required. A description of what the scorer evaluates.
`label`	string	Required. A display label for the scorer version.
`agentAssociation`	AgentAssociation	Required. Associates the scorer with a specific agent.
`engine`	engine[]	Required. The evaluation logic for the scorer.
`outputEnumValue`	outputEnumValue[]	Required. One or more mappings that translate engine output values to pass or fail outcomes.
`specification`	specification[]	Optional. Constraints on the scorer’s output values, such as min, max, step, and threshold.

Field	Type	Description
`isActive`	boolean	Required. Whether the scorer is active for the associated agent. Can only be `true` for versions with `Available` status. Only one agent association per scorer can have `isActive` set to `true`. Set to `false` for scorers with `engineType: Manual` — manual scorers are annotated by human reviewers and have no active state.
`agentApiName`	string	Required. The API name of the agent. The agent must exist in the org. For example, `Copilot_for_Salesforce`.
`inputScope`	string	Optional. The unit of agent activity that this scorer runs against for the associated agent. Valid values: `Session`, `Intent`. Use `Intent` to score each intent in a session individually.
`samplingRate`	double	Optional. A value greater than `0` and up to `1.0` that controls the sampling rate. Default is `1.0`.

Field	Type	Description
`engineType`	string	Required. The type of evaluation engine. Valid values: `PromptTemplate`, `Manual`. Use `Manual` for scorers that accept human-reviewed scores.
`engineRef`	string	Required when `engineType` is `PromptTemplate`. The API name of the prompt template. Omit for `Manual` scorers.

Field	Type	Description
`value`	string	Required. The engine output value that maps to this outcome.
`outcomeType`	string	Optional. The test outcome. Valid values are: `Pass`, `Fail`, `NotApplicable`. Default value is `NotApplicable`.
`description`	string	Optional. A description of what this output value represents.
`isFallback`	boolean	Optional. Whether the LLM can emit this value as the scorer’s fallback — for example, `Neutral` for a `High`/`Normal`/`Low` scorer. Required for `Predefined` scorers with `dataType: Text`; not applicable to `Predefined` scorers with `dataType: Number`.
`isSystemFallback`	boolean	Optional. Whether the system uses this value when the prompt template or LLM execution fails. Often set together with `isFallback` on the same value.

Field	Type	Description
`max`	double	The maximum valid output value.
`min`	double	The minimum valid output value.
`step`	double	The increment between valid output values.
`threshold`	double	Optional. Output values greater than or equal to `threshold` pass.

Each scorer version progresses through a lifecycle controlled by the status field.

Status	Description
`Draft`	A work in progress. Use `Draft` to edit the scorer’s fields freely. A draft version can’t run evaluations.
`Available`	A finalized scorer. For scorers with `engineType: PromptTemplate`, set `agentAssociation.isActive` to `true` to activate it for evaluation. Manual scorers keep `isActive` set to `false` — reviewers post scores directly through the Annotation API.
`Archived`	A retired version. The Testing API no longer runs evaluations against this version.

Create a scorer version in Draft status when you want to iterate on the scorer’s configuration before finalizing it. When the scorer is ready for use, promote the version to Available. After a version is marked Available, its status can’t be changed back to Draft.

This example defines a custom scorer that evaluates whether a customer drops off before a conversation resolves. The scorer uses a prompt template to analyze the session and outputs a value of 0 (no drop-off, pass) or 1 (drop-off detected, fail).

To deploy a custom scorer, create a project directory with this structure:

The package.xml file specifies the scorer to deploy:

The members value must match the filename of your .aiAgentScorerDefinition file (without the extension).

Deploy the scorer to your org with the Salesforce CLI:

If your scorer uses the PromptTemplate engine type, you can deploy both the template and the scorer together. Add a genAiPromptTemplates folder containing your prompt template definition, and add the template to package.xml.

In package.xml, the GenAiPromptTemplate type must appear before AiAgentScorerDefinition. Metadata API deploys types in the order that they appear, and the prompt template must exist before the scorer that references it can successfully deploy.

To retrieve a scorer definition from your org, use the Salesforce CLI:

You can also retrieve a scorer definition by using the same package.xml that you used for deployment.

To update an existing scorer, modify the .aiAgentScorerDefinition file and redeploy. Keep in mind these constraints:

You can add new versions to a scorer, but you can’t delete existing versions.
You can edit a version’s fields while its status is Draft.
You can promote a version’s status from Draft to Available, or move an Available version to Archived. After a version is marked Available, you can’t change its status back to Draft.
You can update the agentAssociation isActive and samplingRate values.
The scorer checks the members name in package.xml. If a scorer with that name already exists, the deployment updates the existing scorer.
Scorers with engineType: Manual don’t have an active state — always set agentAssociation.isActive to false. Human reviewers post scores directly through the Annotation API regardless of the association’s isActive value.