Metrics and Quality

Define multi-objective evaluation criteria and use their results in customer-owned release checks and improvement targets.

Metrics measure the quality of agent conversations. Teams define the dimensions that matter, then evaluate simulations or eligible production conversations through the evaluation surface supported by that API. Classic metrics, Platform production-eval definitions, and Platform metric-store definitions are related concepts with distinct schemas and lifecycles.

What Metrics Measure

Metrics evaluate agent performance across the dimensions that determine success in your domain. Common categories include:

Conversation quality: Was the response clear? Did the agent understand the user's intent? Was the information complete and accurate?

Safety adherence: Did the agent stay within its defined scope? Did it escalate when appropriate? Were there any safety boundary violations?

Goal completion: Did the agent accomplish what the user needed? Was the appointment scheduled, the question answered, the referral made?

Domain-specific dimensions: In healthcare, this includes clinical accuracy, protocol adherence, empathy, and risk disclosure completeness. Each organization defines the dimensions that reflect their standards.

Three Evaluation Sources

Metrics can be generated from three different sources, each serving a different purpose.

Production Evaluation

Production conversations can be evaluated on demand. Where eager post-conversation evaluation is enabled, the platform also schedules eligible completed conversations automatically, subject to execution and daily caps.

Persisted verdicts support performance trends and drift review. Because eager triggering and model-backed evaluation are best-effort, dashboards should distinguish missing, pending, skipped, and error results from completed scores.

Post-session metrics can use model-based judges that assess the conversation evidence available to that evaluation path against your criteria. They run outside the live response path, but the input may be a retained transcript, selected turns, or a conversation summary and can be incomplete. Inspect the definition and evidence before interpreting a score.

Simulation Evaluation

When a simulation run is configured with evaluation criteria, eligible simulated conversations produce the corresponding scores or verdicts. This provides synthetic-interaction evidence before the configuration reaches real users without implying that every simulation produces every workspace metric.

Simulation evaluation serves two purposes:

• Pre-deployment validation: Confirm that metric scores meet your thresholds before promoting a new configuration.

• Comparative analysis: Run the same test set against two configurations and compare metric scores to understand the impact of a change.

Human Calibration

Human reviewers can score representative conversations in an organization's review workflow and compare those judgments with stored platform verdicts. The Platform production-eval API does not itself provide a general-purpose human scoring endpoint. Human calibration is useful for:

• Calibrating automated metrics: Compare human scores against automated scores to verify that your metrics capture what they should.

• High-stakes review: Flag conversations that need human judgment, such as those involving clinical decisions or safety escalations.

• Discovering new dimensions: Human reviewers sometimes notice quality issues that existing metrics do not capture, which can inform new metric definitions.

Multi-Objective Scoring

Healthcare and other high-stakes domains require that the agent performs well across multiple dimensions at the same time. An agent that is clinically accurate but lacks empathy fails to deliver a good outcome. An agent that is empathetic but misses a safety escalation is dangerous.

Multi-objective scoring evaluates a conversation against several criteria at once. Teams can treat selected thresholds as release gates in their own deployment process; the platform records evaluation evidence but does not automatically block a version-set update or make the release decision.

Example: Post-discharge follow-up quality gates

Metric	Threshold	Type
Clinical accuracy	99%	Hard gate (must pass)
Safety escalation accuracy	100%	Hard gate (must pass)
Protocol adherence	95%	Hard gate (must pass)
Empathy score	80%	Soft target
Response completeness	90%	Soft target

In this example, hard gates are customer-defined release requirements: a failing result should block the team's release workflow regardless of other scores. Soft targets inform improvement priorities without blocking deployment on their own.

Configuring Metrics

Evaluation definitions commonly include:

• Name and description: What the metric measures.

• Evaluation criteria: The specific rubric or checklist the evaluator uses to score the conversation.

• Scoring method: Pass/fail for binary requirements, or a numeric scale (typically 0-100) for graded dimensions.

• Threshold or expectation: The value the team's evaluation or release workflow treats as acceptable.

Exact fields depend on the API. Platform production evals use assertion or metric definitions, while Classic metrics use the Classic metric schema.

Start with a small set of high-impact metrics rather than trying to measure everything. A focused set of 5-10 well-calibrated metrics provides more actionable insight than 50 loosely defined ones.

Conversation Quality Check

Automated metrics do not catch every behavioral failure. The forge quality check command scans production conversations against behavioral detectors that catch agent issues metrics may miss: stuck loops where the agent repeats itself, character degeneration, repetitive patterns, incoherent output, and phantom success (the agent claims a tool call succeeded when it actually failed). Quality checks query conversation data directly and report findings with severity, timestamps, and optional message excerpts. See Agent Forge CLI for usage.

Dashboards

The Eval Summary dashboard provides a consolidated view of simulation and eval metric scores across your workspace - broken down by metric, over time, and by categorical outcome. See Intelligence and Analytics for details on built-in dashboards.

Evaluation Metrics vs Operational Metrics

Evaluation metrics and assertions score individual conversations against rubrics or deterministic checks - "How well did the agent handle this interaction?" They provide evidence for testing, simulation, and customer-owned deployment gates.

The Metric Store serves a different purpose: it defines a common value model and catalog for operational, quality, and evaluation metrics whose producers and schedules differ. A Platform production eval can reference an eligible AI-query metric definition and persist both the verdict and the resulting conversation-scoped metric value.

For operational dashboards, alerting, and cross-channel analytics, see Metric Store.

Using Metrics Strategically

Separate safety metrics from quality metrics. Organizations may set zero-tolerance acceptance criteria for observed safety violations and use quality metrics such as empathy, clarity, and completeness as improvement targets. A 100% score describes the evaluated evidence set, not proof that future model behavior cannot fail.

Review metric trends, not just individual scores. A single low empathy score might be an outlier. A sustained downward trend warrants investigation of the population, definition, model, data, and agent configuration before attributing a cause.

Calibrate regularly. Have human reviewers score a sample of conversations and compare against automated scores. If automated and human scores diverge, update your evaluation criteria.

For Developers: See Production Evals and the Metric Store for Platform contracts. For Classic API definitions and evaluation, see the Classic Metrics API.