Testing Overview
Simulations, interactive testing, voice-path checks, metrics, and trend review support deliberate verification before and after deployment.
The platform provides a testing and evaluation framework for verifying agent behavior before deployment and measuring quality in production. This section covers simulations for repeatable behavior checks, the playground for interactive testing, tiered voice testing, metrics for scoring conversation quality, and a practical process for finding degradation in those results over time.
Choose Evidence for the Question
Does the authored workflow handle the case?
Simulations and observed graph/tool activity
Representative context and failure cases
Does the experience work over audio?
Voice Simulation and the intended voice path
Speech, timing, interruption, and transport behavior
Does an external action complete correctly?
Isolated tool and integration tests
Destination acknowledgement or reconciliation under that operation's contract
Is the released workflow improving?
Metrics, comparable cohorts, and production examples
Missing-result accounting and the actual operational outcome
Testing produces evidence for a release decision. The operating model assigns that decision, and the deployment model describes how to select the reviewed configuration. The detailed sections below explain the available test paths.
Testing Philosophy
Healthcare workflows can span data retrieval, identity verification, external writes, and human follow-up. Each step needs evidence appropriate to its failure modes.
Testing every branch through live calls is slow and makes reasoning failures difficult to separate from audio or integration failures. The platform uses three principles:
Use controlled workspace context. Simulations read current workspace state. Dedicated test workspaces or entities provide stable fixtures when comparison across runs matters.
Exercise the real reasoning path without side effects. Text simulations run the production reasoning engine and Context Graphs, but every tool runs in simulation mode and returns a simulated result instead of reaching an integration, a world write, or a platform function. Surface delivery remains disabled. Test live tool behavior in the Playground with isolated test credentials and data.
Validate audio and integrations at their own boundaries. Real-audio tests cover speech and transport behavior. Connector and approval workflows need an appropriately isolated integration environment when their live behavior is under test.
This structure keeps broad reasoning regression tests fast while preserving separate evidence for media and integration behavior.
Simulations
Simulations test an agent against synthetic users in controlled scenarios. You define personas, situations, and evaluation criteria; the platform runs the conversations and scores the results. Saved cases group into suites for repeatable regression testing.
What You Can Validate
Conversation behavior - Exercise context graph branches, escalation rules, tool selection, and expected outcomes.
World-model context - Ground simulated callers in current workspace data and use dedicated test entities when a fixed starting state matters.
Tool selection - Check which tools the agent calls and with what arguments. Tools return a simulated
dry_runresult in simulations, so test real tool results in the Playground. Surface delivery and approval workflows are not available in simulations.Authorization-scoped parameters - SQL tools with authorization-bound parameters run in the text playground with their configured test values when the workspace actor holds the data-query permission. In simulations they return a simulated result like other tools. The Developer Console displays configured test values on tool detail pages and disables inline testing for authorization-bound tools, directing users to the playground instead.
Quality output - Shared evaluation definitions make simulation and production results comparable, while simulation evidence remains in simulation-specific views and stays out of production analytics.
Production Evaluations
Where eager evaluation is enabled, completed conversations can be evaluated against the workspace's active definitions. Triggering is best-effort and subject to task and per-workspace daily caps. A workspace without active definitions does not run this evaluation path. Inspect persisted results rather than assuming that every completed conversation was scored.
New production evaluation definitions must use the metric type. Legacy assertion definitions remain readable and executable while they are migrated, but new assertion definitions can no longer be created. Workspaces should express new evaluation criteria as metrics.
Active definitions determine what the enabled evaluation path scores. Conversations that complete before any definitions are active do not consume its daily cap.
Production evaluation is best-effort. A transient failure in the evaluation path does not affect the conversation itself, and evaluations can be re-run. Supported voice and text conversations can trigger evaluation at teardown when the path is enabled and its prerequisites are met. Completion alone does not establish that a result exists.
Reliable Suite Execution
Long suites are designed to reach a reportable result even when execution is interrupted:
Interrupted runs are retried automatically up to a bounded attempt limit.
Runs that cannot be recovered end with a failure reason, rather than remaining indefinitely in progress.
Each suite run records the number of cases selected at launch, making incomplete starts visible even if a case never produced a run record.
Summary views report run, session, and turn totals, while normalized checks give metrics and assertions a consistent verdict, score, and rationale shape.
Coverage APIs bring states, observed paths, gaps, and session evidence together so teams can identify untested branches. The Developer Console coverage visualization is currently an internal preview.
The Simulations landing page includes a performance overview organized by metric, with pass rates, score distributions, and trends visible before drilling into individual runs. Each conversation within a run shows its evaluation verdicts - assertion outcomes, metric scores, and justification text - so reviewers can trace a result back to the session that produced it.
Inspect the recorded path and outcome for each scenario, including cases that failed to start or complete.
Playground and Interactive Testing
Open Evaluate > Playground to test an eligible service through a browser voice call or turn-by-turn text. Follow First Console Conversation for the text walkthrough. Text mode uses the public conversation create, turn, history, and close operations, with live background-delivery events supplementing the returned results.
Evaluate Workspace
The Evaluate section groups Playground, Runs, Simulations, Case Library, and Suites. Use these alongside the separately available testing controls:
Interactive playground - Try voice and text behavior against a selected service.
Tool testing - Find Tool Tester through Console search or the Tools area to run controlled inputs before using a tool in a conversation.
Cases, suites, and simulations - Move from one-off exploration to repeatable regression coverage.
Test traffic controls - Keep designated test calls and credentials out of production billing, analytics, and downstream writes.
The Playground no longer provides operator takeover controls. Test an API-based intervention integration separately against its supported runtime; see Operators and Escalation.
When a service is configured to use a session-owning voice provider, Playground voice calls run through that provider rather than the default in-house pipeline. The provider generates its own opening audio, so the platform does not pre-render a separate greeting. Workflow tools are selected from the service's Context Graph, and the runtime can also expose supported system tools such as ending the call. Validate the complete tool set on the configured voice path.
Unified Runs
The standard workspace’s Evaluate > Runs page provides conversation inventory and retained framework-run history. Operator-only workspaces show Operate > Runs. Hosted framework-run dispatch and the native-definition registry are retired; historical records do not imply that new framework runs can be started. Runs has two tabs:
All runs - A filterable history across run kind, channel, and status, including a Live status filter.
Overview - Workspace totals and status distributions.
The former Live tab and takeover drawer are retired; old Live-tab bookmarks open All runs. Available live voice calls can appear while in progress. Filters accept multiple values within an axis, so teams can review combinations such as failed and timed-out runs without losing workspace-wide pagination.
Opening a run composes the appropriate detail view:
Voice runs show available audio, transcript, and analysis evidence.
Text, SMS, email, and web runs show the conversation transcript, available memory context, and eligible close actions.
Framework runs show metadata and an ordered structural trajectory of decisions, tool calls, and completion steps.
The separate Runs API retains controls for eligible channels and runtimes; these are not Console actions. Historical views include persisted successful, failed, and timed-out framework runs. Persistence is best-effort, so a run can still reach its correct terminal outcome even if its historical marker or trajectory is unavailable.
Use the playground to inspect how the configured service responds to interactive input and to capture cases for regression testing.
Voice Simulation
Voice testing separates text-tier quality (reasoning, tool selection, context graph coverage, and response content) from voice-path health (audio attachment, transcription, speech rendering, turn timing, latency, and capacity). Teams can run broad text-tier simulations for behavior regressions and reserve real-audio checks for the media path.
Inspect response content and audio delivery separately, including interruption, silence, and incomplete-call cases.
Metrics
Metrics measure the quality of agent conversations across the dimensions that matter to your organization - safety, clinical accuracy, empathy, goal completion, and other use-case-specific criteria. Eligible production conversations and simulations can be evaluated through their supported definitions and execution paths, while human review provides separate calibration evidence. Do not assume that every session is evaluated or that one definition shape applies unchanged across Classic, Platform production-eval, and simulation APIs.
The platform supports two evaluation modes for production conversations:
On-demand evaluation. Trigger evaluation for a specific completed conversation through the API. Useful for spot-checking individual conversations or integrating evaluation into custom workflows.
Eager post-conversation evaluation. Where the feature is enabled, completed voice, text, SMS, email, and web conversations can be evaluated against active definitions after they end. Triggering is best-effort and bounded by task and per-workspace daily caps, so this path does not guarantee a verdict for every conversation. Use persisted eval results to distinguish evaluated, pending, skipped, and error outcomes.
Both modes use the same production-eval definitions and verdict model. Trigger mode alone does not change the rubric, but channel modality, retained evidence, model execution, and missing artifacts can still affect whether two results are meaningfully comparable.
Interpret each metric using its rubric, input evidence, evaluated population, and missing results.
Drift Detection
Use metric trends, run history, and saved simulation baselines to look for sustained changes in inputs or outcomes. Teams choose the comparison windows, investigate representative interactions, update test cases, and make an explicit release decision. The platform does not currently turn a trend threshold into an automatic alert, promotion block, or rollback.
Investigate whether a change reflects agent behavior, source data, workload, or the measurement itself.
How the Pillars Work Together
These capabilities support a repeatable team workflow:
During development, the playground gives interactive feedback on agent behavior, and tool testing exercises individual tools outside a conversation with the side-effect policy documented for that tool family.
Before deployment, simulations exercise target scenarios and measure configured criteria, while targeted real-audio tests provide separate evidence about speech, timing, transport, and capacity changes.
In production, configured evaluations and analytics provide ongoing quality evidence.
Over time, teams compare equivalent periods, investigate material changes, and add newly discovered failure modes to the test set.
When the evidence indicates drift, update the relevant simulations and re-verify before releasing a change. This keeps the test set aligned with production without allowing an uncertain trend to change production automatically.
Last updated
Was this helpful?