Continuous Improvement
Turn feedback into a reviewed workflow, test changes against clear criteria, and use production results to guide the next release.
Use conversation outcomes, tool results, evaluations, and feedback to decide what an agent should do differently. Define the intended behavior, test a candidate change, and compare results after release.
Concurrence provides analytics, simulations, configuration versions, and deployment tools for this work. Your team owns the workflow and release decisions; production feedback does not automatically rewrite or release agent configuration.
Turn Feedback Into a Patient Arc
A patient arc describes the path from a patient's request to completion or handoff. The customer and implementation team agree on its decision branches, required information, and allowed actions. They then implement that design through a Context Graph, its tools, and the surrounding application.
Group feedback by the work the patient is trying to complete. For each group, agree on the intended behavior before editing instructions:
Request and outcome: what the patient needs and what counts as completion.
Current facts: which system supplies each value, how fresh it must be, and what to do when it is missing or contradictory.
Decisions and actions: the conditions for each path, required permissions, and the result to verify.
Handoff: when to involve a person, who owns the next step, and what context they need.
Test cases: representative requests, exceptions, and expected results approved by the workflow owner.
For example, a refill-status arc could start by looking for an existing request:
The workflow owner supplies and reviews the eligibility and escalation rules. If clinical review is required, hand off to the authorized care team. Resolve missing or conflicting records before creating another request.
Retrieve changing values such as eligibility, order status, and appointment availability through authorized data tools. Keep those values separate from the workflow instructions. Put exact calculations and enforceable access checks in application code or tools; model-generated interpretation still needs evaluation.
Map the approved branches to configuration and regression cases. Each case should specify the expected action, answer, or handoff. Test the resulting agent behavior before release; a flowchart alone cannot guarantee model behavior or clinical safety. Build and Prove takes this design through configuration, evaluation, and promotion.
The Improvement Loop
1. Measure
Start with evidence that the platform already records:
Conversation and run outcomes
Configured evaluation results and metric trends
Context graph states and executed tool calls when trace evidence is available
Latency, escalation, and operator-participation signals
Surface completion and outbound delivery outcomes
Production evaluations can run on demand or after completed conversations when eager evaluation is enabled. Dashboards and APIs expose results over configurable time ranges so teams can establish a baseline before making a change.
2. Investigate
Aggregate trends identify where to look; individual interactions explain what happened. Compare periods and relevant segments, then inspect transcripts, executed actions, and trace evidence for representative successes and failures.
Treat a correlation as a candidate explanation, not an automatic configuration instruction. A lower completion rate for one service might reflect a changed caller population, unavailable source data, a tool failure, or agent behavior. Confirm the cause before changing the agent.
3. Author a Versioned Change
Make the smallest change that tests the hypothesis. Depending on the finding, that may be a context graph transition, agent instruction, tool binding, service setting, evaluation definition, or test case.
Agents and Context Graphs use explicit versions, and version sets can pin the versions a service releases. Record related changes to non-versioned settings or evaluation definitions with the same release decision. Use the authoring and inspection interfaces described in Build and Prove.
4. Validate Before Release
Use simulations to exercise the affected paths and compare the candidate against the baseline criteria. Add cases for newly discovered failure modes so the regression remains visible. Use real-audio tests when the change affects speech, timing, or media behavior, and use an isolated integration environment for connector or approval workflows.
Validation should cover both the target improvement and important guardrails. A faster scheduling flow is not an improvement if it lowers information accuracy or increases unsafe escalation behavior.
5. Release and Observe
An authorized user decides which tested versions and settings to release. After release, compare the same measures used for the baseline and watch for changes outside the target metric. Retained agent and context graph versions give the team a known configuration to restore deliberately if the release performs worse.
Set Release Criteria
Agree on release criteria that cover the workflow's outcome and operating requirements. For example:
Task completion meets its target
Safety-critical evaluations remain within policy
Information accuracy does not regress
Caller experience remains acceptable
Duration and cost stay within operating limits
Review the measures together. An improvement in average completion rate should not hide a failure in a smaller patient group or a safety-critical path.
Changes That Require a Decision
Analytics can show that one source, path, or tool correlates with stronger outcomes. That observation does not automatically:
Change event confidence or source precedence in the world model
Rewrite context graph transitions or tool order
Adjust escalation thresholds
Promote a version set
Roll back a production release
Assign an owner to review the evidence and authorize changes to these controls.
Governance
The improvement workflow uses the platform's normal governance boundaries:
Permissioned - Authorized roles control configuration and release actions.
Auditable - Configuration changes and deployment actions retain actor attribution.
Versioned - Candidate and released configurations remain distinguishable.
Bounded - Safety requirements are release constraints, not variables to trade away for a higher average score.
Building Reusable Evidence
Keep the evidence useful for the next release:
Convert production failure modes into permanent simulation cases.
Keep baseline periods and release criteria with the change record.
Reuse proven patterns across services only after validating them in the new context.
Review metrics after each release using the same definitions and segments.
Last updated
Was this helpful?