Voice Agent
Understand the real-time voice pipeline, context graph execution, tool calls, emotion adaptation, operator takeover, recordings, and post-call analysis.
The Concurrence voice agent powers real-time voice conversations. It handles inbound and outbound phone calls with speech understanding, text-to-speech, tool execution, safety monitoring, and context-graph or native/realtime runtime behavior. Supported service configurations can load projected patient context, emit source-attributed world events, and adapt responses from best-effort acoustic analysis; those capabilities vary by runtime and configured tools.
Audio Pipeline Architecture
The standard phone-call runtime uses the five-layer pipeline below. Optional emotion, world-model, and post-call stages depend on service configuration, available data, and runtime support.
Layer 1: Signal Capture
Speech recognition and optional acoustic emotion analysis use separate processing paths. Backpressure on the emotion path can drop analysis segments rather than blocking transcription; shared session, audio, or infrastructure failures can still affect both paths.
Speech-to-Text: real-time streaming transcription. Three sources of domain vocabulary can improve recognition of configured terms:
Service-level keyterms: managed by workspace administrators, applied to all calls for that service.
Workspace voice settings keyterms: API-configurable per workspace (see voice settings).
System defaults: engineering-level fallback vocabulary.
All sources are merged and deduplicated per call. Configurable end-of-turn detection with tunable confidence thresholds determines when the caller has finished speaking, balancing responsiveness against cutting off mid-sentence.
Emotion detection is a best-effort parallel stream. The runtime converts supported caller audio and sends approximately 2-second segments to the emotion service. Each result contains a nine-class acoustic score distribution plus valence and arousal. Raw classes are mapped to runtime labels such as Anger, Disgust, Fear, Joy, Calmness, Sadness, Surprise, and Contemplation.
Final transcripts can be sent separately for sentiment and toxicity analysis. Acoustic inference and transcript analysis are independent inputs; the current contract does not include a separate vocal-burst taxonomy.
The emotion service can return a dominance value for observer telemetry, but dominance is not used in the rolling state or turn-level compound resolver. Runtime adaptation uses the mapped acoustic scores, valence/arousal, transcript signals when available, and behavioral/contextual evidence.
Backpressure behavior: acoustic analysis uses approximately 2-second audio segments. Under load, a segment can be dropped rather than delaying the transcription and call path.
Failure behavior: an initial connection failure leaves the call running without acoustic adaptation. Repeated receive failures disable the emotion stream for that session after the configured limit; the contract does not promise automatic reconnection during the same call.
Layer 2: Intelligence
The Emotional State maintains the latest four acoustic segments (about 8 seconds at the normal segment size) with linear recency weighting. At each caller turn, the controller snapshots that state and keeps up to five turn snapshots for compound-emotion resolution.
Valence/arousal aggregation: the emotion engine supplies valence and arousal per segment. The runtime linearly weights the latest four segments, derives an internal dominant label from the aggregate, and uses the latest acoustic distribution plus the five-turn window to resolve compound labels. Dominance is currently fixed to 0.0 in those turn snapshots and does not affect resolution.
Trend detection: compares first-half vs second-half valence of the rolling window. A delta > 0.1 → improving; delta < -0.1 → deteriorating; otherwise stable. This powers the call-phase escalation system: deteriorating trends trigger increasingly urgent adaptation.
Acoustic/transcript coherence: when transcript sentiment is available, the runtime compares it with acoustic valence and accounts for toxicity. Near-neutral voice is treated as ambiguous rather than automatic agreement. Coherence remains telemetry and steering input; it is absent or less informative when transcript analysis is unavailable.
Behavioral signal tracking: updated in real-time from the session and turn controller:
barge_in_count
Caller interrupts agent speech
≥ 2
Frustration (agent talking too much)
short_response_streak
Consecutive responses ≤ 4 words
≥ 3
Disengagement (caller withdrawing)
silence_gap_count
Gaps ≥ 5 seconds
≥ 2
Confusion, hesitation, or distress
These behavioral signals can be injected into the system prompt alongside emotional steering, giving the LLM additional evidence about both how the caller sounds and how they are behaving.
Layer 3: Context Graph Engine
Each turn processes through a two-stage LLM pipeline:
Navigator: selects the next action or exit condition from the current context graph state. Also generates a filler phrase to cover processing latency. Uses structured output validation with automatic retry (up to 3 attempts) and fallback to first valid action.
Engage LLM: generates the caller-facing response, informed by the selected action, available conversation history with per-message emotion annotations (
[VOICE: EmotionName, valence=V.VVV]), emotional steering context, ambient patient context, and available tools.
Emotion reaches the LLM via two independent paths:
Per-message annotations
Annotated user messages retained in context
Inline [VOICE: Anxiety, valence=-0.312], so the LLM can use the emotional trajectory in the available history
Session-level steering
System prompt
Dominant emotion + trend, quadrant-specific adaptation instructions, behavioral signals, call-phase urgency, coherence warnings
Communication micro-behaviors: when the standard engage template is used, it includes guidelines for micro-level conversational behaviors independently of whether emotion data is available:
Speech rhythm mirroring
If the caller speaks in short bursts, respond concisely; if conversational, match warmth
Emotional name usage
Use the caller's name at moments of emotional significance, not mechanically
Pause injection
When delivering difficult information, pause naturally before the key detail
Pace inversion
When the caller is rushing, slow the pace with longer sentences and gentle transitions
Completion inference
When the caller trails off mid-sentence, acknowledge what they were trying to say
Emotion concealment
Instruct the model not to mention emotion detection explicitly
Natural laughter
Contextual laughter available for naturally warm moments, used sparingly
Layer 4: Audio Output
The engage LLM's text streams to the TTS engine for speech synthesis with per-turn dynamic controls:
Emotion: selected through the voice tone selection path.
Speed: from workspace voice settings.
Volume: from workspace voice settings.
When the active TTS provider supplies word timing metadata, the runtime can retain start/end offsets for transcript-to-audio alignment. Clients must not assume every provider or utterance includes complete word-level timestamps.
Layer 5: Post-Call Intelligence
Terminal operational intelligence, retained analysis artifacts, and separately configured evaluations have different prerequisites and can be absent on an otherwise completed call.
Transcript verification is retired. Current call responses no longer expose verified_transcript, verified_words, transcript_accuracy, or per_turn_accuracy. The transcript_correction_enabled voice setting is also removed. Read the retained live transcript and available recording instead; do not wait for verification fields or manufacture a zero accuracy score. The separate transcript-only sentiment/topic/intent analysis was also removed; terminal operational summaries remain.
Quality analysis: call_analysis remains an optional compatibility field. Current source does not connect the retained post_call_analysis_enabled setting to the former five-dimension recording-analysis job. Use Call Trace Analysis, Voice Judge, and Production Evals for their separately documented outputs and prerequisites.
Call Intelligence Persistence
The platform-managed runtime records a structured intelligence summary at call end, independently of recording analysis. It captures operational signals that are not available from transcript scoring alone.
Fully analyzed call intelligence records contain:
quality_score
float or null
Rule-based operational quality score (0-100), penalty-based
emotion_summary
object
dominant_emotion, average_valence, arousal, peak negative, shifts, final trend
latency_summary
object
Engine response time (avg/p50/p95), audio TTFB (avg/p50/p95), silence ratio
conversation_summary
object
turn_count, states_visited_count, unique_states, loop_count, barge-in count, completion reason
tool_summary
object
Total calls, success/failure counts, failure rate, per-tool breakdown
operator_summary
object
Whether the call escalated and, when it did, whether an operator was active
risk_summary, safety_summary
object
Compatibility fields that are empty in the current runtime; they are not active risk-scoring or safety-triage outputs
completion_reason
string
Why the call ended (hangup, terminal state, silence, etc.)
final_state
string
Last context graph state at call end
Some external voice runtimes produce only a terminal call-intelligence envelope, with quality_score null and the analysis summaries empty. That does not mean their conversation history is empty: the gpt_realtime path separately retains transcripts and tool calls when persistence succeeds. See Call Record & Persistence.
Quality score penalties:
High latency
p95 audio TTFB > 1000ms
-5 to -15
Silence
Silence ratio > 0.2
-10 to -20
Barge-ins
> 2
-5 to -15
Agent loops
> 0 revisited states
-10 to -20
Escalation
Any
-10
Tool failures
Failure rate > 5%
-5 to -15
Call Intelligence Endpoints
The call intelligence endpoint exposes the persisted terminal summary for a completed call:
Aggregated intelligence for a completed call: quality breakdown, key moments, and summaries. Per-turn visualization lives on the unified timeline (call detail endpoint → timeline.segments).
API key issued via POST /v1/{workspace_id}/api-keys. Pass the returned api_key value as a Bearer token.
Successful Response
Layer 4 (Narrative) intelligence for a completed call.
Aggregated summaries, quality breakdown, and key moments — the interpretive layer. Per-turn visualization data lives on the unified timeline (call detail → PlaybackTimeline.segments), not here.
World entity ID for the call
Twilio call SID
Call direction
inboundPossible values: Call duration in seconds
0Quality score 0.0 to 1.0
Why the call ended
Final conversation state name
When intelligence was computed
Intelligence data not found for this call
Validation Error
GET /v1/{workspace_id}/calls/{call_id}/intelligence HTTP/1.1
Host: platform.api-us.concurrence.com
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
{
"call_id": "text",
"call_sid": "text",
"direction": "inbound",
"duration_seconds": 0,
"quality_score": 1,
"quality_breakdown": {
"score": 1,
"components": [
{
"name": "latency",
"penalty": 1,
"value": 1,
"threshold": "text",
"severity": "none"
}
],
"workspace_avg": 1,
"workspace_percentile": 1
},
"key_moments": [
{
"turn": 1,
"type": "latency_spike",
"severity": "warning",
"description": "text"
}
],
"emotion_summary": {
"dominant_emotion": "text",
"average_valence": 1,
"average_arousal": 1,
"peak_negative_valence": 1,
"barge_in_count": 0
},
"risk_summary": {
"composite_score": 0,
"level": "low"
},
"latency_summary": {
"avg_engine_ms": 1,
"p50_engine_ms": 1,
"p95_engine_ms": 1,
"avg_audio_ttfb_ms": 1,
"p50_audio_ttfb_ms": 1,
"p95_audio_ttfb_ms": 1,
"avg_nav_ms": 1,
"avg_render_ms": 1,
"turn_count": 1,
"silence_ratio": 1
},
"conversation_summary": {
"turn_count": 0,
"states_visited_count": 0,
"unique_states": 0,
"loop_count": 0,
"barge_in_count": 0
},
"tool_summary": {
"total_calls": 0,
"succeeded": 0,
"failed": 0,
"failure_rate": 0
},
"safety_summary": {
"match_count": 0
},
"operator_summary": {
"escalated": false,
"operator_active": true
},
"completion_reason": "completed",
"final_state": "text",
"created_at": "2026-01-01T00:00:00.000Z"
}Returns 404 if the call or intelligence data is not found.
Recognition review: compare the live transcript with available recordings and validate recognition keyterms explicitly. Current source does not emit the former stt_suggestions completion payload; clients should not wait for it or treat its absence as a processing failure.
How Calls Work
Supported phone-call flows use a conference architecture, a multi-party audio bridge that lets the caller, AI agent, and optionally a human operator participate simultaneously.
Call identifiers returned as call_sid are opaque platform handles, distinct from run IDs and individual carrier call-leg identifiers. Preserve them unchanged across observer, recording, and supported call-scoped operations; see Calls. Inbound phone identity can be absent: valid caller numbers are normalized to E.164, while anonymous, withheld, invalid, or non-phone origins remain unidentified. Handle missing identity before relying on caller lookup or automatic channel context.
Inbound Call Preparation
Supported inbound telephony paths can reduce startup delay through best-effort parallel pre-warming: engine initialization, greeting preparation, patient-context resolution, and the agent connection can begin while the phone is ringing. Completion before pickup is not guaranteed.
The conference name is available at webhook time in the standard telephony implementation, allowing the runtime to begin creating the agent leg before pickup. Whether the agent connection, context, and greeting are ready at answer time depends on ring duration and downstream dependencies.
Outbound Call Flow
Outbound voice has two entry paths. On-demand outbound calls start directly through the Calls API. Scheduled outreach is represented as an outbound_task entity. This section describes the scheduled path.
Common workflow patterns that an application can implement with outbound tasks include:
Scheduled
Decision made, execution deferred
"I'll call you back tomorrow at 2pm"
Event-reactive
Trigger → evaluate → maybe act
New lab result → is it critical? → call patient
Continuous monitoring
Periodic population sweep
Patients with no contact in 30 days
Conversational follow-through
Track preconditions from agent promises
"I'll call after the doctor reviews" → pending on doctor event
Orchestrated campaign
Achieve outcome for population over time
"Get all 200 patients to complete annual wellness by Q4"
Each outbound task can carry a patient reference, reason, goal, priority (1-10), business-hours window (timezone-aware), retry config, and context from the patient's world-model projection. The dispatch loop can enrich the system prompt with context resolved at dispatch time. Missing, stale, or non-projected facts can still require a tool lookup or caller confirmation.
Outbound preparation: the current standard outbound path prepares the first agent audio before dialing the human leg. If that preparation fails or exceeds its budget, the request fails instead of falling through to an unprepared dial. Provider connection and playback can still add latency after answer, so this is not a time-to-first-audio guarantee.
Conference Architecture
Context Graph Engine
The platform-managed voice runtime executes a Hierarchical State Machine loaded from the service's version set. Each call maintains isolated conversation state. The state types and navigation flow below describe this runtime.
The separate realtime-provider runtime builds workflow instructions from exactly one ActionState. It renders the state's objective, action descriptions, action guidelines, and global guidance into the provider's instructions. Multi-state graphs fail that rendering step; the call can continue with fallback instructions. A connected call is therefore not proof that the configured workflow loaded. The provider selects from the authorized tools without the platform-managed navigator. Operator takeover and session event injection are not supported on this path, and conversation-scoped integration approval is not a realtime voice gate. Confirm runtime support before reusing a workflow or control.
When a forward_call is emitted by a response that the caller barged in on, the realtime turn owner can still dispatch that transfer if its arguments arrived as a complete JSON object, treating the transfer as if it had landed just before the barge-in. Other tool calls from an interrupted response - including reads, writes, and end_call - are not executed and return a not-executed result instead, and a superseded tool result is not spoken after the caller interrupts. A transfer whose arguments are cut off or empty is also not executed. Separately, caller speech while a transfer is still playing its announcement cancels the transfer. This behavior is specific to the gpt_realtime runtime.
State Types
ActionState
Produces user-facing responses, eligible tool calls, and exit-condition evaluation
Yes (engagement and navigation as needed)
DecisionState
Selects among authored exits without producing a user-facing turn
Yes (navigation only)
AnnotationState
Injects authored internal guidance and advances to its fixed next state
No additional model call
DataCollectionState
Collects declared fields across turns, with turn limits and optional Surface fallback
Yes (engageable)
Per-Turn Flow
The navigator handles multi-state traversal automatically. Decision and annotation states can resolve without user interaction before landing on an action state for the engage LLM.
Navigator resilience: structured output validation retries up to 3 total attempts. When all retries are exhausted, the standard runtime can fall back to the first valid action or exit. Filler text from an earlier successful attempt is preserved across retries (first-wins), which can reduce quiet gaps but does not guarantee continuous audio during recovery.
Action State Extensions
Action states support three optional extension fields for asynchronous workflows:
wait_for
string
Pause navigation until an async condition clears. Values: surface_submission, human_approval.
channel_overrides
object
Per-channel overrides keyed by channel kind (voice, sms). Each override can set objective, action_guidelines, and progress (a progress hint for tool waits in that state).
surface_spec_template
object
Surface spec auto-created on state entry. Uses the Surfaces field schema. The entity ID defaults to the session's primary entity.
Wait conditions: when the navigator returns a waiting_for value, the engine skips context graph navigation on subsequent turns. The engage prompt includes a WAITING_FOR_CONDITION block that constrains the agent to empathetic small-talk until the condition clears. For voice sessions, clearance comes via the real-time event stream. For text sessions, the session blocks on a dedicated event listener.
Channel overrides: the channel_kind is set on the engine session (voice for calls, sms for text sessions). Prompt rendering merges the channel-specific objective and guidelines into the engage prompt. For voice, the override can also carry a progress hint that shapes how the agent covers tool waits in that state (see Tool-Wait Progress Hints).
Surface templates: on state entry, if the new state has a surface_spec_template, the engine requests a surface through the Platform API and tracks the surface_id when creation succeeds. This makes surface creation configuration-driven rather than dependent on a model-selected tool call.
Terminal State & Auto-Hangup
When the standard runtime reaches the Context Graph's configured terminal state, it can speak a final response and schedule an automatic hangup:
Navigator lands on terminal state →
is_terminal = true.Agent speaks the goodbye response.
Waits for TTS to finish plus a grace period (audio buffer flush).
Terminates the call via telephony API.
Silence detection: when the caller goes silent, the silence monitor fires check-ins at increasing intervals (10s → 20s → 40s). After 3 unanswered check-ins, the agent says a brief goodbye and auto-disconnects.
Session shutdown behavior: the standard shutdown handlers stop both the session and audio speaker for hangup, STT failure, WebSocket disconnect, and terminal-state paths. Clients should rely on the reported terminal outcome rather than assuming cleanup timing.
World Model Integration
The voice agent connects to the workspace's world model through three data channels. See Storage and Projection Boundaries for the distinction between durable events and asynchronously projected entity state.
Channel 1: Ambient (Pushed)
Supported runtimes can inject selected ambient data without an explicit agent lookup. Patient fields load only after caller verification. Available fields depend on verification, projected state, service configuration, and successful prompt construction, and can include:
Patient demographics: name, DOB, MRN, phone, email, address.
Clinical context: active conditions, medications, allergies (filtered to text-only for LLM consumption).
Upcoming appointments: with patient entity references for cross-referencing.
Insurance coverage: active plans and subscriber info.
Location context: clinic details, available appointment types, hours (resolved from the inbound phone number).
Design principle: ambient over queried. If the LLM will almost certainly need this data, push it into context. Don't make it ask. A voice agent that already has the patient's insurance in context doesn't need to dispatch a tool call to look it up.
Caller Verification
A caller who matches a record by phone number, or a session opened with an entity_id, starts unverified. The agent is told a record may match, but the record's demographics and clinical context stay out of its context. They load after the caller's stated full name and date of birth match exactly one candidate record through the world_confirm_patient_identity tool. A failed attempt does not say which field was wrong, and repeated failures lock verification for the rest of the session. Text sessions restore the verified or locked state from the conversation history on each turn.
Add world_confirm_patient_identity to the Context Graph state that confirms identity, and have that state ask for the caller's full name and date of birth. Until a graph offers the tool, the agent works without the record. Sessions anchored to an authenticated operator skip this check.
Channel 2: Queried (Pulled)
Data that can't be ambient because the search space is too large. The agent calls built-in clinical tools to retrieve specific information.
Key simplification: standard queried tools return human-readable results rather than database internals. Slot search can return doctor names and times while the runtime carries the selected scheduling identifiers into a later booking request, so raw template IDs and slot UUIDs do not need to be exposed to the LLM.
Channel 3: Extracted (Captured)
Supported voice runtimes can extract structured observations such as insurance details, contact information, and preferences from conversation evidence. When extraction and event publication succeed, those observations enter the world model without a separate conversational tool call. Extraction is model-generated, can be incomplete, and is not enabled identically on every runtime.
Extracted data uses a source-appropriate confidence class below authoritative records. Explicit write tools remain the controlled path for supported high-stakes mutations. Extraction complements those tools; it does not prove that every mentioned fact was captured or externally delivered.
Confidence-Gated Writes
World-model observations written by the voice agent during calls carry a confidence level: a source-class ranking, not a calibrated probability of truth. Confidence is one input to the policies that govern external synchronization.
0.0
Rejected
Human-rejected or system-invalidated
0.3
Agent raw
Raw voice agent inference (unverified)
0.5
Self report
Patient-confirmed or patient-submitted data
0.7
Verified
Verified or EHR-ingested data
0.8
EHR trusted
Selected trusted clinical data
0.9
High
Operator-verified or high-quality source
0.95
Human approved
Named registry class; not assigned by proposal review
1.0
Authoritative
Manual entry or authoritative API
Confirmation-gated confidence: the agent chooses a confirmation level per write based on how the information surfaced in conversation - confirmed (the caller explicitly confirmed the fact, written at self-report confidence), mentioned (mentioned in passing, written at agent-raw confidence), or inferred (inferred from context, written below agent-raw confidence).
Outbound sync gate: unattended connector delivery was retired with the connector runner on October 10, 2026, so these observations stay in the world model and are not delivered to external systems by a connector. Confidence remains available as source evidence for workflows that act on it.
External write proposals: the connector workflow that staged and delivered proposals was retired with the connector runner, so new proposals are not staged and approved proposals are not delivered. See Review Queue. This is separate from an integration's conversation-scoped approval_policy, which applies to supported text conversations and is not a voice-call gate.
Patient Safety Isolation
Supported entity-anchored write tools enforce a write scope for the session and reject a target outside the resolved patient anchor. This is an authorization guard, not proof that the selected field, value, or patient anchor is clinically correct. Write tools also have in-flight deduplication: if the model re-calls the same write tool with identical parameters while the first call is still executing, the concurrent duplicate is suppressed. Once the first execution finishes, the same parameters can execute again; this guard is not general request idempotency or an exactly-once guarantee.
Emotional Adaptation
When acoustic data is available, the rolling valence/arousal state can influence prompt steering, TTS tone, filler behavior, and empathy pacing. These effects are best-effort and depend on the active runtime and speech provider; workspace and service settings can override or disable them.
Valence-Arousal Model
The emotion engine returns valence and arousal for each acoustic segment. The runtime aggregates the latest four segments and derives an internal label from that two-dimensional state:
High-arousal negative
De-escalate
calm when supported
Direct, concise, acknowledge frustration, skip pleasantries
Low-arousal negative
Comfort
sympathetic when supported
Warm, patient, gentle language, give extra space
High-arousal positive
Match energy
enthusiastic when supported
Keep momentum and match positive energy
Low-arousal / neutral
Stay steady
content when supported
Conversational and grounded
Voice Tone Selection
The standard voice-context path chooses a TTS tone in this order:
With at least two acoustic segments and aggregate signal strength of at least
0.25, map the rolling internal label to a supported TTS tone.If no new mapped tone is available, reuse the previous mapped tone as momentum.
If neither produces a tone and the current action matches a configured sensitive topic, select
sympathetic.If workspace
toneis configured, it explicitly overrides the computed tone.
Tone momentum prevents a weak or unmapped segment from snapping the voice back to a default:
The turn-level compound resolver can also attach labels derived from acoustic scores, sentiment/toxicity, barge-in and silence behavior, empathy tier, and conversation context. Those compound labels are stored with turn/call intelligence; they are not a separate vocal-burst model or an independent first-priority TTS signal.
Filler Speech
Fillers can cover processing latency. The system uses principle-based guidance rather than relying only on hardcoded phrase lists, generating contextually appropriate fillers from available emotional context, the current action, and expected latency.
Three-layer filler generation:
Latency adaptation
When filler generation runs
Filler length can reflect expected processing time (typically 2-4 words)
Emotional attunement
When emotion data available
Emotional register matches the caller's state. Not specific phrases, but principles like "gentle and reassuring" or "a verbal hand on the shoulder".
Action context
When an action description is available
The action description can steer a filler toward what the agent is about to do
Per-action filler hints: context graph actions can include optional PM-configured filler suggestions. In non-deterministic mode, the LLM sees them as suggestions alongside emotional and contextual guidance, not commands. Explicit deterministic phrases follow the separate contract below.
Suppression rule: when valence < -0.2 AND arousal > 0.4 AND emotion is NOT Anxiety/Fear/Distress → fillers disabled entirely. Frustrated callers don't want acknowledgments, they want the answer. Exception: anxious callers still receive reassuring fillers, because anxiety benefits from reassurance while frustration does not.
Tool-Wait Progress Hints
When a tool is running, the agent narrates the wait based on a ProgressHint declared on the state's channel override and/or on the individual tool binding. The hint describes the shape of the wait, not a phrase list - the engine composes the actual utterance from tool semantics, turn emotion, and conversation context.
mode
"auto" | "silent" | "backchannel" | "verbal"
How the agent covers the wait. auto lets the engine decide from class and latency; silent drops filler text; backchannel emits a single token ("Mm."); verbal produces a contextual filler.
progress_class
"lookup" | "write" | "external_call" | "compute" | "multi_step" | null
Semantic category of the work the tool is doing. Drives the templated language the engine picks for retries.
expected_latency_ms
integer (0-60000) | null
How long the tool is expected to take. Used to choose filler length and pacing.
trigger_delay_ms
integer (0-30000) | null
How long to wait after a tool starts before playing the first filler. Set to 0 for immediate playback. When unset, the delay is calculated from expected_latency_ms; when unset and deterministic phrases are configured, the first filler fires immediately.
interval_ms
integer (100-30000) | null
Milliseconds between subsequent fillers. Overrides the global progress interval for this hint.
deterministic
boolean (default false)
Whether phrases play verbatim. true: scripted - the engine plays the phrases exactly as written, in order. false: the engine picks - phrases are hints.
phrases
array of strings (1-10 items) | null
Ordered filler phrases, each at most 30 words. With deterministic: true, phrases play verbatim in order, cycling back to the start for attempts beyond the list length. With deterministic: false (the default), phrases are used as hints for early attempts - attempt N plays phrases[N-1] while within the list, and later attempts fall back to class templates, vocabulary, and retry fillers. backchannel mode preempts phrases. When set, phrases supersedes custom_phrase.
custom_phrase
string (max 500 chars) | null
Operator-authored filler utterance for the first filler attempt. In auto mode: requires expected_latency_ms >= 4000 and progress_class to be set. In verbal mode: always honored (word cap still applies). Ignored when phrases is set. Subsequent attempts use class templates.
Placement: progress can be set on channel_overrides[channel].progress (state-level default for that channel) and on each entry of action_tool_call_specs or exit_condition_tool_call_specs (per tool). Per-tool fields merge field-wise over the channel override, so a state can declare the default wait shape once and individual tools only override fields that actually differ.
Retry narration: when a tool retries, the engine uses deterministic attempt-aware templates keyed on progress_class - acknowledgement on attempt 1, brief apology on attempt 2, a "still working" update from attempt 3 - so retry audio stays off the LLM hot path and latency stays bounded. With deterministic: true, the engine skips templates and plays the scripted phrases in order, cycling through the list; with deterministic: false, phrases cover early attempts and templates take over for attempts beyond the list.
Deterministic vs non-deterministic: scripted flows (demo flows, regulated disclosures) that need exact control over what the agent says during tool waits must set phrases with deterministic: true. Deterministic fillers require the tool binding's execution: "background" - write-time validation rejects a blocking binding combined with deterministic: true. Use custom_phrase + progress_class when you want to steer the first filler but let the engine handle retries with contextual templates.
Call Phase Escalation
The system automatically increases urgency as calls extend with negative sentiment:
Early
< 5 min
Any
Standard emotional adaptation
Mid
5-10 min
Trend deteriorating
"Focus on resolution speed. Shorten responses."
Late
≥ 10 min
Negative valence
URGENCY. "Prioritize resolution. Be maximally concise. Escalate if unable to resolve."
Proactive Intelligence
The system detects emotionally sensitive topics from the current context graph action before the caller shows distress:
sensitive_topics is configurable via voice settings. Falls back to healthcare defaults: test results, diagnosis, billing, payment, insurance, denial, emergency, referral, specialist, surgery, procedure, medication.
This fallback applies only when acoustic selection and tone momentum produced no tone. An explicitly configured workspace tone then overrides the result.
Coherence Detection
When what the caller says doesn't match how they sound (coherence < 0.4), the system shifts its steering: "The caller's words suggest X but voice sounds Y. Trust the vocal tone over the words, respond to how they sound, not what they claim."
This is injected into the system prompt without the agent ever explicitly mentioning the discrepancy to the caller.
Control Plane ↔ Adaptation
How each workspace voice setting interacts with the automatic emotion adaptation system:
tone
Explicit workspace TTS tone
Replaces acoustic, momentum, and sensitive-topic selection when configured
Whenever the setting is present
speed
Base speech rate
Can be reduced by supported empathy pacing logic
When an empathy baseline is active; configured service floors apply
volume
Base volume
No adaptive override in the standard voice-context path
Provider support still applies
voice_id
Voice persona
Per-agent voice config overrides
Agent version has voice config set
keyterms
Domain vocabulary for STT boost
Merged with service keyterms and deduplicated
Subject to selected STT provider support
sensitive_topics
Topics for proactive tone softening
Falls back to healthcare defaults
Preemptive, not reactive
post_call_analysis_enabled
Retained workspace setting
None
Does not establish an active automatic scoring pipeline; use the documented evaluation paths
Key principle: automatic acoustic selection is best-effort. Explicit workspace and service settings remain authoritative where the active runtime supports them.
Graceful Degradation
The standard runtime implements the best-effort behaviors below. Optional emotion and post-call failures can leave those features unavailable without ending an otherwise healthy call, but failures in required telephony, speech, model, or context-engine dependencies can interrupt or end the session.
Emotion connection
Auth error, timeout, unavailable service
Session continues without emotion detection
No acoustic adaptation; configured defaults remain
Emotion stream
Repeated processing error or connection close
Disable for the remainder of the session after the configured consecutive-failure limit
No automatic same-call recovery guarantee
Emotion detection
Insufficient data (< 2 segments)
No emotional steering, default fillers
First few seconds may lack adaptation
Transcript analysis
No sentiment/toxicity result
Continue with acoustic and behavioral inputs
Compound/coherence evidence is reduced
Voice settings
Parse error
Defaults (filler on, emotion on)
Baseline experience still works
Post-call analysis
Any error
Logged, not raised (fire-and-forget)
Quality data missing, call unaffected
TTS connection
Close/error mid-stream
A later turn can attempt reconnection
Audio gap; session can fail if recovery does not succeed
STT connection
Connection loss
Reconnection is attempted where supported
Transcription gap; session can fail after recovery is exhausted
Context graph engine
Backend unavailable
Any configured runtime fallback is attempted
The response or session can fail when no fallback is available
Tool Execution
Tool bindings can execute inline or in the background during calls. Background bindings let the agent acknowledge the action and continue speaking while results arrive through the configured continuation-delivery mode.
Execution Model
Tool calls are routed by tool family, and each state-level tool binding declares how the call executes and how its result is delivered:
World tools
world_*
Built-in reads/writes against the world model (patient, appointment, insurance, typed entity lookups)
Skills
(by slug)
LLM-backed companion micro-agents: multi-turn reasoning with the skill's configured integration and static tools (see Skills)
System tools
-
forward_call, complete_transfer, resume_patient, get_tool_call_status, cancel_background_task; handled inline by the turn controller
Blocking vs background execution: each state x tool binding declares an execution axis. Blocking tools are awaited inline. Background tools start without being awaited, so the agent can acknowledge and keep talking. When background execution and delivery succeed, the result can arrive as a continuation turn. The binding's delivery axis controls how that result lands: interrupt (a continuation turn that can cut current speech) or queue (folded in on the next user turn). A separate failure_delivery axis can route reported failures differently from successes.
State gating: in the standard HSM runtime, tool calls are validated against the tools offered for the current context-graph state, and an unavailable call is rejected. Native and realtime runtimes can also attach shared platform or system tools globally to each phase. In every runtime, authorization, explicit tool enablement, and patient/write scope remain the security boundary; state membership alone is not one.
In-flight write deduplication: an identical write call is suppressed only while the first call is still running. It does not deduplicate a later retry after completion and does not provide request-level exactly-once semantics.
Built-in Clinical Tools
The Platform API publishes a catalog of built-in world_* tools. A supported runtime can expose eligible catalog entries after applying service configuration, Context Graph bindings, authorization, and entity scope. Catalog presence is therefore discovery information, not proof that a tool is callable in every session.
Read tools:
world_patient_lookup
Search patients by DOB, name, phone, or MRN (DOB preferred for accuracy)
world_confirm_patient_identity
Verify the caller: compares the stated full name and date of birth with the session's candidate records and loads the record when exactly one matches
world_persons_lookup
Generic person-entity lookup with server-side filters
world_slot_search
Available appointment slots by location and date. Returns human-readable times and doctor names while retaining the identifiers needed for a later scheduling call
world_appointment_lookup
Patient's existing appointments, with references for cancel/confirm/reschedule
world_payer_search
Look up insurance payers/carriers
world_reference_data_lookup
Look up reference data (locations, appointment types, providers)
world_get_medications
Patient's medications and prescription references
world_get_encounters
Patient's encounter history
world_queue_lookup
Read the queue of pending outbound call tasks
world_conditions_lookup, world_allergies_lookup, world_observations_lookup, world_medication_statements_lookup, world_clinical_notes_lookup, world_family_history_lookup, world_questionnaire_responses_lookup, world_intake_uploads_lookup, world_care_plans_lookup, world_goals_lookup, world_tasks_lookup, world_service_requests_lookup, world_consents_lookup, world_diagnostic_reports_lookup
Typed read tools over projected entity shapes
world_conversations_lookup, world_calls_lookup
Conversation and call record lookups
world_state_resolve
Batch entity-ID to current world-state resolver
Write tools:
world_patient_create
Create a patient after the supported duplicate checks; ambiguous matches still require reconciliation
world_patient_update
Update contact info (phone, email, address); requires an entity reference
world_save_patient
Create or update after the supported duplicate check; accepts natural field names and flexible date formats
world_schedule_appointment
Book from a supported slot-search result and resolve its scheduling identifiers
world_cancel_appointment
Cancel by appointment reference
world_confirm_appointment
Confirm a booked appointment
world_reschedule_appointment
Move an appointment to a new slot
world_insurance_create
Create an insurance event that an eligible configured connector can consider for delivery
world_refill_prescription
Submit a refill request for a specific prescription
world_log_call, world_log_triage
Record call and triage outcomes
world_ticket_create
Create a support-ticket event for a supported downstream workflow
World write results carry source and confidence metadata. Only eligible events with a supported connector mapping are considered for external delivery, where additional source, confidence, review, and destination policy applies. Entity-anchored tools also apply the available write scope; catalog membership alone is not a delivery or safety guarantee.
Call Forwarding
The service's voice_config.forwarding controls whether the runtime exposes forward_call. A null or omitted block disables forwarding. When configured, forward_to selects a fixed E.164 destination. When it is null, the service uses dynamic mode, and forward_call offers only the numbers in approved_destinations, each with its label. Any other number is refused without dialing, and a dynamic service with no approved destinations offers no forward_call. warm_transfer must explicitly select warm or cold transfer. There is no workspace destination fallback.
Confirm destination approval in the provisioned channel as well. Channel Manager enforces a per-use-case forwarding allowlist, allowed_forwarding_numbers, separately from service configuration. See Forwarding destinations.
State policy can further restrict forwarding. Authored instructions to transfer do not prove that the tool was invoked or that the destination answered.
On the conference warm-transfer path, the caller stays on hold while the destination is dialed and briefed. The briefing waits for a human-answer classification; voicemail, fax, failure, or timeout returns the call to the agent. After briefing, the caller must be taken off hold successfully before the agent is removed. If that step fails, the runtime retains the agent. Provider answer classification is fallible.
See Voice Configuration for authoring and Operators and Escalation for the operational sequence.
Safety & Escalation
Safety behavior comes from the agent definition, Context Graph guidance and turn policy, eligible tools, and operator workflows. Model reasoning can still miss or misclassify a concern, so these controls require scenario testing and a human fallback. Supported escalation paths include:
Call forwarding: the agent invokes the built-in
forward_calltool to transfer the caller to a human.Operator join: a human operator joins in listen or takeover mode. Takeover suspends agent speech while the operator drives; listen mode does not.
The Console's Takeover navigation, operator pages, and Runs Live tab are retired. Use All runs and its Live status filter to inspect available active records. Intervention requires a separate integration with the retained APIs and a supported runtime. See Operators for those resources and Safety & Monitoring for the safety model, audit trail, and compliance reporting.
Observer WebSocket
Monitor active calls in real time via a WebSocket connection:
Requires a valid workspace API key, sent only in the Sec-WebSocket-Protocol header as auth, <token>. The server accepts the auth subprotocol. A ?token= query parameter is not read, and a connection without the subprotocol closes with 4001. The Platform SDK's observers.subscribe sends the credential this way. Any observer can monitor any active call in the workspace, regardless of which server handles the call.
Late-join replay: observers connecting mid-call receive a buffered replay of recent events before transitioning to the live stream. Events carry monotonic sequence numbers for ordering.
Observer Event Types
The observer stream emits session lifecycle, transcript, state transition, tool, timing, emotion, and call control events. The full catalog with field lists and PHI annotations lives on the observer events page:
Session Event Injection
External systems can inject events into supported platform-managed voice sessions. The separate realtime-provider runtime does not process these injections. When delivery and response generation succeed, the platform-managed runtime handles the event without Context Graph navigation and can speak a response. Injection acceptance does not guarantee response timing, exact wording, or that a response will be produced.
Injection Paths
Use the Platform session injection operation below. The Voice Agent route behind it (POST /agent/sessions/{call_sid}/event) accepts only the platform's own service credential: since October 6, 2026, a workspace API key gets 401 and an identity user token gets 403, so do not call it directly. It is not published in the Platform OpenAPI document. The /test-call WebSocket also accepts control frames for developer playground testing.
Platform Session Injection
Use workspace authentication for frontend or third-party injection.
Inject an external event or guidance message into an active voice session. The agent incorporates it into its next response. Use event_type='guidance' for operator coaching, 'external_event' for system notifications.
API key issued via POST /v1/{workspace_id}/api-keys. Pass the returned api_key value as a Bearer token.
Systemexternal_event (factual info) or guidance (instructional)
external_eventPossible values: Successful Response
Session not found.
Validation Error
Voice agent service unavailable.
POST /v1/{workspace_id}/sessions/{call_sid}/inject HTTP/1.1
Host: platform.api-us.concurrence.com
Authorization: Bearer YOUR_SECRET_TOKEN
Content-Type: application/json
Accept: */*
Content-Length: 66
{
"message": "text",
"sender": "System",
"event_type": "external_event"
}{
"status": "delivered",
"call_sid": "text"
}Operator Guidance
Use operator-scoped guidance when the caller should be identified and checked for Operator:Update permission.
Inject operator guidance into an active call. The agent incorporates it into its next response.
API key issued via POST /v1/{workspace_id}/api-keys. Pass the returned api_key value as a Bearer token.
Successful Response
Delivery status
Call SID the guidance was sent to
Operator or active call not found.
Validation Error
Voice agent service unavailable.
POST /v1/{workspace_id}/operators/{operator_id}/send-guidance HTTP/1.1
Host: platform.api-us.concurrence.com
Authorization: Bearer YOUR_SECRET_TOKEN
Content-Type: application/json
Accept: */*
Content-Length: 36
{
"call_sid": "text",
"message": "text"
}{
"status": "delivered",
"call_sid": "text"
}Injected Event Types
external_event
Queues behind current speech. Cancels silence monitor.
"Appointment confirmed for 2pm tomorrow"
guidance
Interrupts current speech and cancels silence monitor.
"Ask for their insurance ID before confirming"
The distinction matters: external events carry factual information that can wait for the agent to finish speaking, while guidance is prioritized as an instruction and can interrupt current speech. Delivery can still be delayed or fail, and the model is not guaranteed to follow the instruction verbatim.
Request Format
The Voice Agent route receives this body from the Platform API. It is shown to explain the fields; send your request to the Platform session injection operation above.
The workspace_id field is required and must match the workspace of the call being injected into; a mismatch is rejected with 403. The event_type field accepts "external_event" or "guidance" (default "external_event"). The message field is limited to 5,000 characters. The sender field (max 200 characters, default "System") can be retained with the injected event when processing and call persistence succeed. Returns 404 if no active session exists for the call_sid.
Response
The endpoint returns delivery status indicating whether the event was received by the session:
A status of "queued_no_subscriber" indicates the event was published but no active session was listening. This can happen during a brief window when a session is initializing or if the call has already ended. A status of "deduplicated" indicates that an identical injection (same message, sender, and event type) for the same call was suppressed within a 30-second window. This reduces duplicate processing for ordinary retries but is not an end-to-end delivery or exactly-once guarantee.
Delivery Resilience
Injection can cross runtime instances to reach the active call. Inspect the returned status: delivered confirms that a session listener received the publication, not that the model followed it or that a caller heard a response. Runtime interruptions can delay or prevent later processing.
Active Sessions
List currently active sessions via the platform API:
Real-time list of active voice sessions from Valkey (cross-pod).
API key issued via POST /v1/{workspace_id}/api-keys. Pass the returned api_key value as a Bearer token.
Successful Response
""""GET /v1/{workspace_id}/sessions/active HTTP/1.1
Host: platform.api-us.concurrence.com
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
Successful Response
[
{
"call_sid": "text",
"service_id": "123e4567-e89b-12d3-a456-426614174000",
"workspace_id": "123e4567-e89b-12d3-a456-426614174000",
"caller_id": "text",
"direction": "text",
"duration_seconds": 1,
"current_state": "",
"scenario": ""
}
]Returns a best-effort real-time list of active sessions with call metadata. A live session can appear late or be temporarily absent, so use the canonical Runs surface for durable inventory.
A trusted backend can use an entry's call_sid to set verified context on that call. Match exactly one entry before doing so; see Set Context on a Live Voice Call.
WebSocket Control Channel
Test calls and direct streams accept text-frame control messages for injection and session control:
Test-Call Scenarios
The /test-call WebSocket endpoint supports scenario-based testing:
scenario
inbound
inbound (agent greets first), outbound (task context greeting), silent (no greeting)
caller_id
empty string
Simulated caller phone number. Test calls are tagged with source playground independently of caller_id.
outbound_task_entity_id
-
Entity ID for outbound task context (required for outbound scenario)
system_prompt
-
Freeform prompt override (takes precedence over scenario-derived prompts)
Send the credential in Sec-WebSocket-Protocol, as on the observer socket. A ?token= query parameter is not read, and a connection without the auth subprotocol closes with 4001. The audio encoding must be mulaw; any other encoding closes with 4008. A workspace can hold up to 5 test calls at once, and starting another returns 429 until one ends.
A test call always runs in the credential's workspace. A workspace_id query parameter that names a different workspace closes the socket with code 4003 before the session starts; omitting it uses the credential's workspace.
Call Record & Persistence
Realtime-provider phone and playground calls attempt to open a conversation record when the call starts and finalize the same record when it ends. Finalization records the end time, duration, completion reason, final state, and a turn count that accounts for provider-reported turns and captured transcript/tool entries. Call startup can continue after a persistence failure; failed finalization can leave a record non-terminal.
For gpt_realtime, completed user and agent transcripts and completed tool calls are retained in the provider's conversation-item order, rather than the order in which asynchronous transcription finishes. Tool entries include the name, call identifier, arguments, full output, duration, and success/failure. An observer's shortened tool-output preview is not the full retained result.
History is written incrementally during the call. An unfinished earlier item can delay later entries; at teardown, unfinished items are skipped and completed unsaved entries are retried. Missing provider completions or failed writes can still leave gaps. A retained lifecycle record, turn count, or live observer event does not establish complete history, recordings, or quality analysis. This transcript and tool behavior is specific to the gpt_realtime runtime.
The platform-managed pipeline normally persists the call record at completion and can also retain intermediate checkpoints. Use Runs for the relationship between live visibility and retained history. Depending on the runtime, enabled capture, and successful persistence, the record can include:
Turns: each turn carries a timing model with four field groups (all fields in milliseconds):
User speech boundaries:
user_speech_start_ms,user_speech_end_ms(when the caller's speech started and ended).Processing breakdown:
engine_ms,nav_ms,render_ms,audio_ttfb_ms(reasoning-engine, navigation, render, and audio time-to-first-byte latency).Agent speech boundaries:
agent_speech_start_ms,agent_speech_end_ms(when agent audio playback started and ended).Audio window:
audio_window_start_ms,audio_window_end_ms(the turn's bounding audio interval).
Tool calls: name, input, output, duration, success/failure.
State transitions: recorded context graph navigation events.
Emotional summary: see below.
Escalation history: recorded escalation lifecycle events when an operator joined.
Config snapshot: version set, agent version, context graph version used.
Calls API
Voice Run Inventory
Paginated, newest-first list of runs for the workspace behind the unified Run contract. Federates framework runs (Delta world.runs MV) and conversation runs (Lakebase world.conversations) at read time. Filter by kind (conversation / framework), channel (voice/sms/whatsapp/email/web/imessage — conversation runs only), status (live expands to running + paused), service_id (service UUIDs), and search (entity name, caller ID or phone number). Repeated filter values are ORed within each axis; axes combine with AND. Service filtering applies to conversation and live voice runs before pagination. Historical framework runs have no stored service attribution and are excluded when service_id is set. continuation_token is an opaque page cursor.
API key issued via POST /v1/{workspace_id}/api-keys. Pass the returned api_key value as a Bearer token.
[]50Filter conversation and live voice runs by up to 100 service IDs. Historical framework runs have no stored service attribution and are excluded. Repeated values are ORed; combines with other filters using AND.
[]Case-insensitive substring match on a conversation run's entity name, caller ID or phone number, applied before pagination. Framework runs carry none of these and are excluded when search is set. Combines with other filters using AND.
Successful Response
Validation Error
GET /v1/{workspace_id}/runs HTTP/1.1
Host: platform.api-us.concurrence.com
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
{
"items": [
{
"run_id": "123e4567-e89b-12d3-a456-426614174000",
"workspace_id": "123e4567-e89b-12d3-a456-426614174000",
"kind": "conversation",
"status": "running",
"channel": "voice",
"framework": "claude-agent-sdk",
"entity_id": "123e4567-e89b-12d3-a456-426614174000",
"service_id": "123e4567-e89b-12d3-a456-426614174000",
"started_at": "2026-01-01T00:00:00.000Z",
"ended_at": "2026-01-01T00:00:00.000Z",
"entity_name": "text",
"service_name": "text",
"caller_id": "text",
"phone_number": "text",
"direction": "text",
"turn_count": 1,
"completion_reason": "text",
"source_call_sid": "text",
"source_conversation_id": "123e4567-e89b-12d3-a456-426614174000",
"source_framework_run_id": "123e4567-e89b-12d3-a456-426614174000",
"takeover": {
"eligible": true,
"mode_options": [
"listen"
],
"reason": "text"
}
}
],
"has_more": true,
"continuation_token": null
}/runs is the canonical workspace inventory for current clients. Filter by live status, conversation kind, and voice channel for active calls; omit the status filter for history. Active-call visibility is best-effort, so clients should tolerate a live call appearing late or being temporarily absent.
Compatibility Voice Listing
Query call entities with date range, status, and duration filters. Enriched with quality_score and final_state from call_intelligence.
API key issued via POST /v1/{workspace_id}/api-keys. Pass the returned api_key value as a Bearer token.
Start date (inclusive)
End date (inclusive)
Filter by call status
Filter by direction (inbound, outbound, playground, simulated)
Minimum duration in seconds
Maximum duration in seconds
Filter by service ID
Include simulated sessions
false200Alias for offset (pagination cursor)
Successful Response
Validation Error
GET /v1/{workspace_id}/calls HTTP/1.1
Host: platform.api-us.concurrence.com
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
{
"items": [
{
"entity_id": "123e4567-e89b-12d3-a456-426614174000",
"call_sid": "text",
"direction": "inbound",
"phone_number": "text",
"caller_id": "text",
"started_at": "2026-01-01T00:00:00.000Z",
"duration_seconds": 1,
"status": "initiated",
"escalation_status": "text",
"turns": 1,
"has_recording": true,
"quality_score": 1,
"final_state": "text",
"completion_reason": "completed",
"service_id": "123e4567-e89b-12d3-a456-426614174000",
"source": "real",
"run_id": "text",
"parent_session_id": "text",
"fork_turn_index": 1
}
],
"total": 1,
"has_more": true,
"continuation_token": 1
}This voice-specific compatibility listing is deprecated and returns Deprecation: true and Sunset: 2026-07-01 response headers. Migrate inventory reads to /runs.
Compatibility Call Detail
Full call detail including turns, escalation state, safety state, and recording info. Proxied from voice-agent, with simulation session fallback.
API key issued via POST /v1/{workspace_id}/api-keys. Pass the returned api_key value as a Bearer token.
Successful Response
Full call detail — the canonical response for GET /calls/{call_id}.
This is the single source of truth for the call detail shape. The OpenAPI spec is generated from this model. The SDK TypeScript types are generated from the spec. The developer-console consumes SDK types directly.
completed0[][]falseCall not found
Validation Error
GET /v1/{workspace_id}/calls/{call_id} HTTP/1.1
Host: platform.api-us.concurrence.com
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
{
"id": "123e4567-e89b-12d3-a456-426614174000",
"call_sid": "text",
"workspace_id": "123e4567-e89b-12d3-a456-426614174000",
"service_id": "123e4567-e89b-12d3-a456-426614174000",
"caller_id": "text",
"phone_number": "text",
"direction": "inbound",
"status": "completed",
"error": "text",
"call_start_time": "2026-01-01T00:00:00.000Z",
"call_end_time": "2026-01-01T00:00:00.000Z",
"call_duration_seconds": 0,
"media_start_time": "2026-01-01T00:00:00.000Z",
"config": {
"agent_id": "123e4567-e89b-12d3-a456-426614174000",
"agent_version": 1,
"agent_name": "text",
"context_graph_id": "123e4567-e89b-12d3-a456-426614174000",
"context_graph_version": 1,
"initial_state": "text",
"gpt_audio_model": "text",
"navigation_model": "text"
},
"turns": [
{
"id": "123e4567-e89b-12d3-a456-426614174000",
"trigger": "",
"user_transcript": "text",
"agent_transcript": "text",
"agent_action": "text",
"interrupted": false,
"incomplete": false,
"filler_type": "text",
"speaker_id": "text",
"speaker_role": "text",
"emotion_label": "text",
"emotion_valence": 1,
"compound_emotions": [
{
"ANY_ADDITIONAL_PROPERTY": "anything"
}
],
"stt_confidence": 1,
"eot_confidence": 1,
"state": "",
"tool_calls": [
{
"tool_name": "text",
"call_id": "text",
"input": {
"ANY_ADDITIONAL_PROPERTY": "anything"
},
"output": "text",
"output_truncated": false,
"output_original_length": 1,
"output_truncated_to": 1,
"duration_ms": 1,
"succeeded": true,
"error_message": "text",
"integration_name": "text",
"endpoint_name": "text",
"protocol": "text"
}
],
"state_transitions": [
{
"type": "text",
"previous_state": "",
"next_state": "",
"annotation": "text",
"nav_ms": 1,
"from_state": "text",
"to_state": "text"
}
],
"inner_thoughts": [
"text"
],
"timestamp": "2026-01-01T00:00:00.000Z",
"engine_ms": 1,
"user_speech_start_ms": 1,
"user_speech_end_ms": 1,
"agent_speech_start_ms": 1,
"agent_speech_end_ms": 1,
"audio_window_start_ms": 1,
"audio_window_end_ms": 1,
"nav_ms": 1,
"render_ms": 1,
"audio_ttfb_ms": 1,
"signal_kind": "text",
"signal_source": "text",
"state_before": "text",
"state_after": "text",
"primary_effect_kind": "text",
"tool_name": "text",
"tool_succeeded": true,
"tool_ms": 1,
"modality": "text",
"empathy_tier": 1,
"decision_bearing": true
}
],
"states_visited": [
"text"
],
"final_state": "text",
"triggered_behaviors": [
"text"
],
"completion_reason": "completed",
"emotional_summary": {
"dominant_emotion": "text",
"average_valence": 1,
"average_arousal": 1,
"peak_negative_valence": 1,
"peak_negative_emotion": "text",
"emotional_shifts": 1,
"final_trend": "text",
"segment_count": 1,
"barge_in_count": 1,
"compound_emotions": [
{
"ANY_ADDITIONAL_PROPERTY": "anything"
}
]
},
"forwarding": {
"forward_to": "text",
"warm_transfer": true,
"forwarded_at": "2026-01-01T00:00:00.000Z"
},
"conference_sid": "text",
"participants": [
{
"participant_id": "text",
"role": "caller",
"display_name": "text",
"phone_number": "text",
"joined_at": "2026-01-01T00:00:00.000Z",
"left_at": "2026-01-01T00:00:00.000Z"
}
],
"barge_in_events": [
{
"type": "barge_in",
"interrupted_text": "text",
"discarded_texts": [
"text"
],
"interrupted_speaker_id": "text",
"interrupting_speaker_id": "text",
"total_barge_ins": 1
}
],
"recording_path": "text",
"twilio_recording_sid": "text",
"twilio_recording_duration": 1,
"has_recording": false,
"timeline": {
"turns": [
{
"turn_index": 1,
"seek_to": 1,
"active_start": 1,
"active_end": 1
}
],
"segments": [
{
"type": "agent_speech",
"lane": "agent",
"track": "agent",
"lane_id": "text",
"order": 1,
"actor": {
"kind": "agent",
"role": "agent",
"label": "text",
"participant_id": "text"
},
"start": 1,
"end": 1,
"label": "text",
"turn_index": 1,
"stt_confidence": 1,
"eot_confidence": 1,
"audio_window_start": 1,
"audio_window_end": 1,
"engine_ms": 1,
"nav_ms": 1,
"render_ms": 1,
"audio_ttfb_ms": 1,
"e2e_ttfb_ms": 1,
"tool_name": "text",
"call_id": "text",
"parent_call_id": "text",
"integration_name": "text",
"endpoint_name": "text",
"protocol": "text",
"succeeded": true,
"duration_ms": 1,
"from_state": "text",
"to_state": "text",
"emotion": "text",
"valence": 1,
"state": "text"
}
],
"duration_seconds": 1,
"timebase": {
"unit": "seconds",
"origin": "media_start",
"start": 0,
"end": 0
},
"lanes": [
{
"id": "text",
"track": "agent",
"label": "text",
"order": 1,
"actor": {
"kind": "agent",
"role": "agent",
"label": "text",
"participant_id": "text"
}
}
],
"total_user_speech_seconds": 0,
"total_agent_speech_seconds": 0,
"total_silence_seconds": 0,
"avg_processing_gap_ms": 1,
"max_processing_gap_ms": 1,
"avg_perceived_latency_ms": 1
},
"call_analysis": {
"ANY_ADDITIONAL_PROPERTY": "anything"
},
"source": "real",
"run_id": "text",
"score": 1,
"score_rationale": "text",
"parent_session_id": "text",
"fork_turn_index": 1,
"escalation": {
"status": "none",
"escalation_id": "text",
"requested_at": "2026-01-01T00:00:00.000Z",
"connected_at": "2026-01-01T00:00:00.000Z",
"completed_at": "2026-01-01T00:00:00.000Z",
"trigger": "text",
"trigger_source": "text",
"concept": "text",
"similarity": 1,
"agent_confidence": 1,
"operator_type": "text",
"regulatory_basis": "text",
"immediate": false,
"risk_score": 1,
"operator_entity_id": "text",
"wait_seconds": 1,
"handle_time_seconds": 1,
"human_segment_turn_count": 1
},
"safety": {
"peak_concern_level": 0,
"concern_turn_count": 0,
"trajectory": "text",
"triage_history": [
{
"ANY_ADDITIONAL_PROPERTY": "anything"
}
],
"concept_matches": [
{
"concept": "text",
"similarity": 1,
"agent_action": "text",
"agent_confidence": 1,
"agent_reasoning": "text",
"at": "2026-01-01T00:00:00.000Z"
}
]
},
"human_segments": [
{
"speaker_role": "text",
"transcript": "text",
"timestamp": "2026-01-01T00:00:00.000Z"
}
],
"audit": {
"event_count": 0,
"latest_event": {
"ANY_ADDITIONAL_PROPERTY": "anything"
}
},
"quality_score": 1,
"conversation_summary": {
"ANY_ADDITIONAL_PROPERTY": "anything"
},
"safety_summary": {
"ANY_ADDITIONAL_PROPERTY": "anything"
}
}Available call-record fields can include turns with timing data, tool calls, state transitions, an emotional summary, escalation history, safety state, and a configuration snapshot.
This compatibility detail route returns the same deprecation and sunset headers. Use /runs/{run_id} for the canonical run record; call-specific subordinate analytics and recording routes remain documented below where no run-scoped replacement is available.
Recordings
Presigned S3 URLs for the inbound (caller) and outbound (agent) recording WAVs.
API key issued via POST /v1/{workspace_id}/api-keys. Pass the returned api_key value as a Bearer token.
Successful Response
Presigned S3 URLs for a call recording's WAV objects.
Twilio call SID
Presigned URL for the inbound (caller) WAV
Presigned URL for the outbound (agent) WAV
Presigned URL for the merged stereo (caller left / agent right) WAV
Validation Error
GET /v1/{workspace_id}/recordings/{call_sid} HTTP/1.1
Host: platform.api-us.concurrence.com
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
{
"call_sid": "text",
"inbound_url": "text",
"outbound_url": "text",
"stereo_url": "text"
}The operation checks workspace access and issues temporary links to stored caller, agent, and stereo audio. It does not check whether the objects exist before issuing the links. A successful response is therefore not proof that recording finalization completed; handle missing objects and expired links when downloading.
Current recordings store separate 8 kHz mu-law mono WAV files and create a decoded PCM stereo WAV during finalization. Read each file's header, especially for older recordings. Older calls may lack the merged stereo artifact even when mono files remain.
The former metadata, URL-suffix, and download-proxy endpoints have been removed. Playback clients fetch the current links and derive metadata or waveforms from the audio. See Call Recordings for retention and availability boundaries.
Outbound Calls
Outbound calls are placed through the Platform API outbound call endpoint, which handles use-case-based caller ID selection, an explicit startup instruction, response reuse for some same-key retries, and best-effort world-model lineage. The patient identifier validates and anchors lineage but is not passed as a dedicated voice-runtime context field:
Text Sessions
Text conversations - REST, WebSocket, SMS, and WhatsApp - can use the same Context Graph definitions and many of the same tool families as voice calls, but transport behavior, runtime tool exposure, turn handling, safety fallbacks, and persistence details differ by channel. The canonical documentation for text transports lives on Conversations: WebSocket connection and close codes, the tool_events default, SSE streaming event shapes, message queuing and coalescing, outbound SMS parameters, and inbound SMS and WhatsApp behavior.
WhatsApp Voice Notes
Run one WhatsApp-style voice-note turn against the service's agent.
Accepts OGG Opus audio (or any format Deepgram's prerecorded endpoint
accepts). Server keys the conversation on (workspace, service, phone_number) and returns the agent's spoken reply as OGG Opus — or WAV
PCM16 when the Accept header prefers audio/wav.
API key issued via POST /v1/{workspace_id}/api-keys. Pass the returned api_key value as a Bearer token.
^\+[1-9]\d{1,14}$Agent spoken reply — OGG Opus by default, WAV PCM16 (16 kHz mono) when the request's Accept header prefers audio/wav
Agent elected silence
Concurrent voice turn in progress for this speaker
Validation Error
POST /v1/{workspace_id}/services/{service_id}/voice-turn HTTP/1.1
Host: platform.api-us.concurrence.com
Authorization: Bearer YOUR_SECRET_TOKEN
Content-Type: multipart/form-data
Accept: */*
Content-Length: 38
{
"audio": "text",
"phone_number": "text"
}No content
This operation transcribes a recorded audio turn, runs the configured reasoning path, and returns synthesized speech to the calling integration. The integration is responsible for delivering that audio to WhatsApp or another destination; the operation does not send a managed WhatsApp reply.
The response defaults to OGG Opus. A client that prefers WAV through content negotiation can receive PCM WAV instead. The embedded contract defines the accepted request, response media types, silence response, and concurrent-turn conflict.
Voice-note sessions use workspace, service, and phone number for continuity. Their state remains separate from managed WhatsApp conversations, which use business-scoped recipient identity and asynchronous text delivery. Voice notes do not provide live barge-in or operator takeover.
Desktop Sessions
Remote desktop automation sessions - screenshots, mouse and keyboard actions, and session lifecycle - are documented on their own page:
Agent Trace & Debugging
Three endpoints provide deep inspection into agent reasoning and call quality.
Execution Trace
Per-turn execution log showing recorded agent decisions. Each turn can include:
action
string
The action taken (speak, navigate, tool call, escalate)
signal_kind
string
What triggered this turn (caller speech, tool result, state transition, barge-in)
effect_kind
string
What the agent did (spoke, invoked tool, navigated, escalated)
tools
array
Tool calls with name, parameters, result, and duration
state
string
Context graph state at this turn
emotion
object
Detected caller emotion at this turn
inner_thought
string or null
Agent's internal reasoning (when available)
latency_ms
integer
Response latency for this turn
barge_in
object or null
Barge-in details if the agent was interrupted (interrupted text, discarded utterances)
Prompt Trace
Captured LLM prompt data can include the system prompt, conversation history, tool definitions, and model response for a turn. Availability depends on runtime capture and retention, and the data is intended for debugging unexpected agent behavior.
Call Trace Analysis
Audio-native intelligence for a completed call: emotional arc, key decision moments with causal attribution, counterfactual reasoning, and actionable coaching. Produced by the Amigo intelligence pipeline from the raw call recording.
Latency: 500ms-2s (reads from analytics warehouse, not transactional store).
Status values: ready = analysis complete, pending = analysis started on first request (typically ready in 2-5 minutes; poll again), unavailable = no recording or analysis transiently unavailable (retry later).
API key issued via POST /v1/{workspace_id}/api-keys. Pass the returned api_key value as a Bearer token.
^[A-Za-z0-9][A-Za-z0-9_-]*$Analysis ready or status indicating progress
Deep call understanding produced by the Amigo intelligence pipeline.
Analyzes the raw call audio to identify emotional dynamics, key decision moments, and actionable coaching - understanding that exceeds what either participant had during the call itself.
Latency note: This endpoint reads from the analytics data warehouse. Expect 500ms-2s response time, not sub-10ms like transactional endpoints.
Call identifier (Twilio call SID or direct session ID)
ready: analysis complete. pending: call completed, analysis processing. unavailable: analysis not possible for this call.
readyPossible values: 2-3 sentence narrative of the call - the emotional story, not just what happened
Overall call outcome
Number of key decision moments identified (typically 3-8)
0Caller emotional trajectory as a readable sequence (e.g. 'curious -> confused -> frustrated -> relieved')
Synthesis of why this call went the way it did - combining audio perception with execution data
Moments where the agent could have taken a better action
Cross-call or cross-turn patterns detected by the analysis
UUID string of the associated world.entities_synced call row (if resolved)
When the analysis was produced by the pipeline
Call not found in this workspace
Validation Error
Analytics warehouse not configured
GET /v1/{workspace_id}/calls/{call_id}/trace-analysis HTTP/1.1
Host: platform.api-us.concurrence.com
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
{
"call_sid": "CA493e4610386f87f43b02102068eb6ad3",
"deep_understanding": "The call failed not because of agent behavior but because of environmental factors - the caller was pulled into a side conversation that the agent could not compete with.",
"emotional_arc": "neutral -> cooperative -> distracted -> disengaged",
"key_moment_count": 2,
"outcome": "abandoned",
"status": "ready",
"summary": "The caller initially cooperated with scheduling but became distracted by a nearby conversation, ultimately abandoning the call without booking."
}Audio-native intelligence analysis computed asynchronously after call completion. See Call Trace Analysis for the full endpoint reference.
Returns status: pending while analysis is in progress and status: unavailable if analysis is not possible for this call.
Emotional Summary
When acoustic and behavioral telemetry is available and persistence succeeds, the call detail response can include an emotional summary:
Emotional Intelligence Direction
The emotional intelligence system continues to deepen along four axes: audio-level prosodic planning (breath-like pauses, per-word pacing), sub-second emotional adaptation within a conversational beat, cross-call emotional profiles surfaced proactively from the world model, and blended emotion expressed through TTS-level control. For the conceptual treatment of emotion detection and adaptation, see Emotion Detection in the platform docs.
Last updated
Was this helpful?