Cost and Latency Optimization

Plan deployment cost and latency measurements, distinguish customer billing from internal cost allocation, and understand runtime optimization trade-offs.

Estimate the Deployment

Start with the workflow and the commercial terms proposed for your deployment. Record expected eligible volume, channel mix, conversation length, source processing, integration work, evaluation, and human exception handling. Ask which items are included, which vary with usage, and which are billed by another system. These are planning inputs, not a published universal price formula.

Use the integration effort worksheet to separate access/provisioning lead time, implementation, validation, and ongoing operation. Compare like-for-like outcomes: count incomplete work, retries, and human follow-up in the denominator where they are part of the workflow. Cost per attempted interaction and cost per verified outcome answer different questions.

Separate Software, Operation, and Cloud Charges

For each proposed line item, record its payee, included work, unit, rate, expected quantity, and responsible budget owner. Separate platform licensing, implementation, ongoing managed operations, Concurrence-billed usage, customer-billed cloud consumption, and external provider charges. These are review categories, not a fixed Concurrence SKU list. Identify bundled items so that the same compute or delivery cost is not counted twice.

For AWS commitments, distinguish direct AWS consumption from a seller purchase through AWS Marketplace. AWS describes eligible Marketplace transactions as potentially contributing to committed spend. Confirm the proposed product and transaction, buyer agreement, applicable limits, and timing with the buyer's AWS and procurement owners. An AWS-hosted architecture alone does not establish eligibility. A waived fee contributes no purchase amount, even if separately billed cloud usage continues.

Commitment retirement describes how qualifying spend counts against a purchase obligation. Compare total program cost and verified outcomes separately; qualifying spend is still a cost.

Measure the Latency That Matters

Measurement	Start and end boundary	Report alongside it
First visible text / audible response	Accepted user input to the first rendered text or audible output	Channel, service configuration, and whether the response is only an acknowledgement
Completed answer	User request to the actual final answer, including supported background work	Pending and timed-out cases; exclude neither silently
External outcome	User request to target acknowledgement and read-back where required	Target-system delay, approvals, and unresolved cases
Human exception	Exception raised to the agreed human response or resolution	Staffing window and workload

Choose the observation window and percentiles before testing. Report sample size, failures, missing observations, configuration, and the intended channel. A simulation result is not a latency guarantee for a live phone call or an external scheduling system. Any contractual service level must be confirmed separately.

Runtime Optimization

Concurrence's platform-managed conversation runtime uses a stable prompt prefix, per-task model routing, and timing controls for live responses. Provider-owned speech-to-speech sessions have a different execution path. Measure the selected runtime under the expected workload before changing its configuration.

Stable-Prefix Prompt Caching

The platform-managed conversation runtime builds prompts in two parts:

• Static prefix - Session-constant instructions such as the agent identity, service description, capabilities, context-graph topology, and output format.
• Dynamic suffix - Per-turn state, objectives, active guidance, current time, and bounded conversation history.

When the selected model provider supports prompt-prefix caching, sending the stable content first lets the provider reuse work across turns with the same prefix. The dynamic suffix still changes as the conversation advances, and conversation history remains part of that per-turn content.

This split reduces repeated processing without changing what the model receives. A provider that does not support compatible prefix caching can process the complete prompt normally; correctness does not depend on a cache hit.

Per-Task Model Routing

Different parts of the pipeline use different models based on what the task demands:

Task	Model Selection	Why
Navigation	Separately configured low-latency model	Produces a small, structured routing decision on each turn
Response generation	Separately configured engagement model	Produces the caller-facing response, where language quality matters most
Filler speech	No separate model call	Short time-buying phrases come from prepared templates, while context-aware fillers are produced inline by the same model that generates the response - so no extra model round trip is added
Post-call data review	Separately configured offline processing	Runs outside the live turn path, so its latency and model choice can differ from conversation generation
Metric evaluation	Definition- and pipeline-specific model tier	Eligible evaluations can run eagerly or in scheduled processing without blocking the live conversation

Separate model roles let teams and platform defaults balance latency, quality, and cost by task. Actual usage depends on the resolved service configuration, runtime, fallbacks, and enabled evaluation paths.

Pipeline Latency Budget

For voice interactions, measure transcription, first audio, and full response latency separately. End-to-end response time varies with the selected models, speech provider, tools, and network path. Three mechanisms reduce the delay callers perceive:

Filler speech covers the gap between when the caller finishes speaking and when the agent's full response is ready. The caller hears "Let me check on that" or a similar phrase while the LLM generates the real response.

Prompt caching can reduce repeated input processing when the selected provider supports it. The session-constant prefix stays stable while the current state and conversation history remain in the dynamic suffix.

Two-phase initialization can prepare the agent during ring time on supported conference-based voice paths. It loads available context and can begin generating the greeting before answer. Provider-side prompt caching and preparation can still miss, so this reduces expected delay rather than guaranteeing instant audio.

Usage Visibility

The Platform API exposes workspace-scoped billing reads for current-period usage, usage trends, invoice lists, and invoice detail. Values depend on the meters that producers emitted and on asynchronous projection, so an accepted interaction is not proof that every billing rollup has already refreshed.

Teams can use those API responses for their own reporting and reconciliation. Invoice detail includes the line items generated for that billing period, with meter key, quantity, unit price, and amount; the actual catalog depends on configured prices and emitted usage rather than a fixed list of channel examples.

Cost to Serve Visibility

Cost to Serve distinguishes workspace LLM usage, customer-visible calculated model cost where enabled, and staff-only cost reports. The former monthly inference and infrastructure endpoints are retired; current LLM reports do not provide infrastructure allocations. Review attribution and unpriced usage before comparing costs, and use the agreed pricing schedule and customer billing records for procurement estimates and invoice reconciliation.