> For the complete documentation index, see [llms.txt](https://docs.concurrence.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.concurrence.com/platform-overview/cost-and-latency.md). # Cost and Latency Optimization Plan deployment cost and latency measurements, distinguish customer billing from internal cost allocation, and understand runtime optimization trade-offs. ## Estimate the Deployment Start with the workflow and the commercial terms proposed for your deployment. Record expected eligible volume, channel mix, conversation length, source processing, integration work, evaluation, and human exception handling. Ask which items are included, which vary with usage, and which are billed by another system. These are planning inputs, not a published universal price formula. Use the [integration effort worksheet](/platform-overview/evaluation-packet.md) to separate access/provisioning lead time, implementation, validation, and ongoing operation. Compare like-for-like outcomes: count incomplete work, retries, and human follow-up in the denominator where they are part of the workflow. Cost per attempted interaction and cost per verified outcome answer different questions. ## Separate Software, Operation, and Cloud Charges For each proposed line item, record its payee, included work, unit, rate, expected quantity, and responsible budget owner. Separate platform licensing, implementation, ongoing managed operations, Concurrence-billed usage, customer-billed cloud consumption, and external provider charges. These are review categories, not a fixed Concurrence SKU list. Identify bundled items so that the same compute or delivery cost is not counted twice. For AWS commitments, distinguish direct AWS consumption from a seller purchase through AWS Marketplace. [AWS describes eligible Marketplace transactions as potentially contributing to committed spend](https://aws.amazon.com/blogs/awsmarketplace/private-offer-auto-renewals-scale-predictable-revenue-with-aws-marketplace/). Confirm the proposed product and transaction, buyer agreement, applicable limits, and timing with the buyer's AWS and procurement owners. An AWS-hosted architecture alone does not establish eligibility. A waived fee contributes no purchase amount, even if separately billed cloud usage continues. Commitment retirement describes how qualifying spend counts against a purchase obligation. Compare total program cost and verified outcomes separately; qualifying spend is still a cost. ## Measure the Latency That Matters | Measurement | Start and end boundary | Report alongside it | | ------------------------------------- | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | | First visible text / audible response | Accepted user input to the first rendered text or audible output | Channel, service configuration, and whether the response is only an acknowledgement | | Completed answer | User request to the actual final answer, including supported background work | Pending and timed-out cases; exclude neither silently | | External outcome | User request to target acknowledgement and read-back where required | Target-system delay, approvals, and unresolved cases | | Human exception | Exception raised to the agreed human response or resolution | Staffing window and workload | Choose the observation window and percentiles before testing. Report sample size, failures, missing observations, configuration, and the intended channel. A simulation result is not a latency guarantee for a live phone call or an external scheduling system. Any contractual service level must be confirmed separately. ## Runtime Optimization Concurrence's platform-managed conversation runtime uses a stable prompt prefix, per-task model routing, and timing controls for live responses. Provider-owned speech-to-speech sessions have a different execution path. Measure the selected runtime under the expected workload before changing its configuration. ## Stable-Prefix Prompt Caching The platform-managed conversation runtime builds prompts in two parts: * **Static prefix** - Session-constant instructions such as the agent identity, service description, capabilities, context-graph topology, and output format. * **Dynamic suffix** - Per-turn state, objectives, active guidance, current time, and bounded conversation history. When the selected model provider supports prompt-prefix caching, sending the stable content first lets the provider reuse work across turns with the same prefix. The dynamic suffix still changes as the conversation advances, and conversation history remains part of that per-turn content. This split reduces repeated processing without changing what the model receives. A provider that does not support compatible prefix caching can process the complete prompt normally; correctness does not depend on a cache hit. ## Per-Task Model Routing Different parts of the pipeline use different models based on what the task demands: | Task | Model Selection | Why | | ------------------------- | -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Navigation** | Separately configured low-latency model | Produces a small, structured routing decision on each turn | | **Response generation** | Separately configured engagement model | Produces the caller-facing response, where language quality matters most | | **Filler speech** | No separate model call | Short time-buying phrases come from prepared templates, while context-aware fillers are produced inline by the same model that generates the response - so no extra model round trip is added | | **Post-call data review** | Separately configured offline processing | Runs outside the live turn path, so its latency and model choice can differ from conversation generation | | **Metric evaluation** | Definition- and pipeline-specific model tier | Eligible evaluations can run eagerly or in scheduled processing without blocking the live conversation | Separate model roles let teams and platform defaults balance latency, quality, and cost by task. Actual usage depends on the resolved service configuration, runtime, fallbacks, and enabled evaluation paths. ## Pipeline Latency Budget For voice interactions, measure transcription, first audio, and full response latency separately. End-to-end response time varies with the selected models, speech provider, tools, and network path. Three mechanisms reduce the delay callers perceive: **Filler speech** covers the gap between when the caller finishes speaking and when the agent's full response is ready. The caller hears "Let me check on that" or a similar phrase while the LLM generates the real response. **Prompt caching** can reduce repeated input processing when the selected provider supports it. The session-constant prefix stays stable while the current state and conversation history remain in the dynamic suffix. **Two-phase initialization** can prepare the agent during ring time on supported conference-based voice paths. It loads available context and can begin generating the greeting before answer. Provider-side prompt caching and preparation can still miss, so this reduces expected delay rather than guaranteeing instant audio. ## Usage Visibility The Platform API exposes workspace-scoped billing reads for current-period usage, usage trends, invoice lists, and invoice detail. Values depend on the meters that producers emitted and on asynchronous projection, so an accepted interaction is not proof that every billing rollup has already refreshed. Teams can use those API responses for their own reporting and reconciliation. Invoice detail includes the line items generated for that billing period, with meter key, quantity, unit price, and amount; the actual catalog depends on configured prices and emitted usage rather than a fixed list of channel examples. ## Cost to Serve Visibility [Cost to Serve](/intelligence-and-analytics/cost-to-serve.md) distinguishes workspace LLM usage, customer-visible calculated model cost where enabled, and staff-only cost reports. The former monthly inference and infrastructure endpoints are retired; current LLM reports do not provide infrastructure allocations. Review attribution and unpriced usage before comparing costs, and use the agreed pricing schedule and customer billing records for procurement estimates and invoice reconciliation. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.concurrence.com/platform-overview/cost-and-latency.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.