> For the complete documentation index, see [llms.txt](https://docs.concurrence.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.concurrence.com/developer-guide/platform-api/safety/simulation-coverage.md).
# Simulation Coverage
Explore Context Graph coverage, step and fork text simulations, run saved cases and suites, read pass rates, and choose which reads a launch runs for real.
The simulation coverage API provides branch-and-bound exploration of context graph state space. Instead of sampling conversations at random, it steers them toward untested states, tools, and transitions.
{% hint style="info" %}
**Conceptual overview.** For background on how coverage testing works, see the [Simulation Coverage](https://docs.concurrence.com/testing/testing/simulations#simulation-coverage) conceptual documentation.
{% endhint %}
## Core Model
```mermaid
%%{init: {"theme":"base","themeVariables":{"fontFamily":"Inter, sans-serif","fontSize":"16px","background":"#FAF8F6","primaryColor":"#F1EAE7","primaryTextColor":"#1A1817","primaryBorderColor":"#D7D2D0","secondaryColor":"#FDFCFB","tertiaryColor":"#F6F2F0","lineColor":"#575452","textColor":"#3D3937","clusterBkg":"#FDFCFB","clusterBorder":"#E8E2DF","edgeLabelBackground":"#FAF8F6","actorBkg":"#F1EAE7","actorTextColor":"#1A1817","actorBorder":"#D7D2D0","actorLineColor":"#575452","signalColor":"#575452","signalTextColor":"#3D3937","labelBoxBkgColor":"#F1EAE7","labelBoxBorderColor":"#D7D2D0","labelTextColor":"#3D3937","loopTextColor":"#3D3937","noteBkgColor":"#F6F2F0","noteBorderColor":"#D7D2D0","noteTextColor":"#3D3937","activationBkgColor":"#F1EAE7","activationBorderColor":"#D7D2D0","altSectionBkgColor":"#FAF8F6","altSectionColor":"#3D3937"},"fontFamily":"Inter, sans-serif"}}%%
flowchart TB
subgraph Run["Coverage Run (tools in simulation mode)"]
S1["Session 1"] --> T1a["Turn 1"] --> T1b["Turn 2"] --> T1c["Turn 3"]
T1c -->|Fork| S2["Session 2 (child)"]
T1c -->|Fork| S3["Session 3 (child)"]
S2 --> T2a["Turn 4"]
S3 --> T3a["Turn 4"]
end
Run --> Graph["Knowledge Graph"]
subgraph Graph_Detail["Graph Layers"]
OBS["Observed: per-state session counts + pass rates"]
GHOST["Ghost nodes: states never reached"]
end
Graph --> Graph_Detail
```
Coverage is organized into three levels:
| Resource | Description |
| -------------------- | -------------------------------------------------------------------------------------------- |
| **Coverage Run** | A campaign that contains simulation sessions, whose tools run in simulation mode. |
| **Coverage Session** | A conversation path within a run. Sessions can be forked from other sessions at any turn. |
| **Coverage Turn** | A single agent turn within a session, recording context graph state, tool calls, and scores. |
Sessions form a tree structure through the fork mechanism. Each session tracks its `parent_session_id` and `fork_turn_index`, so the full exploration tree is reconstructable from the session list.
## Endpoints
All endpoints are scoped to a workspace and require a workspace API key. Reads, including benchmark results, need `Service:View`. Launches, session operations such as step, fork, recommend, and score, and other writes need `Service:Update`.
### Create Coverage Run
{% openapi src="" path="/v1/{workspace\_id}/simulations/runs" method="post" %}
{% endopenapi %}
Returns the created run with ID, status (`running`), tags, and optional branch label metadata. The label does not create a separate database branch.
Run and session tags that start with `suite:`, `suite_run:`, or `safety:` are set by the server, and sending one returns `422`. Each tag holds up to 128 characters.
A workspace can have 10 simulation launches running at once. A case, suite, benchmark, or bridge launch beyond that returns `429`, and a suite or benchmark batch counts as one launch however many cases it runs. Wait for a running launch to finish before you retry.
### Create Coverage Session
Creates a new conversation session within a run. The session starts with a fresh agent state. A run that does not exist in the workspace returns `404`, and a `service_id` that is not the run's service returns `422`.
{% openapi src="" path="/v1/{workspace\_id}/simulations/runs/{run\_id}/sessions" method="post" %}
{% endopenapi %}
### Step a Session
Sends a simulated user message and records the agent's response as a turn.
{% openapi src="" path="/v1/{workspace\_id}/simulations/sessions/{session\_id}/step" method="post" %}
{% endopenapi %}
Returns the turn observation including agent response, context graph state, tool calls, and turn index. Each tool call record says how the call ran; see [Tool Call Records](#tool-call-records). The step uses the session in the path. A body `session_id` that names a different session returns `422`. A runtime failure or timeout returns `502`, an unreachable runtime returns `503`, and a `4xx` from the runtime passes through.
### Recommend Caller Messages
Suggests up to `n` candidate caller messages (default 5, maximum 20) for the session's current state. Use them as inputs to [step](#step-a-session) or [fork](#fork-a-session). Suggestions are model-generated, so review them rather than treating them as a coverage guarantee.
{% openapi src="" path="/v1/{workspace\_id}/simulations/sessions/recommend" method="post" %}
{% endopenapi %}
The response carries `suggestions`, a list of message strings.
### Fork a Session
Atomically clones the parent session state N times, steps each clone with a different simulated message, and returns all observations. This is the core branch-and-bound primitive.
{% openapi src="" path="/v1/{workspace\_id}/simulations/sessions/{session\_id}/fork" method="post" %}
{% endopenapi %}
Each child session inherits the parent's full conversation history up to the fork point. The parent session remains unchanged. A fork with many alternatives can take up to about three minutes to return its branches, so give the request a client timeout longer than that.
### Promote a Session
Interactive playground sessions are created without a coverage run, so run-scoped operations like [fork](#fork-a-session) and [score](#score-a-session) are unavailable on them. Promotion binds a run-less session to a newly created coverage run, which unblocks fork and score. This lets developers iterate in the playground and then promote interesting sessions for deeper analysis without re-running the conversation.
{% openapi src="" path="/v1/{workspace\_id}/simulations/sessions/{session\_id}/promote" method="post" %}
{% endopenapi %}
The response returns the `session_id` (unchanged), the `run_id` of the coverage run the session is bound to, and `already_bound`. Promotion is idempotent: a session that was already run-scoped returns `already_bound: true` with its existing `run_id` unchanged.
### Score a Session
Evaluates the session against configured metrics. Scores are attributed per-session, not per-state-visit.
{% openapi src="" path="/v1/{workspace\_id}/simulations/sessions/{session\_id}/score" method="post" %}
{% endopenapi %}
Returns a confirmation status (`{ "status": "scored" }`). The supplied score and rationale are persisted; this endpoint does not return computed metric scores. Aggregated metric and coverage scores are read back through the coverage graph, the session list, or the benchmark results endpoint.
### Get Coverage Graph
Returns the knowledge graph for the run. The graph has two layers:
| Layer | Description |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| **Observed turns** | Aggregated per context graph state: session count, pass rate, total turns |
| **Topology overlay** | Ghost nodes for context graph states that exist in the state machine but were never reached. These represent coverage gaps. |
{% openapi src="" path="/v1/{workspace\_id}/simulations/services/{service\_id}/graph" method="get" %}
{% endopenapi %}
### Complete a Coverage Run
Marks the run as finished: `failed` when you pass an `error` reason, up to 500 characters, and `completed` otherwise. Sessions and turns remain available for read access. A run finishes once: a run that does not exist returns `404`, and a run that already finished returns `409` and stays as it was.
{% openapi src="" path="/v1/{workspace\_id}/simulations/runs/{run\_id}/complete" method="post" %}
{% endopenapi %}
### List Coverage Runs
Lists the workspace's simulation runs, newest first, with paging on the server. Pass `sort_by` as `-created_at` (the default) or `+created_at`, and `limit` from 1 to 200 (default 20). While `has_more` is `true`, pass the returned `continuation_token` back unchanged to read the next page. `total` counts every run that matches the filters, across all pages.
Filter by `service_id`, `status` (`running`, `completed`, or `failed`), `source` (`case`, `suite`, or `spot_check`, read from each run's tags), `case_id`, `suite_id`, and `tags`. Repeated `status` or `source` values match any of them, a run must carry every tag you pass, and different filters combine with AND. List items leave out each run's objective, request, and scenarios; [Get Coverage Run](#get-coverage-run) returns them.
{% openapi src="" path="/v1/{workspace\_id}/simulations/runs" method="get" %}
{% endopenapi %}
### Get Coverage Run
Returns the full run payload, including the stored `bridge_request` (original request body) and `scenarios` (generated persona/scenario list) for runs created through the [Simulation Bridge](#simulation-bridge). The list runs endpoint omits these fields to keep responses compact.
`snapshot` records what the run executed against. Its `context_graph.available_tools` lists the tools the Context Graph's states offer the agent, and `model_versions.agent` names the engage model the service's release version set pins, or `null` when the set pins none. `tool_execution` returns the [tool settings](#real-reads-and-mocked-results) a saved-case run launched with, and is `null` for a bridge run, which simulates every tool.
The response also includes a `checks` array alongside the raw `eval_results` - a normalized, read-only projection that flattens each metric and assertion verdict into one uniform shape. Per-case assertion detail responses carry the same `checks` array. Use it for consistent rendering; `eval_results` keeps the raw records. A judge-graded assertion's check carries `grading`: `observed_vs_expected` for an `llm_boolean` assertion, whose `actual` is what the judge observed, or `legacy_verdict` for a legacy assertion, whose verdict came from the judge with no observed value. An `llm_boolean` check also carries its `instructions`. See [Case Evals](#case-evals).
{% openapi src="" path="/v1/{workspace\_id}/simulations/runs/{run\_id}" method="get" %}
{% endopenapi %}
### Get Simulation Performance
The performance operation aggregates recent graded case and suite runs into one overview. Use `service_id` to restrict the result and `limit` (1 to 50, default 15) to bound the recent runs analyzed.
`pass_rate` is passed divided by passed plus failed. Error, pending, and skipped verdicts are counted on every rollup but stay out of the rate, and the rate is `null` when there is no passed or failed verdict. Each case, suite, and assertion rollup reports `passed`, `failed`, `error`, `pending`, and `skipped` next to its `pass_rate`. The overview reports `evals_run` (passed plus failed), `error_count`, `pending_count`, and `skipped_count` next to `overall_pass_rate`, and each metric's per-run points report `passed` and `measured`, which leaves out the same three statuses.
Every rollup reports its error, pending, and skipped counts next to its pass rate.
The three statuses outside the rate mean different things:
* **`error`** - the check couldn't be graded. Either grading broke, for example a judge answer that never arrived or a metric eval with no metric key, or the simulation stopped the conversation early, before the agent finished it. It does not mean the agent failed the check. When the agent's own turn fails, a model provider error included, or doesn't finish in time, the conversation's checks aren't `error`: each is marked `failed` without being graded.
* **`pending`** - the check has no grade yet, such as a metric value that is not available for the conversation yet.
* **`skipped`** - the check did not apply, the judge could not answer it from the conversation, or a metric check has no expectation to compare with. A conversation that forked is graded through its branches, so the checks on the parent are `skipped`.
A verdict's `rationale` says which case applies. A conversation the simulation stopped early also has no session score. The simulation stops a conversation early, for example, when a step fails for a reason that isn't the agent's, such as a platform storage error or a configured model that isn't available; when a retried step couldn't safely run again; or when the run hits its time limit. A conversation whose agent turn failed or didn't finish in time scores `0`.
LLM-judge verdicts carry `passed`, `rationale`, and `references` with no score, and `checks` entries do not include `score_label`. Metric checks include `score`.
When `metrics[].checks` is present, use it for a consistent display across metric and assertion verdicts. It normalizes the source, key, label, verdict, expected and actual values, optional score, rationale, references, and run/session identifiers while preserving `results` for callers that need the raw evaluation records.
{% openapi src="" path="/v1/{workspace\_id}/simulations/performance" method="get" %}
{% endopenapi %}
### Get Suite Run Performance
Returns the same rollup as [Get Simulation Performance](#get-simulation-performance) over every run of one suite run: pass rates, per-metric results, and per-case checks for the whole suite run in one request, however many runs it has, instead of over the workspace's most recent runs.
{% openapi src="" path="/v1/{workspace\_id}/simulations/suite-runs/{suite\_run\_id}/performance" method="get" %}
{% endopenapi %}
### List Suite Run Summaries
Repeat `suite_run_id` once per suite run, up to 50, to read their summaries in one request. Each summary carries the suite run's status, its run counts by status, the expected case count, the `service_ids` it ran against, and session and turn totals. Summaries come back in request order, and a suite run with no runs in the workspace is left out. Use it to show many suite runs at once, such as a run history.
{% openapi src="" path="/v1/{workspace\_id}/simulations/suite-runs" method="get" %}
{% endopenapi %}
## Simulation Bridge
The simulation bridge turns a natural language test objective into a set of autonomous test scenarios and runs them end-to-end against a service.
{% openapi src="" path="/v1/{workspace\_id}/simulations/bridge" method="post" %}
{% endopenapi %}
The bridge can infer a target spec when none is supplied, steer scenario generation toward coverage gaps, fork conversations when enabled, and run generation asynchronously. When asynchronous generation is enabled, poll [Get Coverage Run](#get-coverage-run) for progress.
Returns the created coverage run with generated scenarios. Each scenario includes a persona (name, background, temperament) and conversation strategy (initial message, instructions).
When `async_generation` is true, the response returns `status: running` with an empty `scenarios` array. Use the returned `run_id` with [Get Coverage Run](#get-coverage-run) to read generated scenarios and final status as they become available.
When automatic target-spec inference is enabled, the response includes the inferred spec and its rationale.
The bridge response itself carries no execution summary. Read `total_sessions` and per-session results back through [Get Coverage Run](#get-coverage-run) once execution finishes.
Each session is tagged with metadata for joining sessions back to their source scenarios: `scenario_index:N` identifies which generated scenario produced the session, and `temperament:X` records the persona temperament used.
The bridge creates a tracked coverage run, so results are visible through the existing coverage graph, session list, and scoring endpoints. A bridge run simulates every tool; to run chosen reads for real, launch [saved cases](#saved-cases-and-suites) instead.
This is the REST API equivalent of `forge platform sim bridge` - same scenario generation and autonomous execution, accessible without the CLI.
## Saved Cases and Suites
A saved case holds a persona, a scenario, optional grounding, and the evals that grade its runs. A suite selects cases by explicit case IDs, by required tags, or both. See [Simulations](https://docs.concurrence.com/testing/testing/simulations#cases) for how cases and suites fit together.
### List Cases
The case list pages on the server with `limit` (1 to 200, default 50) and an opaque `continuation_token`, and `total` counts every case that matches, across all pages. Pass `sort_by` as `+field` or `-field` on `created_at`, `updated_at`, or `description`; repeated values apply left to right, and the default is newest created first. `search` is a case-insensitive substring match on the case description, and `service_id` and `tags` filter the list.
{% openapi src="" path="/v1/{workspace\_id}/simulations/cases" method="get" %}
{% endopenapi %}
Deleting a case removes it from every suite that lists it.
Case and suite tags can't carry `suite_run:` or a `suite:` tag that names a saved suite, because the platform sets those on runs; a tag-defined suite such as `suite:scheduling-core` is allowed. `safety:` tags are reserved for the platform's own safety cases.
### List a Suite's Cases
Returns one page of the cases a suite selects, resolved on the server: its explicit `case_ids` plus every case that carries all of its `required_tags`. It takes the same paging, sort, and search as the case list, and without `search`, `total` equals the suite's `case_count`. Read a suite's cases here rather than one case at a time.
{% openapi src="" path="/v1/{workspace\_id}/simulations/suites/{suite\_id}/cases" method="get" %}
{% endopenapi %}
### Case Evals
A case's `evals` grade each conversation its runs record. A case write accepts three shapes:
* **Metric evals** (`type: metric`) compare a metric's value with `expected`.
* **Deterministic assertions** (`type: assertion`) are checked in code against the transcript, the tools called, or the final state. `assertion_kind` is `transcript_contains`, `must_contain`, `transcript_not_contains`, `must_not_contain`, `tool_called`, or `final_state`, and each reads its target from `expected` or from a `params` field such as `phrase`, `tool_name`, or `state`. One with no target is recorded as an `error` verdict.
* **`llm_boolean` assertions** ask a judge a yes-or-no question about the conversation. `instructions` holds the question and what makes its answer true or false, and `expected` is the boolean answer that passes.
```json
{
"type": "assertion",
"key": "offered_callback",
"assertion_kind": "llm_boolean",
"instructions": "Did the agent offer a callback from the scheduling team? True if it offered one, false if it did not.",
"expected": true
}
```
The judge reads the case and the transcript, never `expected`, and answers `true`, `false`, or that the conversation doesn't show enough to tell. The check passes when the answer equals `expected` and fails otherwise, and the check's `actual` records what the judge observed. When the judge can't tell, the check is `skipped`, with `details.ungraded_reason` set to `insufficient_evidence`, and stays out of pass rates. An answer outside this contract is recorded as an `error` verdict. The judge is a model, so calibrate each question against conversations you have reviewed.
Legacy assertions, saved with `assertion_kind` `llm_judge` or a free-text kind, are graded by a judge that returns the pass or fail verdict itself, and case reads return them with `grading: legacy_verdict`. A case write that carries one returns `422` with a message naming the accepted shapes, so rewrite each as an `llm_boolean` or deterministic assertion before you save the case's `evals` again.
### Run a Case
Runs the saved case and returns the created run. `service_id` runs the case against another service for this launch only; without it, the case's saved service runs. `entity_id` binds caller context, and without it the run uses the entity in the case's grounding, if any. `tool_execution` chooses which tools run for real; see [Real Reads and Mocked Results](#real-reads-and-mocked-results).
{% openapi src="" path="/v1/{workspace\_id}/simulations/cases/{case\_id}/run" method="post" %}
{% endopenapi %}
### Run a Suite
Runs the cases a suite selects as one batch, up to `max_cases` (default 20, maximum 200), with up to `concurrency` cases running at a time. The response returns a stable `batch_id`, the started run IDs, each skipped or failed-to-start case with its `reason`, and the `tool_execution` the cases run with. The batch counts as one launch toward the workspace's limit.
{% openapi src="" path="/v1/{workspace\_id}/simulations/suites/{suite\_id}/run" method="post" %}
{% endopenapi %}
## Saved Case Benchmarks
Saved-case benchmarks run a batch of simulation cases selected by suite or by required tags and queue each selected case as its own bridge run. The benchmark response returns immediately with a stable `batch_id`, the selected case IDs, started run IDs, skipped cases, failed-to-start cases, an aggregate start summary, and the `tool_execution` the cases run with.
{% openapi src="" path="/v1/{workspace\_id}/simulations/benchmarks/run" method="post" %}
{% endopenapi %}
### Results
Use the returned `run_ids` to aggregate completion status, scores, pass counts, metric availability, per-run summaries, and capability breakdowns. A session the simulation stopped early has no score and counts in a run's `error_session_count`, outside `average_score` and the pass and fail counts. A session whose agent turn failed or didn't finish in time scores `0` and counts as failed.
In case, suite, and benchmark runs, the platform retries a simulation step after a transient failure or a timeout. A retried step doesn't save the agent's turn twice, though it can run the turn's live reads again.
{% openapi src="" path="/v1/{workspace\_id}/simulations/benchmarks/results" method="post" %}
{% endopenapi %}
## Bridge Plan
There is no plan-only bridge endpoint. Prepare the target specification before submitting it through the supported bridge workflow above. Starting a bridge run executes simulations; it is not a dry-run planning operation.
## Write Isolation
Simulation sessions read the workspace's current configuration and run through the production reasoning path, with every tool in simulation mode unless a launch names it otherwise. A tool in simulation mode does not reach an integration, a world write, or a platform function. Reads and writes alike return a simulated result instead of executing:
```json
{"dry_run": true, "tool": "world_schedule_appointment", "would_execute": true}
```
The mode is fixed when the session starts and applies to direct tool calls, companion-skill sub-tools, and every forked branch. A simulated call to an integration endpoint that declares a write counts as the write its request would make, so the reply check accepts the agent's confirmation as it would after that write succeeded. Calls from workspace test caller numbers and text turns from test credentials run every tool the same way. The text Playground and browser test calls run tools live, so use them, with test identities and non-production credentials, when a test needs real writes. Tools that need an operator's approval, such as an endpoint an approval policy gates or a skill marked approval-required, are not offered in simulations. Restricted Surface tools remain unavailable in simulations.
### Real Reads and Mocked Results
A case, suite, or benchmark launch can pass `tool_execution` to run chosen reads for real and to return a fixed result in place of a call:
```json
{
"tool_execution": {
"live_read_tools": ["world_appointment_lookup"],
"tool_mocks": {"payer_check_eligibility": "{\"eligible\": true}"}
}
}
```
* `live_read_tools` names up to 64 read tools that run for real. List only tools that the [Resolve Available Tools](https://docs.concurrence.com/developer-guide/platform-api/functions/tool-testing#resolve-available-tools) operation marks `live_read_eligible` for the service, and don't list a tool in `tool_mocks` too. Writes, approval-gated tools, and companion skills never run for real.
* `tool_mocks` maps up to 64 tool names to the result string each returns. To mock a companion skill, mock the tools it calls.
* Every other tool returns the simulated result. The settings apply to every case, step, and fork of the launch.
Use the tool names the Resolve Available Tools operation returns, where an integration endpoint is named `_`. A name the service's simulations can't honor returns `422` naming the problem. A live read also needs the permission you need to test that tool: `Integration:View` for integration endpoints, `Workspace:View` for platform functions and data queries, and `Service:Update`, which every launch needs, for world reads. A case anchored to an external principal runs its live reads as that principal, so it also needs `Conversation:CreateForPrincipal`. A missing permission returns `403`, and settings that can't be checked against the service's tools return `503`. In a suite or benchmark, either one stops the whole batch before any case starts, while a case whose service can't honor the settings is skipped with the reason and the other cases run.
The launch response echoes the settings, and [Get Coverage Run](#get-coverage-run) returns them as `tool_execution`. Coverage runs, their sessions, and bridge runs simulate every tool.
### Tool Call Records
Each tool call record in a step, fork, turns, or coverage graph response carries `execution` and `duration_ms`. `execution` is `live` for a call that ran for real, `mocked` for one answered from `tool_mocks`, or `dry_run` for one that returned the simulated result. A companion skill's record is `dry_run` if any tool it can call returns the simulated result, otherwise `mocked` if any is mocked, and `null` when every tool it can call runs for real or it can call none. It is never `live`, because a skill's output is its own text rather than a system's response.
Only a `live` record shows what an external system returned; a `mocked` or `dry_run` record supports only checks of how the agent behaved given that output. A check that needs evidence the run doesn't have can be left `skipped`, with `details.ungraded_reason` set to `insufficient_evidence`, rather than passed or failed, and stays out of pass rates.
### Audio Scope
This is Tier-1 text simulation: the simulated caller sends text through the production reasoning path, with tools in simulation mode. It does not test telephony, media attachment, speech recognition, text-to-speech, or other audio behavior. Use a real-audio test workflow for those concerns.
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://docs.concurrence.com/developer-guide/platform-api/safety/simulation-coverage.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
