> For the complete documentation index, see [llms.txt](https://docs.concurrence.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.concurrence.com/developer-guide/guides/arbiter-voice-tests.md).

# Run Arbiter Voice Tests

Validate and apply an Arbiter suite, place black-box test calls to an approved phone target, gate on exit codes, and read reports and transcripts.

Arbiter places real phone calls to a voice agent, plays the caller, and grades what the caller can observe. The target can be a phone agent built outside Concurrence at an approved, reachable number.

## Before You Start

Arrange Arbiter access with your Concurrence implementation team. Before testing, confirm the following.

* Provisioned Arbiter access, with the `arbiter` CLI installed and its profile and authentication configured for you
* A phone target you are approved to call, registered as an Arbiter target
* Synthetic records in the target system, and a provisioned caller-ID number when the scenario depends on phone lookup

Runs place real calls and can change the target system. Use synthetic records and test endpoints. Check the setup before the first run:

```bash
arbiter config doctor
```

## Inspect Targets, Suites, and Cases

```bash
arbiter target list
arbiter suite list
arbiter suite get <suite-id>
arbiter case list <suite-id>
```

Add `--json` to any command for machine-readable output.

## Validate and Apply a Suite File

A suite file is YAML or JSON in the Arbiter API format. `display_name` and `agent_target_id` are required.

```yaml
display_name: Scheduling regression
agent_target_id: <target-id>
language: en-US
cases:
  - name: Reschedule for a synthetic patient
    mode: llm
    persona: A polite caller who wants to move an appointment to next Tuesday.
    number_of_runs: 3
    assertions:
      - type: call_completed
      - type: llm_judge
        prompt: The agent confirms the new appointment day before the call ends.
        expect: true
```

Replace `<target-id>` with the `id` from `arbiter target get <name> --json`. A case uses `mode: llm` with a persona or `mode: scripted` with a `script`.

```bash
arbiter suite validate suite.yaml
arbiter suite apply -f suite.yaml
```

`suite validate` runs offline and exits `0` when the file is valid and `2` otherwise. Add `--strict` to treat warnings as errors. `suite apply` creates a suite when the file has no top-level `id` and replaces the suite when it has one. An unknown `id` exits `4` and does not create a suite.

## Run a Suite and Gate on the Result

```bash
arbiter suite run <suite-id> --fail-on-suite-fail --fail-on-inconclusive \
  -o report.json --junit arbiter.xml
```

Use `arbiter case run <suite-id> <case-id>` for one case. The CLI prints `launched run <run-id>` when the launch is accepted. Record that ID.

`--wait` alone exits `0` for any completed run, including one with failed cases. Add a gate flag to fail a pipeline. Both gate flags imply `--wait`.

| Exit code | Meaning                                                                                                                                |
| --------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| `0`       | The run completed and no selected gate failed                                                                                          |
| `10`      | The run completed, and the suite verdict failed with `--fail-on-suite-fail` or a result was inconclusive with `--fail-on-inconclusive` |
| `11`      | The run ended as failed or cancelled                                                                                                   |
| `12`      | `--wait-timeout` elapsed. The run keeps going                                                                                          |
| `130`     | The wait was interrupted. The run keeps going unless you passed `--cancel-on-interrupt`                                                |

The CLI writes `-o` and `--junit` after the run ends and before it decides the exit code. A report or file write error can still prevent an artifact, so have CI check that the expected files exist and retain them. The JUnit file reports an inconclusive case as skipped unless `--fail-on-inconclusive` is set.

## Read the Report and Transcripts

```bash
arbiter run get <run-id>
arbiter run report <run-id> -o report.json --junit arbiter.xml
arbiter run transcript <run-id> --case <case-id>
```

`run report` retries while the report is being written, up to `--report-timeout`. Expect passed, failed, inconclusive, and errored counts, per-case assertion results, and transcripts. Review inconclusive and errored results as part of the total.

## Recover Without Duplicate Calls

* **Unknown launch.** A launch is sent once and is not retried. If it fails with a network or server error, or is interrupted before it returns, the run may or may not exist. Check `arbiter run list` before you launch again, because a second launch places a second set of calls.
* **Timeout or interrupt.** After exit `12` or `130` the run continues. Re-attach with `arbiter run watch <run-id>`, or stop it with `arbiter run cancel <run-id>`.
* **Cleanup.** Reverse any changes the calls made to synthetic records. `arbiter suite delete <suite-id> --yes` archives a finished suite.

## What a Passing Run Shows

A passing run is caller-observable evidence for the target and suite you ran. It does not show the target's internal tool calls or confirm an external write. Pair it with logical tests and external-action checks as described in [Healthcare Agent Testing Method](https://docs.concurrence.com/testing/testing/healthcare-agent-testing). [Black-Box Voice Testing](https://docs.concurrence.com/testing/testing/voice-simulation) explains what each call artifact supports.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.concurrence.com/developer-guide/guides/arbiter-voice-tests.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
