Amigo is now Concurrence
Ctrlk
Go to website
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Production Evals

Define eval criteria for production calls and evaluate completed calls against them.

Production evals let you run the same assertion and metric checks used in simulations against your workspace's live completed calls. You define eval definitions (what to check), evaluate completed calls on demand, and retrieve the verdicts.

Metric evals reference metrics defined in the metric store, so define the metric there first:

Metric Store

Production Eval Definitions

A production eval definition specifies an eval that runs against live calls. Definitions can be scoped to a specific service or applied workspace-wide (service ID omitted). When a call is evaluated, workspace-wide definitions apply to all calls, and service-scoped definitions apply only to calls handled by that service. If both a workspace-wide and a service-scoped definition share the same eval key, the service-scoped definition takes priority for that service's calls.

Create a Production Eval Definition

Create Production Eval Definition

post

Create a production eval definition.

metric_key must name an active ai_query metric on call_intelligence in this workspace's metric settings; any other key returns 422.

Authorizations
AuthorizationstringRequired

API key issued via POST /v1/{workspace_id}/api-keys. Pass the returned api_key value as a Bearer token.

Path parameters
workspace_idstring · uuidRequired
Body
eval_keystring · min: 1 · max: 128Required
eval_typestring · enumRequiredPossible values:
service_idstring · uuid · nullableOptional
metric_keystring · max: 128 · nullableOptional
expectedanyOptional
activebooleanOptionalDefault: true
Responses
201

Successful Response

application/json

An eval (assertion or metric-eval) that runs against a workspace's live calls — the production analogue of a sim case's evals. service_id None applies to every service in the workspace. expected is JSON: a phrase string for assertions, or a {gte|lte|equals|...} threshold for metric-evals.

idstring · uuidOptional
workspace_idstring · uuidRequired
service_idstring · uuid · nullableOptional
eval_keystring · max: 128Required
eval_typestring · enumRequiredPossible values:
assertion_kindstring · max: 128 · nullableOptional
metric_keystring · max: 128 · nullableOptional
expectedanyOptional
activebooleanOptionalDefault: true
created_atstring · date-time · nullableOptional
updated_atstring · date-time · nullableOptional
post/v1/{workspace_id}/production-eval-definitions
POST /v1/{workspace_id}/production-eval-definitions HTTP/1.1
Host: platform.api-us.concurrence.com
Authorization: Bearer YOUR_SECRET_TOKEN
Content-Type: application/json
Accept: */*
Content-Length: 190

{
  "eval_key": "text",
  "eval_type": "metric",
  "service_id": "123e4567-e89b-12d3-a456-426614174000",
  "metric_key": "text",
  "expected": null,
  "params": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "active": true
}
{
  "id": "123e4567-e89b-12d3-a456-426614174000",
  "workspace_id": "123e4567-e89b-12d3-a456-426614174000",
  "service_id": "123e4567-e89b-12d3-a456-426614174000",
  "eval_key": "text",
  "eval_type": "assertion",
  "assertion_kind": "text",
  "metric_key": "text",
  "expected": null,
  "params": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "active": true,
  "created_at": "2026-01-01T00:00:00.000Z",
  "updated_at": "2026-01-01T00:00:00.000Z"
}

Creates a new production eval definition.

Errors:

  • 409 Conflict - A definition with this eval key already exists in this scope

  • 422 Unprocessable Entity - Validation error (e.g. metric type without metric_key, or service_id not in this workspace)

List Production Eval Definitions

List Production Eval Definitions

get
Authorizations
AuthorizationstringRequired

API key issued via POST /v1/{workspace_id}/api-keys. Pass the returned api_key value as a Bearer token.

Path parameters
workspace_idstring · uuidRequired
Query parameters
sort_bystring[]OptionalDefault: []
limitinteger · max: 200OptionalDefault: 50
continuation_tokenanyOptional
service_idstring · uuid · nullableOptional
activeboolean · nullableOptional
Responses
200

Successful Response

application/json
has_morebooleanRequired
continuation_tokenanyOptional
get/v1/{workspace_id}/production-eval-definitions
GET /v1/{workspace_id}/production-eval-definitions HTTP/1.1
Host: platform.api-us.concurrence.com
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*

{
  "items": [
    {
      "id": "123e4567-e89b-12d3-a456-426614174000",
      "workspace_id": "123e4567-e89b-12d3-a456-426614174000",
      "service_id": "123e4567-e89b-12d3-a456-426614174000",
      "eval_key": "text",
      "eval_type": "assertion",
      "assertion_kind": "text",
      "metric_key": "text",
      "expected": null,
      "params": {
        "ANY_ADDITIONAL_PROPERTY": "anything"
      },
      "active": true,
      "created_at": "2026-01-01T00:00:00.000Z",
      "updated_at": "2026-01-01T00:00:00.000Z"
    }
  ],
  "has_more": true,
  "continuation_token": null
}

Returns a paginated list of production eval definitions.

Get a Production Eval Definition

The separate single-definition read endpoint has been removed. Read definition records through the list operation above; updates and deletion still use the definition identifier.

Update a Production Eval Definition

Update Production Eval Definition

patch

Update a production eval definition.

A patch that sets metric_key or eval_type, or sets active to true, re-checks the metric: a metric eval must name an active ai_query metric on call_intelligence, else 422. Clearing metric_key on a metric eval also returns 422. Any other patch, including active: false, leaves the stored key alone.

Authorizations
AuthorizationstringRequired

API key issued via POST /v1/{workspace_id}/api-keys. Pass the returned api_key value as a Bearer token.

Path parameters
workspace_idstring · uuidRequired
definition_idstring · uuidRequired
Body
service_idstring · uuid · nullableOptional
eval_keystring · min: 1 · max: 128 · nullableOptional
eval_typestring · enum · nullableOptionalPossible values:
assertion_kindstring · max: 128 · nullableOptional
metric_keystring · max: 128 · nullableOptional
expectedanyOptional
activeboolean · nullableOptional
Responses
200

Successful Response

application/json

An eval (assertion or metric-eval) that runs against a workspace's live calls — the production analogue of a sim case's evals. service_id None applies to every service in the workspace. expected is JSON: a phrase string for assertions, or a {gte|lte|equals|...} threshold for metric-evals.

idstring · uuidOptional
workspace_idstring · uuidRequired
service_idstring · uuid · nullableOptional
eval_keystring · max: 128Required
eval_typestring · enumRequiredPossible values:
assertion_kindstring · max: 128 · nullableOptional
metric_keystring · max: 128 · nullableOptional
expectedanyOptional
activebooleanOptionalDefault: true
created_atstring · date-time · nullableOptional
updated_atstring · date-time · nullableOptional
patch/v1/{workspace_id}/production-eval-definitions/{definition_id}
PATCH /v1/{workspace_id}/production-eval-definitions/{definition_id} HTTP/1.1
Host: platform.api-us.concurrence.com
Authorization: Bearer YOUR_SECRET_TOKEN
Content-Type: application/json
Accept: */*
Content-Length: 217

{
  "service_id": "123e4567-e89b-12d3-a456-426614174000",
  "eval_key": "text",
  "eval_type": "assertion",
  "assertion_kind": "text",
  "metric_key": "text",
  "expected": null,
  "params": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "active": true
}
{
  "id": "123e4567-e89b-12d3-a456-426614174000",
  "workspace_id": "123e4567-e89b-12d3-a456-426614174000",
  "service_id": "123e4567-e89b-12d3-a456-426614174000",
  "eval_key": "text",
  "eval_type": "assertion",
  "assertion_kind": "text",
  "metric_key": "text",
  "expected": null,
  "params": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "active": true,
  "created_at": "2026-01-01T00:00:00.000Z",
  "updated_at": "2026-01-01T00:00:00.000Z"
}

Partial update. Include only the fields you want to change.

Errors:

  • 404 Not Found - Definition does not exist

  • 409 Conflict - Update would create a duplicate eval key in this scope

  • 422 Unprocessable Entity - No fields provided, or service_id not in this workspace

Delete a Production Eval Definition

Delete Production Eval Definition

delete
Authorizations
AuthorizationstringRequired

API key issued via POST /v1/{workspace_id}/api-keys. Pass the returned api_key value as a Bearer token.

Path parameters
workspace_idstring · uuidRequired
definition_idstring · uuidRequired
Responses
204

Successful Response

No content
delete/v1/{workspace_id}/production-eval-definitions/{definition_id}
DELETE /v1/{workspace_id}/production-eval-definitions/{definition_id} HTTP/1.1
Host: platform.api-us.concurrence.com
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*

No content

Evaluating a Call

Evaluate Call

post

Run the workspace's active eval definitions against one completed call and persist the verdicts. Synchronous (the caller waits on the judge); the eager post-call trigger is a separate change.

Authorizations
AuthorizationstringRequired

API key issued via POST /v1/{workspace_id}/api-keys. Pass the returned api_key value as a Bearer token.

Path parameters
workspace_idstring · uuidRequired
conversation_idstring · uuidRequired
Responses
200

Successful Response

application/json
conversation_idstring · uuidRequired
post/v1/{workspace_id}/calls/{conversation_id}/evaluate
POST /v1/{workspace_id}/calls/{conversation_id}/evaluate HTTP/1.1
Host: platform.api-us.concurrence.com
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*

{
  "conversation_id": "123e4567-e89b-12d3-a456-426614174000",
  "results": [
    {
      "id": "123e4567-e89b-12d3-a456-426614174000",
      "workspace_id": "123e4567-e89b-12d3-a456-426614174000",
      "service_id": "123e4567-e89b-12d3-a456-426614174000",
      "conversation_id": "123e4567-e89b-12d3-a456-426614174000",
      "call_sid": "text",
      "eval_key": "text",
      "eval_type": "assertion",
      "assertion_kind": "text",
      "metric_key": "text",
      "status": "passed",
      "passed": true,
      "score": 1,
      "expected": null,
      "actual": null,
      "rationale": "text",
      "justification": "text",
      "cited_turns": [],
      "computed_at": "2026-01-01T00:00:00.000Z",
      "created_at": "2026-01-01T00:00:00.000Z"
    }
  ]
}

Runs all active production eval definitions against one completed call and persists the verdicts. The call must exist in the workspace. Definitions scoped to the call's service and workspace-wide definitions are both applied. If a workspace-wide and a service-scoped definition share the same eval key, the service-scoped definition takes priority.

This endpoint is synchronous - the caller waits while the eval judge processes. Assertion evals (transcript contains, tool called, final state) are deterministic and fast. AI judge assertions and metric evals invoke an AI model and may take several seconds per eval.

Re-evaluating a call overwrites prior verdicts for the same eval keys rather than creating duplicates.

Errors:

  • 404 Not Found - Conversation does not exist in this workspace

Notes:

  • An empty results array means the call was found but no active definitions apply to it.

  • Each eval definition that encounters an error (e.g. AI model unavailable) produces an individual error status verdict rather than failing the entire evaluation.

Retrieving Call Eval Results

Get Call Eval Results

get

A call's verdicts. Empty when the call exists but hasn't been evaluated; 404 when it doesn't exist.

Authorizations
AuthorizationstringRequired

API key issued via POST /v1/{workspace_id}/api-keys. Pass the returned api_key value as a Bearer token.

Path parameters
workspace_idstring · uuidRequired
conversation_idstring · uuidRequired
Responses
200

Successful Response

application/json
conversation_idstring · uuidRequired
get/v1/{workspace_id}/calls/{conversation_id}/eval-results
GET /v1/{workspace_id}/calls/{conversation_id}/eval-results HTTP/1.1
Host: platform.api-us.concurrence.com
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*

{
  "conversation_id": "123e4567-e89b-12d3-a456-426614174000",
  "results": [
    {
      "id": "123e4567-e89b-12d3-a456-426614174000",
      "workspace_id": "123e4567-e89b-12d3-a456-426614174000",
      "service_id": "123e4567-e89b-12d3-a456-426614174000",
      "conversation_id": "123e4567-e89b-12d3-a456-426614174000",
      "call_sid": "text",
      "eval_key": "text",
      "eval_type": "assertion",
      "assertion_kind": "text",
      "metric_key": "text",
      "status": "passed",
      "passed": true,
      "score": 1,
      "expected": null,
      "actual": null,
      "rationale": "text",
      "justification": "text",
      "cited_turns": [],
      "computed_at": "2026-01-01T00:00:00.000Z",
      "created_at": "2026-01-01T00:00:00.000Z"
    }
  ]
}

Returns all persisted eval verdicts for a completed call.

Permissions

All production eval endpoints require workspace authentication.

Operation
Required Permission

List, get definitions

Service view

Create, update, delete definitions

Service update

Evaluate a call

Service update

Get eval results

Service view

Assertion Kinds

Kind
Behavior

transcript_contains / must_contain

Passes if the expected phrase appears in the transcript (case-insensitive)

transcript_not_contains / must_not_contain

Passes if the expected phrase does NOT appear in the transcript

tool_called

Passes if the expected tool was called during the conversation

final_state

Passes if the conversation ended in the expected state

llm_judge (default)

An AI judge evaluates the transcript against the assertion's expected text

For transcript_contains and tool_called, the expected value can be provided as the expected field or within params (as phrase, text, tool_name, or state depending on the kind).

Metric Eval Expectations

Metric evals compute the metric value for the call and compare it against the expected threshold:

Expected Format
Behavior

{"gte": N} or {"min": N}

Passes if value >= N

{"lte": N} or {"max": N}

Passes if value <= N

{"equals": V}

Passes if value equals V

{"contains": S}

Passes if string representation contains S

Bare value

Passes if value equals the expected value

null / omitted

No threshold check; the metric value is computed and returned without a pass/fail judgment

Multiple threshold keys can be combined (e.g. {"gte": 0.5, "lte": 1.0}).

PreviousVoice Judge
NextSimulation Coverage

Last updated

Was this helpful?