> ## Documentation Index
> Fetch the complete documentation index at: https://docs.uselamina.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent Runtime Benchmarks

> Metrics, scenarios, and pass criteria for measuring Lamina as an agent-native creative runtime.

Use this guide to measure whether an AI agent can install Lamina, authenticate, create assets, and retrieve outputs without custom recovery code.

## What To Measure

Track every agent run against five stages:

| Stage | Success Signal | Failure Stage |
| - | - | - |
| Install | Client discovers `/mcp/agent/v2` and completes dynamic registration or manual install | `install` |
| Auth | OAuth authorization and token exchange succeed for the selected workspace | `auth` |
| Input | The agent supplies enough brief, brand, and asset context to start a run | `input` |
| Runtime | Lamina queues and completes the selected creative workflow | `runtime` |
| Output | The agent receives usable final assets or structured outputs | `output` |

The hosted MCP runtime emits benchmark-oriented telemetry events when server telemetry is enabled with `POSTHOG_SERVER_API_KEY`.

| Event | When It Fires |
| - | - |
| `agent_runtime.install.discovery_challenged` | A remote MCP client discovers that `/mcp/agent/v2` requires OAuth |
| `agent_runtime.install.client_registered` | Dynamic MCP OAuth registration succeeds |
| `agent_runtime.install.failed` | Dynamic registration fails |
| `agent_runtime.auth.authorize_redirected` | The OAuth authorize request validates and redirects to consent |
| `agent_runtime.auth.approved` | A signed-in user approves workspace access |
| `agent_runtime.auth.token_issued` | Authorization code or refresh-token exchange succeeds |
| `agent_runtime.auth.succeeded` | A bearer token is accepted for `/mcp/agent/v2` |
| `agent_runtime.auth.failed` | OAuth authorization, token exchange, or bearer validation fails |
| `agent_runtime.tool_call.completed` | An MCP tool call completes and is classified |

Every `agent_runtime.tool_call.completed` event includes:

```json theme={null}
{
  "tool_name": "lamina_plan",
  "success": true,
  "outcome": "success",
  "failure_stage": null,
  "failure_category": null,
  "duration_ms": 1420,
  "auth_mode": "oauth",
  "run_status": "awaiting_approval",
  "needs_input": false,
  "output_count": 0
}
```

## Benchmark Scenarios

Run these scenarios for each supported MCP client before calling the distribution ready.

| Scenario | Required Proof |
| - | - |
| Hosted OAuth install | The client discovers metadata, registers or uses its configured client, shows consent, receives tokens, and lists the Lamina tools |
| First image run | The agent plans from a short prompt, obtains approval, calls `lamina_execute`, and receives an image through `lamina_status` |
| First video run | The agent plans and executes a video task, then waits or polls until a terminal result |
| Brand-aware planning | The agent calls `lamina_brand_context`, passes its `brandProfileId` to `lamina_plan`, and preserves the same workspace context |
| Multi-step pipeline | The agent plans an image-to-video task and Lamina executes the frozen cross-step bindings without the client copying intermediate URLs |
| Clarification loop | An intentionally underspecified request returns `needs_clarification`; the agent asks `questions[]` and plans again |
| Auth recovery | An expired or insufficient-scope token yields a clear OAuth error and the client can reauthorize |

## Pass Criteria

Use these thresholds for the preferred-runtime scorecard:

| Metric | Target |
| - | - |
| Install success rate | 95% or higher per supported client |
| First successful generation time | Under 5 minutes from clean client install |
| Tool call auth failure rate | Under 1% after successful install |
| Clarification-loop rate | Tracked separately from hard failures |
| Run completion rate | 90% or higher for benchmark workflows |
| Webhook or polling delivery success | 99% for terminal run visibility |
| Output usability rate | 95% of completed runs have at least one usable output |

Do not count `needs_clarification` as a runtime failure. It is a planning outcome and should be optimized by improving examples and prompt mapping. Likewise, required inputs on an `awaiting_approval` plan are not runtime failures.

## Suggested Benchmark Record

Store one record per client, scenario, and run:

```json theme={null}
{
  "client": "claude-code",
  "scenario": "first-image-run",
  "startedAt": "2026-04-23T07:00:00.000Z",
  "completedAt": "2026-04-23T07:02:11.000Z",
  "installSucceeded": true,
  "authSucceeded": true,
  "toolCalls": [
    { "name": "lamina_plan", "outcome": "success", "durationMs": 812 },
    { "name": "lamina_execute", "outcome": "success", "durationMs": 931 },
    { "name": "lamina_status", "outcome": "success", "durationMs": 1304 }
  ],
  "runId": "00000000-0000-0000-0000-000000000000",
  "finalStatus": "completed",
  "outputCount": 1,
  "failureStage": null,
  "notes": "Clean install, OAuth consent, one image output."
}
```

## Dashboard Breakdown

At minimum, build dashboard cards for:

* Install starts, successful registrations, and failed registrations by MCP client
* OAuth approvals, token issues, bearer-token failures, and insufficient-scope failures
* Tool-call success rate by `tool_name`
* `needs_clarification` and missing-required-input rates by requested modality
* Runtime failure rate by workflow/app when available
* Empty-output and failed-output rate after terminal `completed` status
* Time from first install event to first completed output

## Related Guides

* [MCP OAuth Install](/guides/mcp-oauth-install)
* [Agent Integration Patterns](/guides/agent-integration-patterns)
* [Handle Long-Running Executions](/guides/handle-long-running-executions)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.