Agent Runtime Benchmarks

Use this guide to measure whether an AI agent can install Lamina, authenticate, create assets, and retrieve outputs without custom recovery code.

What To Measure

Track every agent run against five stages:

Stage	Success Signal	Failure Stage
Install	Client discovers `/mcp/agent/v2` and completes dynamic registration or manual install	`install`
Auth	OAuth authorization and token exchange succeed for the selected workspace	`auth`
Input	The agent supplies enough brief, brand, and asset context to start a run	`input`
Runtime	Lamina queues and completes the selected creative workflow	`runtime`
Output	The agent receives usable final assets or structured outputs	`output`

The hosted MCP runtime emits benchmark-oriented telemetry events when server telemetry is enabled with POSTHOG_SERVER_API_KEY.

Event	When It Fires
`agent_runtime.install.discovery_challenged`	A remote MCP client discovers that `/mcp/agent/v2` requires OAuth
`agent_runtime.install.client_registered`	Dynamic MCP OAuth registration succeeds
`agent_runtime.install.failed`	Dynamic registration fails
`agent_runtime.auth.authorize_redirected`	The OAuth authorize request validates and redirects to consent
`agent_runtime.auth.approved`	A signed-in user approves workspace access
`agent_runtime.auth.token_issued`	Authorization code or refresh-token exchange succeeds
`agent_runtime.auth.succeeded`	A bearer token is accepted for `/mcp/agent/v2`
`agent_runtime.auth.failed`	OAuth authorization, token exchange, or bearer validation fails
`agent_runtime.tool_call.completed`	An MCP tool call completes and is classified

Every agent_runtime.tool_call.completed event includes:

{
  "tool_name": "lamina_plan",
  "success": true,
  "outcome": "success",
  "failure_stage": null,
  "failure_category": null,
  "duration_ms": 1420,
  "auth_mode": "oauth",
  "run_status": "awaiting_approval",
  "needs_input": false,
  "output_count": 0
}

Benchmark Scenarios

Run these scenarios for each supported MCP client before calling the distribution ready.

Scenario	Required Proof
Hosted OAuth install	The client discovers metadata, registers or uses its configured client, shows consent, receives tokens, and lists the Lamina tools
First image run	The agent plans from a short prompt, obtains approval, calls `lamina_execute`, and receives an image through `lamina_status`
First video run	The agent plans and executes a video task, then waits or polls until a terminal result
Brand-aware planning	The agent calls `lamina_brand_context`, passes its `brandProfileId` to `lamina_plan`, and preserves the same workspace context
Multi-step pipeline	The agent plans an image-to-video task and Lamina executes the frozen cross-step bindings without the client copying intermediate URLs
Clarification loop	An intentionally underspecified request returns `needs_clarification`; the agent asks `questions[]` and plans again
Auth recovery	An expired or insufficient-scope token yields a clear OAuth error and the client can reauthorize

Pass Criteria

Use these thresholds for the preferred-runtime scorecard:

Metric	Target
Install success rate	95% or higher per supported client
First successful generation time	Under 5 minutes from clean client install
Tool call auth failure rate	Under 1% after successful install
Clarification-loop rate	Tracked separately from hard failures
Run completion rate	90% or higher for benchmark workflows
Webhook or polling delivery success	99% for terminal run visibility
Output usability rate	95% of completed runs have at least one usable output

Do not count needs_clarification as a runtime failure. It is a planning outcome and should be optimized by improving examples and prompt mapping. Likewise, required inputs on an awaiting_approval plan are not runtime failures.

Suggested Benchmark Record

Store one record per client, scenario, and run:

{
  "client": "claude-code",
  "scenario": "first-image-run",
  "startedAt": "2026-04-23T07:00:00.000Z",
  "completedAt": "2026-04-23T07:02:11.000Z",
  "installSucceeded": true,
  "authSucceeded": true,
  "toolCalls": [
    { "name": "lamina_plan", "outcome": "success", "durationMs": 812 },
    { "name": "lamina_execute", "outcome": "success", "durationMs": 931 },
    { "name": "lamina_status", "outcome": "success", "durationMs": 1304 }
  ],
  "runId": "00000000-0000-0000-0000-000000000000",
  "finalStatus": "completed",
  "outputCount": 1,
  "failureStage": null,
  "notes": "Clean install, OAuth consent, one image output."
}

Dashboard Breakdown

At minimum, build dashboard cards for:

Install starts, successful registrations, and failed registrations by MCP client
OAuth approvals, token issues, bearer-token failures, and insufficient-scope failures
Tool-call success rate by tool_name
needs_clarification and missing-required-input rates by requested modality
Runtime failure rate by workflow/app when available
Empty-output and failed-output rate after terminal completed status
Time from first install event to first completed output

Overview

Install

Concepts

API Reference

Guides

Agent Runtime Benchmarks

What To Measure

Benchmark Scenarios

Pass Criteria

Suggested Benchmark Record

Dashboard Breakdown

​What To Measure

​Benchmark Scenarios

​Pass Criteria

​Suggested Benchmark Record

​Dashboard Breakdown

​Related Guides

What To Measure

Benchmark Scenarios

Pass Criteria

Suggested Benchmark Record

Dashboard Breakdown

Related Guides