Observe the whole agent run—not just its final answer
Correlate orchestration, model calls, retrieval, tools, retries, handoffs, approvals, tokens, and business outcomes in one trace. Then enforce runtime limits before a stalled agent becomes an incident or an unexplained bill.
One run, one connected execution graph
Traditional APM sees HTTP calls and exceptions. Agent observability adds the reasoning-loop structure and the outcome semantics needed to interpret them.
Start a root span when the workflow accepts a task. Every model call, retrieval, tool execution, agent handoff, queue wait, and human approval becomes a child operation. Carry the same trace identity through services using the W3C's interoperable Trace Context. OpenTelemetry's GenAI work defines agent-oriented operation and usage fields, but the project explicitly remains under active development; pin the convention version you implement rather than assuming field names will never change.
agent.run order_exception_v4 8.42s
├─ retrieval policy_index 0.31s
├─ model.chat route_exception 1,840 tok 1.26s
├─ tool.call fetch_order 0.18s
├─ tool.call fetch_carrier_event 0.46s
├─ model.chat propose_resolution 2,510 tok 1.74s
├─ approval operations_user 3.91s
└─ tool.call update_case success 0.22s
outcome=resolved retries=0 estimated_cost=0.047 EURDebug the path
Find the exact prompt version, tool argument, response, retry, or handoff that changed the result.
Explain the spend
Attribute usage to workflow, tenant, environment, model, feature, team, and completed business unit.
Control the runtime
Terminate or escalate when the run exceeds allowed turns, time, spend, errors, or side-effect scope.
Define the telemetry contract before choosing a dashboard
The contract is the durable asset. It keeps telemetry comparable when models, frameworks, providers, and visualization tools change.
Identity and version
run ID, workflow and agent name, workflow version, prompt/config version, deployment, environment, tenant or cost centre, release, and parent trace
Operation and dependency
operation type, parent/child relationship, provider, requested and resolved model, tool name, retrieval source, queue, start/end time, status, finish reason, and retry count
Usage and estimated cost
input, output, cached, cache-write, and reasoning tokens when supplied; tool units; pricing key and version; currency; estimated model, tool, and total cost
Outcome and quality
terminal state, business outcome, automated and sampled evaluation results, human approval or correction, policy decision, incident link, and useful business-unit count
Do not put prompts into ordinary logs by accident. OpenTelemetry warns that system instructions, input messages, tool arguments, and results may contain sensitive information. Yarify's recommendation is metadata-first collection, explicit content capture, field-level redaction, separate access policy, and tested deletion—not an unrestricted debug switch. See the GenAI conventions.
Token counts are evidence. They are not yet a bill.
Provider usage, effective prices, and invoiced charges arrive from different systems and at different times. Keep all three.
Calculate an immediate estimate for operational decisions, then reconcile it with billing exports. A price catalogue must be effective-dated because model names, tiers, cache rules, batch discounts, currencies, and prices change. Never overwrite old rates: a historical run must remain reproducible.
Per-run estimate
Σ(model usage × effective rate) + tool API charges + retrieval + sandbox/compute + allocated observability cost
Store the pricing version, currency, source timestamp, and any assumptions beside the result. Mark missing usage or price data instead of silently treating it as zero.
The FinOps Framework distinguishes resource-efficiency units such as cost per token from business units such as cost per transaction or case. Use both: token cost locates consumption; outcome cost tells product and finance whether the system creates value.
A runaway-loop alert is not a safety control
The runtime must enforce the boundary. Monitoring explains and escalates what happened around it.
Hard run budgets
Maximum elapsed time, turns, model calls, tool calls, tokens, estimated spend, retries, and concurrent child agents. Stop or request approval at the boundary.
No-progress detection
Fingerprint repeated tool calls and arguments, compare state transitions, and detect cycles that consume resources without moving toward a terminal outcome.
Side-effect policy
Keep read and write tools distinct. Gate high-impact or irreversible writes through deterministic authorization, idempotency keys, transaction limits, and approval.
Degraded modes
Define fallbacks for provider errors, rate limits, tool outages, and telemetry failure: retry with bounds, switch an approved route, queue safely, or escalate to a human.
Spend-velocity alerts
Alert on cost acceleration per run, workflow, tenant, model, or release—not only a monthly total after the damage is done.
Kill and contain
Operators need a tested way to cancel runs, revoke tool credentials, disable a workflow version, quarantine output, and preserve the incident trace.
OWASP's agentic threat guidance is non-binding security guidance, not a certification. It is useful for threat modelling tool misuse, identity, memory, excessive agency, and cascading failure. Translate that model into controls for the concrete tools and privileges in your system. See OWASP's agentic guidance.
Dashboards should answer operating decisions
Separate release health, runtime reliability, economics, and risk. A single token chart cannot serve engineering, product, finance, support, and security equally well.
Release health
Engineering + AI product
- task success and escalation rate by workflow version
- p50 / p95 / p99 end-to-end and model latency
- tool error, retry, and invalid-argument rate
- quality regressions against a versioned evaluation set
Unit economics
Product + FinOps
- tokens and estimated cost per successful outcome
- model, tool, retrieval, and infrastructure cost share
- allocated versus unattributed provider spend
- cost and volume variance by tenant, feature, and release
Control health
Platform + security
- budget stops, policy denials, and approval frequency
- loop, no-progress, and retry-storm detections
- privileged tool usage and rejected side effects
- trace completeness, dropped telemetry, and redaction failures
Sampling rule
Keep 100% of failed, stopped, high-cost, policy-denied, and privileged-write traces. Tail-sample routine successful runs, while retaining aggregate metrics. Verify that sampling does not hide rare tenants or low-volume workflows.
Alert rule
Page only on an actionable condition with an owner, runbook, and safe response. Use tickets or review queues for slower drift such as rising cost per success, model-route changes, and evaluation degradation.
A pilot ends with one explainable production workflow
Scope around a real run path and existing APM stack. Avoid a platform-wide instrumentation programme before the contract is proven useful.
Map the run and decisions
Choose one workflow; enumerate models, tools, queues, identities, approvals, side effects, outcomes, current logs, billing sources, and incident scenarios.
Specify and instrument
Define the versioned semantic contract, propagation boundaries, redaction rules, usage normalization, pricing catalogue, and SDK integration tests.
Reconcile and control
Validate trace completeness, compare estimates with billing, add run budgets and no-progress detection, and exercise cancellation and degraded modes.
Operate and hand over
Ship decision-specific dashboards, SLOs, alerts, runbooks, ownership, retention rules, convention upgrade notes, and a backlog for the next workflows.
Use your current backend
Export OpenTelemetry into an existing compatible APM or warehouse when it meets retention, access, query, and regional requirements.
Own the semantic layer
Keep the event contract, price history, outcome definitions, and billing reconciliation independent of one dashboard vendor.
Test observability itself
Synthetic runs should prove propagation, usage capture, redaction, budget enforcement, alert routing, and trace access before launch.
NIST AI 600-1 is voluntary risk-management guidance rather than a binding requirement. Its lifecycle approach supports continuous measurement, monitoring, incident handling, and documentation; the concrete fields, thresholds, and runtime policies above are Yarify's implementation recommendations. See the NIST GenAI Profile.
AI agent observability FAQ
What should AI agent observability capture?
Capture one correlated trace for each agent run, with child spans for orchestration, model calls, retrieval, tool calls, handoffs, queues, and approvals. Add low-cardinality identity, version, environment, timing, status, retry, token-usage, estimated-cost, and outcome fields. Prompt or response content should be opt-in and governed separately because it can contain sensitive data.
How do you calculate the cost of an AI agent run?
Record provider-reported usage for every model call, map it to the exact provider, resolved model, service tier, region, and pricing-version effective at call time, then add non-model costs such as search, tools, vector storage, sandbox compute, and observability. Label the result estimated until it is reconciled with provider billing exports and credits.
How can monitoring detect a runaway agent loop?
Use several signals together: turn and tool-call counts, repeated call fingerprints, unchanged state, repeated errors, token or spend velocity, elapsed time, and lack of progress toward a terminal state. Enforce hard limits in the runtime; an alert alone arrives too late to prevent spend or side effects.
Can we use our existing APM platform?
Usually yes if it accepts OpenTelemetry traces, metrics, and logs and can retain the dimensions you need. The missing work is normally instrumentation, semantic normalization, pricing and billing reconciliation, agent-specific dashboards, and policy controls—not another telemetry backend.
Should prompts and model responses be stored in traces?
Not by default. Start with metadata, hashes, classifications, sizes, and stable references. Enable content capture only for an approved purpose with redaction, access control, retention limits, regional storage, and audit logging. Sampling must preserve failed and high-risk runs without turning the observability store into an uncontrolled copy of business data.
What is the most useful cost KPI for an agent?
Cost per successful business outcome is usually more useful than cost per token. Tokens diagnose model consumption, but an apparently cheap run that fails, retries, or requires extensive correction can be more expensive than a higher-token run that completes the task correctly.
Primary sources and status
Reviewed 4 September 2026. W3C Trace Context is a technical Recommendation. OpenTelemetry's GenAI conventions are open technical specifications under active development. NIST, FinOps, and OWASP materials are voluntary guidance, not binding law or a certification. Yarify's trace schema, cost method, thresholds, retention model, and runtime controls are implementation recommendations that must be adapted to each system.
- OpenTelemetry GenAI semantic conventionsThe upstream repository for GenAI spans, metrics, events, agents, model calls, tool execution, and MCP telemetry; these conventions remain under active development.
- OpenTelemetry — Inside the LLM Call: GenAI ObservabilityOfficial worked example of agent, model, and tool spans plus token and latency metrics using the GenAI conventions.
- W3C Trace Context RecommendationThe interoperable traceparent and tracestate format for correlating work across service and vendor boundaries.
- NIST AI 600-1 — Generative AI ProfileVoluntary companion to the AI RMF covering lifecycle monitoring, incident disclosure, measurement, and management of generative-AI risks.
- FinOps Framework — Unit EconomicsCommunity framework for connecting technology cost to useful units such as cost per token, transaction, assist, or business outcome.
- OWASP — Agentic AI Threats and MitigationsNon-binding threat-model guidance for agentic systems, including excessive agency, tool misuse, memory, identity, and cascading failures.