What is AI agent observability?

AI agent observability is the ability to reconstruct, after the fact, the decision an agent made at the moment it took an action: what it saw, what it could do, why it chose, and what changed as a result. Four things make that possible: decision context per tool call, reasoning captured before the call runs, side-effect logging with reversibility metadata, and replay against the same instructions.

How is agent observability different from LLM observability?

LLM observability records how a model generated text: prompts, tokens, latency, cost and call traces. Agent observability records how an agent decided: the context it read, its stated reasoning before each tool call, the side effects it caused, and a replay path. They are different categories, and most tools sell the first as the second. Major keeps one record because the agent reasons and the app executes.

Can you audit an AI agent decision after the fact?

Only if the platform captured the right things at decision time. If it logged inputs and outputs alone, you cannot reconstruct why the agent acted. If it logged the reasoning, the tools available and the side effects taken, you can. Re-prompting the model later to ask why is a fresh generation, not evidence. On Major the record is written at the point where the agent acts and stored in the app.

What metrics matter for AI agent monitoring?

Latency, cost and tool-call failure rate are necessary and insufficient. Governance metrics matter more to a CISO: decisions per agent per day, irreversible actions per session, and human-override rate. The first set keeps agents running. The second tells you whether they are safe to keep running. Major can report both because each action is an attributable app query with a scoped identity.

Is agent observability the same as agent evaluation?

No. Observability tells you what the agent decided and why. Evaluation tells you whether the decision was correct. A perfect decision record can sit next to a bad outcome, so observability is a precondition for governing agents and not a substitute for it. Major covers the observability and enforcement side, with scoped credentials, role checks at the query layer and audit written where the agent acts.

AI Agent Observability Isn't a LangSmith Feature

LLM observability tools don't observe agents. They observe tokens. Here's what real agent observability requires, and the questions to ask any vendor before you ship agents into production.

Rahul RamakrishnanJune 14, 2026Updated October 21, 2026

Operations engineer reviewing system activity on a computer at a monitoring desk

What is the difference between agent observability and LLM observability?

LLM observability tools record what a model said. Agent observability records why an agent acted, so you can reconstruct the decision after the fact. Most platforms sold as agent observability cover the first and leave an audit gap on the second. Major closes that gap structurally: the agent reasons, then acts through deterministic apps that log every action at the query layer, so reasoning and execution land in one record.

Consider a refund agent that issues a $4,200 refund it should have escalated. The log shows a dispute came in, a refund tool was called, and money moved. It does not show what the agent read before choosing, which policy it weighed, what else it could have done, or whether a human could have stopped it. The action is in the record. The decision is not.

Here is the definition used in this piece. Agent observability is the ability to reconstruct, after the fact, the decision an agent made at the moment it acted: what it saw, what it could do, why it chose, and what changed as a result.

Why LLM observability tools miss the decision

Langfuse, LangSmith, Arize Phoenix, Helicone and Weights & Biases capture prompts, tokens, latency, cost and call traces. If your problem is a model that hallucinates, drifts or runs up a bill, that is the right category. An agent is a different system. It reads context, picks a path and changes the state of the world. Tokens tell you what it said. They do not tell you why it acted, and they give you no way to stand the decision back up and inspect it.

Datadog frames AI observability as an extension of application performance monitoring, and for traces, latency and cost that lineage is a strength. LangSmith treats observability as trace visualization of LLM calls. Arize extends ML monitoring, with strong accuracy and drift metrics, into agents. Each is good at the system it was built to watch. None of them natively records reasoning at decision time or reversibility, and that is the part an auditor asks about.

The four things agent observability requires

Decision context per tool call. A snapshot of what the agent could see when it acted: the data read, the state of the systems it was about to change, the constraints in front of it. A trace proves a call happened. Context shows whether the call was reasonable.
Stated reasoning, captured at decision time. Asking the model afterward why it did something produces a fresh generation, not evidence. The reasoning has to be written down before the action runs.
Side effects with reversibility metadata. Log every external write and mark whether it can be undone. A refund, a sent email and an updated contract sit in a different risk class than a drafted summary.
Replay against the same instructions. Rerun the agent on the same inputs and instructions and see whether it reaches the same decision. This is the primitive most vendors lack.

The NIST AI Risk Management Framework points the same way. MEASURE 2.8 asks that risks associated with transparency and accountability are examined and documented, and the framework lists "Accountable and Transparent" among the characteristics of trustworthy AI (NIST AI 100-1). You cannot document a decision you never recorded.

Eight questions to ask any agent platform vendor

At a past decision point, can you show the full context the agent read before it acted?
Can you retrieve the stated reasoning for a specific tool call, captured then and not regenerated now?
Does the log record every external write and mark which were irreversible?
Can you replay a past run against the same instructions and compare the result?
Is the decision record enforced by the platform, or does each agent need hand instrumentation?
Does the trail cover reasoning and execution, or only model calls?
Can the record be exported to your SIEM in a format your security team already reads?
Are governance metrics available, such as decisions per agent per day, irreversible actions per session and human-override rate, and not only latency and cost?

Question eight is where most monitoring stops. Latency and failure rate measure the agent as infrastructure. Override rate and irreversible actions measure it as an actor.

How to build observable agents on Major

The structural answer is to stop letting the model be the execution layer. On Major, an agent works out a repeatable step once, builds an app for it, and then runs the app. The app is code, so the step runs the same way every time and writes its own logs.

For a refund workflow, the build looks like this:

Connectors. Stripe for payments, Zendesk for disputes, Slack for escalation, each with scoped credentials rather than a shared admin key.
The app that holds state. A refunds app with managed database tables for disputes, policy checks, decisions and outcomes. History lives in the app, not in a context window that disappears.
The agent and its judgment. The agent decides only what needs judgment: is this dispute within policy, and does it need a human. Policy lookups, amount thresholds and the refund call itself run as deterministic app queries.
The trigger. A workflow fires on new disputes and routes anything above the threshold to a person for approval in Slack.
Governance. Role-based access is enforced where the query runs, every action is attributed to a scoped identity, and the audit log can be exported to your security tooling.
Replay. Because the deterministic stretch is code with typed inputs, you can rerun it against the recorded inputs. Only the judgment step needs a model, and its stated reasoning is stored alongside the action.

Start building this on Major and the same pattern applies to any workflow that crosses several systems of record.

What we hold as a position at Major

Observability should be a property of the platform, not a flag a team remembers to set. When repeatable work runs as apps, the record of what happened is a side effect of running, and it is the same record whether a person or an agent triggered the step. Reason once, run forever also means audit once: the control lives in code that is inspected, not in a prompt that is hoped about.

Scope matters. A workflow simple enough to fit in one prompt does not need this. It earns its place when the workflow is long, branches, touches several systems and someone will eventually explain to an auditor what happened in March. Major does not remove the model either. The judgment step still runs through it, and its reasoning is still generated text that you read critically.

What this article does not cover

This is an argument about observability, not evaluation. Observability tells you what the agent decided and why. It does not tell you whether the decision was correct, and a perfect record can sit quietly next to a bad outcome. Evaluation is a separate and harder problem that deserves its own piece.

Start building on Major

Build agents whose repeatable work runs as governed apps, with scoped credentials, role-based access and an audit trail written where the agent acts.

Frequently asked questions

What is AI agent observability?: AI agent observability is the ability to reconstruct, after the fact, the decision an agent made at the moment it took an action: what it saw, what it could do, why it chose, and what changed as a result. Four things make that possible: decision context per tool call, reasoning captured before the call runs, side-effect logging with reversibility metadata, and replay against the same instructions.
How is agent observability different from LLM observability?: LLM observability records how a model generated text: prompts, tokens, latency, cost and call traces. Agent observability records how an agent decided: the context it read, its stated reasoning before each tool call, the side effects it caused, and a replay path. They are different categories, and most tools sell the first as the second. Major keeps one record because the agent reasons and the app executes.
Can you audit an AI agent decision after the fact?: Only if the platform captured the right things at decision time. If it logged inputs and outputs alone, you cannot reconstruct why the agent acted. If it logged the reasoning, the tools available and the side effects taken, you can. Re-prompting the model later to ask why is a fresh generation, not evidence. On Major the record is written at the point where the agent acts and stored in the app.
What metrics matter for AI agent monitoring?: Latency, cost and tool-call failure rate are necessary and insufficient. Governance metrics matter more to a CISO: decisions per agent per day, irreversible actions per session, and human-override rate. The first set keeps agents running. The second tells you whether they are safe to keep running. Major can report both because each action is an attributable app query with a scoped identity.
Is agent observability the same as agent evaluation?: No. Observability tells you what the agent decided and why. Evaluation tells you whether the decision was correct. A perfect decision record can sit next to a bad outcome, so observability is a precondition for governing agents and not a substitute for it. Major covers the observability and enforcement side, with scoped credentials, role checks at the query layer and audit written where the agent acts.