AI Agent Governance: Why Control Must Move to the Point of Action

Key Takeaways

• Agents take actions, so governance has to operate at runtime, not only at model-review time.
• Scoped credentials and role-based access at the point of action are the floor of agent governance.
• Per-action audit logs should capture the decision chain, not just the final tool call.
• Human-in-the-loop gates exist to protect irreversible and high-stakes actions.
• When agent work lives in deterministic apps, governance becomes structural instead of bolted on.

What Is AI Agent Governance?

AI agent governance is the set of identity, access, audit, and approval controls that decide what an agent can do, on what data, under what conditions, and who answers for the outcome. It operates where the agent acts, at the moment a tool is called or a record is written, rather than where the model is trained or scored.

That is a different job from the one most AI programs were built to do. Traditional enterprise AI governance sets org-level policy: a risk taxonomy, model cards, bias and safety testing, an approval process before a model ships. It governs outputs. It asks whether a response is accurate, fair, and lawful. Agent governance has to govern actions, because an agent can move money, delete a row, or send a message on its own authority. Palo Alto Networks draws the same line, separating output risk from action risk, where agents may initiate transactions or modify records autonomously.

The two are complementary, and confusing them is the most common failure. Org-level policy tells you an agent is allowed to exist. Agent-level governance decides what that specific agent, holding delegated authority, is permitted to do at 3 a.m. with nobody watching. If you want the definition of what an AI agent is in this context, it is a system that initiates actions and calls tools under delegated authority, not one that returns text for a person to act on.

AI governance and AI agent governance differ across five dimensions:

• Scope. Org-level: the whole AI program and model portfolio. Agent-level: a single agent's delegated authority.
• Focus. Org-level: output risk, is the response accurate, fair, lawful. Agent-level: action risk, is the action permitted, scoped, reversible.
• Key controls. Org-level: risk taxonomy, model cards, bias and safety testing, approval to ship. Agent-level: scoped credentials, RBAC per action, per-action audit, approval gates.
• Accountability layer. Org-level: the model and org policy. Agent-level: the acting identity, per action.
• Enforcement point. Org-level: pre-deployment review. Agent-level: runtime, at the point of action.

Why This Matters Now

For a decade, the enterprise AI risk that security teams modeled was mal-information: a model that says something false, biased, or leaking. Agents move the risk to mal-action. A compromised agent, in the Cloud Security Alliance's summary of Five Eyes guidance, reasons about how to pursue attacker-controlled goals and can take consequential actions in the world with limited or intermittent human oversight. The output being wrong is no longer the worst case. The action being taken is.

The same guidance names five risk categories worth holding in your head when you review any agent deployment:

• Privilege escalation. An agent inherits or acquires more access than the task needs, then acts on all of it.
• Design and configuration flaws. Tool wiring, prompt scaffolding, or permission setup that quietly permits actions no one intended.
• Behavioral unpredictability. The same input produces different actions across runs, because the model reasons fresh each time.
• Cascading structural failures. One agent's action triggers another's, and a small error propagates across systems.
• Accountability deficits. When something goes wrong, no one can reconstruct who acted, with what authority, and why.

Notice that behavioral unpredictability and accountability deficits are properties of how the agent runs, not of the model's weights. You cannot test them out of a model before deployment. They only show up at the point of action, which is exactly why governance has to live there. The CSA note recommends teams use its MAESTRO threat model to drive control selection, because the seven MAESTRO layers map onto these Five Eyes categories.

A caution on frameworks. The CSA note also reports that the current U.S. administration directed NIST to revise the AI Risk Management Framework to remove specific risk-oriented content. The NIST AI RMF and its Govern, Map, Measure, Manage functions are still a sound scaffold for thinking about AI risk, but treating any single framework as a fixed compliance checklist is a mistake when the frameworks themselves are moving. Build controls that hold regardless of which document is current.

Four Controls That Do the Work

Strip away the framework language and agent governance comes down to four controls, all enforced where the agent acts. None of them is exotic. What is new is the demand to apply them per action, on an actor that decides for itself what to do next.

Scoped credentials

An agent should never hold a standing credential broader than the task in front of it. The pattern is just-in-time, least-privilege access issued through a credential proxy: the agent asks to act, the proxy checks the request against policy, and it hands over a narrowly scoped token that expires. The agent never sees the raw secret, and the grant is tied to a specific tool and data source. This is the single most effective control, because it caps blast radius before anything else fails. Palo Alto Networks puts least privilege at the top of its implementation checklist for the same reason.

RBAC at the point of action

Role-based access control is old news for human users. The move that matters is binding it to the agent as a first-class identity, then enforcing it at the moment of the tool call, not at login. A support-triage agent can read tickets and draft replies. It cannot issue refunds. That boundary should be checked by the system every time the agent reaches for the refund tool, and denied by default when a role is not present. Treating an agent as its own principal, rather than borrowing a human's session, is what makes this auditable later.

Per-action audit logging

Most logging captures the tool call: agent X called API Y at time Z. That answers what happened and misses why. A useful agent audit log captures the decision chain, the reasoning and the inputs that led to the action, alongside the action and its side effects. This is where governance and agent observability meet: you want a record complete enough that a reviewer can reconstruct the moment of action months later. If you cannot answer who acted, with what authority, on what data, and why, you have an accountability deficit in the exact sense the Five Eyes guidance warns about.

Human-in-the-loop gates

Some actions are irreversible or high-stakes enough that no amount of testing justifies full autonomy. Sending money above a threshold. Deleting production data. Emailing a regulator. For these, the agent should stop and require a named human to approve, with the approval itself logged. The engineering discipline is deciding the threshold deliberately rather than by accident, and making the gate a property of the action, so it fires no matter which agent or path reaches it. A good gate is boring. It exists so that the one time it matters, the irreversible thing does not happen unattended.

How the Controls Fit Together

In practice these controls form a single path that every agent action travels. A text schematic:

agent decides to act
|
v
[ credential proxy ] -- checks role (RBAC) --> deny if not permitted
| issues short-lived, task-scoped token
v
[ human-approval gate ] -- fires only for high-risk / irreversible actions
| (approval recorded)
v
[ scoped action ] --> database / API / SaaS tool
|
v
[ audit log ] <-- captures decision chain + action + side effects
(who, what, when, on what data, why)

Four Misconceptions Worth Killing

An agent is just a service account. A service account has fixed permissions and does what it is told. An agent decides what to do, chains actions, and can be steered by a crafted input. Give it a service account's standing access and you have handed broad, always-on privilege to a component that improvises. Agents need identity plus scoped, revocable authority, not a static key.

Governance is a pre-deployment gate. Reviewing an agent before launch is necessary and not sufficient. The risks that matter, behavioral drift and cascading actions, appear at runtime and change with every input. If your controls stop at the launch review, you are governing the version you tested, not the agent that is running.

Monitoring model outputs is governing the agent. Watching what a model says tells you about output quality. It says little about the actions taken downstream. Two agents with identical outputs can have very different side effects depending on what they were permitted to touch. Govern the actions, not just the tokens.

A policy document equals enforcement. A written policy that agents run least-privilege is a statement of intent. Enforcement is a system that denies the over-scoped request at the moment it is made. The gap between the two is where incidents live. Controls that are not enforced in code are aspirations.

What Agent Governance Does Not Do

Being honest about the limits is part of the argument. The controls in this piece reduce operational blast radius and create an audit trail. They do not eliminate prompt injection; a well-crafted input can still steer an agent toward an action it is technically permitted to take, which is why scope matters so much. They do not replace legal review, model risk management, or the org-level policy work of enterprise AI governance. And they do not, on their own, guarantee regulatory compliance. What they do is make the difference between an agent whose actions you can constrain and reconstruct, and one you cannot. This piece is about runtime action control. Agent evaluation, red-teaming, and model selection are adjacent problems that deserve their own treatment.

Where Major Stands

Most of the market treats agent governance as a layer you wrap around an agent framework: a control plane here, an agent registry there, a runtime policy engine bolted on top. That works, and it is better than nothing. Our position is that the wrapper is fighting the architecture. If an agent re-reasons every task inside the model and hides its work in a prompt context, the governance layer is always reconstructing what the agent did from the outside, after the fact.

Major takes a different route. When an agent figures out a repeatable piece of work, it builds an app for that piece, and the app is where the work runs from then on. Because the work lives in a deterministic app rather than a fresh round of reasoning, the four controls are properties of the app, not a net thrown over it. Scoped credentials flow through the credential proxy. RBAC is enforced at the point of action. Every action writes to a per-action audit log. High-risk actions hit an approval gate. The app is the control surface a CISO inspects and the execution layer the agent runs through, which is what makes governance structural instead of after-market. Reason once, run forever means the repeatable action is code you can read, not a probability you re-sample.

This is the claim an agent that reasons every run cannot make. A learned skill is still a behavior the model performs again each time, which means it has to be observed to be trusted. An app is code the model wrote once and then stepped out of, which means it can be governed the way you already govern software. We think that distinction is the whole game for enterprises deciding how to adopt agents without building a shadow system no one can audit.

If you are weighing how to keep autonomous agents inside real controls, see how Major enforces scoped credentials and per-action audit inside the app layer.

Related articles

• AI Agent Observability Isn't a LangSmith Feature
• Enterprise AI Agents: What "Enterprise-Grade" Actually Means
• What Is an AI Agent? Definition, Types, and How to Build One