AI Model Governance Can't See What Your Agent Did

What a CISO actually needs to know

A firm can hold a complete model inventory, clear its validation review, and still have no record of what any of its agents did last Tuesday. Model governance registers artifacts. Agents take actions. The distance between them is where the incident happens.

The model AI governance framework is the lifecycle discipline of registering, validating, documenting, monitoring, and retiring every model an organisation runs, most often structured against NIST AI RMF 1.0 or ISO/IEC 42001:2023.

Four things worth holding onto before the detail:

• Model governance was built to govern a scoring artifact, and it does that job well.
• An agent's tool call, credential use, and action attribution have no control in that framework.
• The Federal Reserve superseded SR 11-7 with SR 26-2 on 17 April 2026, and put agentic AI out of scope.
• Adding an LLM row to the model inventory is filing. It produces no record of any action.

What AI model governance actually is

Model governance is the set of controls that keeps every model an organisation runs an accountable asset with a named owner, a risk tier, a documented purpose, and a monitored performance history. Vendors differ on labels. The lifecycle underneath is consistent, and Atlan's account is representative of how the discipline is practised:

1. Registration and cataloguing. The model enters an inventory with an owner, a purpose, and a risk level.
2. Versioning. Each deployed version is tracked with its own performance metrics.
3. Pre-production approval gates. Intake materials and risk assessments are reviewed before release.
4. Model cards. Intended use, training data, evaluation results, and limitations are documented.
5. Continuous drift and performance monitoring. Performance, bias, and fairness are assessed against thresholds that trigger review.
6. Deprecation and decommissioning. Retired models are marked inactive with their audit history preserved.

This works. A validation function running conceptual soundness review, ongoing monitoring, and outcomes analysis catches failures nothing else in an enterprise catches: a model that was sound at approval and has quietly stopped being sound, a training set that never represented the population it now scores, a vendor model nobody can explain. Those failures are silent and they compound. Drift monitoring is the only control in most organisations pointed at them, and nothing here argues for weakening it.

The claim is about coverage.

Where the vocabulary comes from

Model governance governs artifacts because it was designed for artifacts, in banking, fifteen years ago. The SR 11-7 supervisory guidance of 4 April 2011 is where most of this field's vocabulary comes from, and its definition is worth reading closely:

For the purposes of this document, the term model refers to a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates. A model consists of three components: an information input component, which delivers assumptions and data to the model; a processing component, which transforms inputs into estimates; and a reporting component, which translates the estimates into useful business information.

Three components: input, processing, reporting. The guidance then extends the definition to approaches whose inputs are qualitative or based on expert judgment, "provided that the output is quantitative in nature."

That clause is load-bearing. It is the boundary of the entire discipline. An agent that reads a contract, decides the payment terms were met, drafts a collections note, and updates a CRM record produces no quantitative estimate at any point. It produces a side effect in a system of record. The framework the reader is being asked to extend over agents was defined, at its foundation, around something agents do not produce.

Two further facts most pages on this topic have not caught up with. SR 11-7 is no longer current guidance. The Federal Reserve, FDIC, and OCC issued SR 26-2, Revised Guidance on Model Risk Management on 17 April 2026, superseding SR 11-7 and SR 21-8, and most relevant to banking organisations with over $30 billion in total assets. The commonly cited URL for the old letter now 404s, so the historical attachment linked above is the only live path to the original text.

The revised guidance also narrows the definition rather than widening it. SR 26-2 defines a model as "a complex quantitative method, system, or approach that applies statistical, economic, or financial theories to process input data into quantitative estimates," and excludes "deterministic rule-based processes and software where there are no statistical, economic, or financial theories underpinning their design or use." Footnote 3 goes further: "Generative AI and agentic AI models are novel and rapidly evolving. As such, they are not within the scope of this guidance."

The regulator that authored this vocabulary has said, in April 2026, that agentic AI sits outside it, and that a banking organisation's own risk management practices should determine the controls for anything not covered. That is an instruction to build the control somewhere else, which is where we landed from the engineering side in governing what the agent does.

The argument: what a model card cannot tell you

A mature model risk programme can pass its audit while holding no record of what any agent actually did. The audit examines artifacts, the artifacts are accurate, and the actions were never in scope.

A traced agent action

Here is one agent action of the kind already running in finance operations, traced against the controls a model governance programme has in place.

1. The agent reads an overdue invoice. Which control governs its read access to the full receivables table rather than one account? None. Data lineage documents the model's training and input data, not the records an agent reached at run time.
2. The agent retrieves the contract and interprets the payment terms. Which control governs whether that interpretation was correct? The model card offers evaluation results measured on a benchmark set months before this contract existed. No control observes this reading.
3. The agent decides the terms were breached. Which control records the reasoning? Drift monitoring would catch a scoring model degrading against a threshold. There is no threshold here and no score.
4. The agent calls a tool to draft and send a collections email. Which control governs the tool call? None. The phrase does not appear in the model governance vocabulary.
5. The agent writes the status change to the CRM under a service credential. Which control governs that credential's scope and records this use? Identity systems log the service account. They do not log which agent, on which task, under whose delegated authority.
6. A customer disputes the note six weeks later. Which artifact reconstructs the decision context at the moment of action? Model card, validation report, drift chart, inventory row. None contains the answer.

Every control in that trace catches real failures. None is pointed at the thing that went wrong.

The three things with no owner

Three surfaces recur in every agent deployment and appear in no model governance framework.

The tool call. An agent's capability is defined by its tool inventory, which changes when someone edits a config file. The model card describes a model whose available actions it does not know.

The credential. Agents act with delegated authority, usually a long-lived service credential scoped more broadly than any single task requires. Model risk does not cover credentials, and identity governance does not cover the agent's decision to use one. It falls between two functions, which is why it stays open. We have written separately about scoped credentials and the agent threat model.

The action's audit record. Model monitoring records latency, output distribution, and evaluation scores. It does not record side effects. If your agent updated 400 records overnight, the monitoring dashboard shows a normal-looking day.

What good looks like in practice

These layer. Agent controls sit on top of model controls, and a regulated organisation needs both.

| What you need to know | Model governance control | Does it cover agent actions? | Artifact produced | |---|---|---|---| | Is the model performing as designed | Ongoing monitoring, back-testing, outcomes analysis | Partly. Covers output quality, not what was done with the output | Monitoring report, performance history | | Is the model biased | Fairness and bias testing at validation and in monitoring | No. Tests the scoring, not the action taken on it | Validation report, model card section | | Where did the data come from | Input data documentation and lineage | Partly. Covers training and input data, not the records the agent touched at run time | Lineage record | | Which tools could this agent call at 03:14 | None | No | None | | Whose credentials did it act under, and with what scope | None | No | None. Identity logs show a service account, not an agent or a task | | What did it do, and can it be replayed | None | No | None |

The last three rows are the article. Closing them takes a different kind of control: an authorisation check at the point the agent acts, a credential scoped to the task rather than to the agent, and a per-action record capturing the decision and its side effects rather than the model's output. We have set out the specifics of that record in what an audit trail has to capture, and the work of getting a written policy to that level of enforcement in from policy to implementation.

One question to put to any vendor selling you agent oversight. For an action taken at a specific time, can you reconstruct the tools available, the authority used, the side effects landed, and a path to replay it? Four answers. If any is no, the gap is still open.

Where vendors are getting this wrong today

Three patterns are worth recognising, argued as patterns rather than aimed at any one company.

The inventory row. A programme adds "GPT-based collections assistant" to the model inventory with an owner and a risk tier, and reports agent governance as complete. The row is accurate. It describes a model. The agent's tool inventory, credential scope, and action history are not in it, and no review of that row surfaces them.

Model observability sold as agent oversight. Token counts, latency, and evaluation scores describe the model's behaviour precisely and say nothing about side effects. An agent that performed 400 unintended writes and one that performed none produce similar-looking telemetry.

Superseded citations. Pages published in 2026 still cite SR 11-7 as live guidance, and some link a URL that has been dead since April. For a reader whose job is regulatory accuracy, a stale citation says something about everything else on the page.

The same instinct shows up in multi-model deployments, where each new model widens the artifact inventory without touching the action surface. That surface is better handled by coordinating models across an app layer.

What we're doing about this at Major

Model governance answers a real question: is this model fit for use. It cannot answer what this thing did on Tuesday, with whose credentials, and who approved it, because the model is not where the action happened. The action happened between the model's output and the system it touched, and in most deployments that space holds a prompt, a tool definition, and an API key. None of the three is a governed object.

Our position is that the space is best closed by putting software in it. On Major, when an agent works out how to handle a repeatable part of a task, it builds an app for that part and then runs the app. The app holds scoped credentials rather than the agent holding a key, enforces role-based access on the records it touches, and writes its own audit log. What did it do, with what authority, and when becomes answerable by inspection, because the action happened inside an application rather than inside a model's reasoning. The model still reasons for the judgment calls, and does less of the rest each time. Reason once, run forever.

SR 26-2 corroborates this quietly. The revised guidance draws its scope line at whether statistical, economic, or financial theories underpin a process, and puts deterministic rule-based software on the far side of that line. Work that moves out of the model and into code moves into ordinary software controls, which enterprises have run for decades. That does not resolve anyone's regulatory position, and a supervisory examiner will have views. The structural point stands: deterministic software is governable by mechanisms that already exist.

Scope this honestly. Major is not a model risk platform and does not validate models. If you run credit, pricing, or capital models, you still need validation, drift monitoring, and an inventory, and this sits alongside that work. Nor does an app layer solve prompt injection or decide which actions a human should approve. It makes the actions inspectable, which is the precondition for every other control being enforceable.

If the gap in front of you is action attribution rather than model validation, the question is whether your agents act through software you can permission and read afterwards. You can see how Major's agents build apps that hold scoped credentials and write their own audit log.

Related articles

• AI Governance Framework: One You Implement, Not Attest To
• AI Agent Governance: Why Control Must Move to the Point of Action
• AI Agent Observability Isn’t a LangSmith Feature