AI Agent Architecture: Components, Patterns, and the App Layer

Key takeaways

• Agent architecture has three layers: what reasons inside one agent, how agents coordinate, and what runs as plain code.
• Microsoft's five orchestration patterns describe coordination accurately. None of them describes work leaving the model permanently.
• Most systems should stay on the lowest rung that works. One agent with tools is the right default far more often than it is chosen.
• Determinism is a quantity that can grow over a system's life, not a property you assign a workload at design time.
• Code an agent wrote still needs versioning, tests, and an owner. The third layer is cheaper to run and not free to keep.

What agent architecture means

An architecture diagram that shows only what reasons at runtime is incomplete, because it cannot express how the system changes as it learns. That is the argument here, and it earns a hearing only after the agreed parts are described properly.

Agent architecture covers two things in the current literature. The first is what sits inside a single agent: the model that reasons, the memory that carries context, the tools that touch external systems, and the loop tying them together. The second is how several agents coordinate when one is not enough. IBM's explainer covers the first, Microsoft's orchestration guide covers the second in depth, and neither covers a third layer where work that has been figured out stops being reasoning and becomes software the agent calls.

This assumes you know what an AI agent actually is. That ground is well covered and re-covering it would waste your time.

Layer one: what is inside a single agent

The parts of an AI agent are the reasoning model that interprets a task and chooses what to do, memory that carries context within and across runs, tools that read from and write to external systems, a planning step that decomposes a goal into ordered actions, and a control loop that decides when to act again and when to stop. Remove any one and a specific failure appears.

That last point is the useful way to learn the components. A list of five nouns teaches nothing; what each part fails at when missing teaches the architecture.

| Component | What it does | What breaks without it | Where its state lives | | --- | --- | --- | --- | | Reasoning model | Interprets the task, selects the next action, and produces output | The system becomes a fixed script and cannot handle inputs it was not written for | Nowhere. Weights are static and the reasoning is discarded after each call | | Planning | Decomposes a goal into ordered subtasks before acting | The agent acts one step at a time and thrashes on tasks needing four or more dependent steps | A plan or task ledger, usually in the prompt or an orchestrator's memory | | Memory | Carries conversation, prior results, and learned facts across steps and sessions | Every run starts cold. The agent repeats work and contradicts its own earlier answers | Context window for short-term, a vector or relational store for long-term | | Tools | Read and write to APIs, databases, and file systems | The agent can describe an action but cannot take one | The external system. The agent holds only the call and its response | | Control loop | Decides whether to act again, escalate, or stop | Infinite tool-call loops, or premature stops that leave the task half done | Iteration counters and stop conditions in the runtime | | Reflection | Evaluates its own output against criteria and revises | Errors ship. Nothing catches a plausible wrong answer before it reaches a system of record | The critique itself, held in context and lost when the run ends |

Read the right-hand column. Four of the six rows describe state that lives in a context window and disappears when the run ends. Google Cloud's architecture guidance is direct about the consequence: "when you use in-memory storage, session state isn't persistent," and "if the application restarts, it loses all conversation history." Redis makes the same point from the data side, splitting agent memory into short-term conversational context and episodic memory that "captures specific events with full temporal and contextual information."

IBM's taxonomy sorts single agents into reactive, deliberative, and cognitive designs. Reactive architectures "map situations directly to actions." Deliberative ones reason and plan against an internal world model. Cognitive ones add learning and adaptation, often through the BDI model of beliefs, desires, and intentions. That lineage is real and worth knowing. Its limit is worth naming too. All three categories describe how an agent reasons at runtime. None describes what an agent leaves behind.

Layer two: how agents coordinate

Microsoft's Azure Architecture Center publishes the strongest primary reference on this layer, a 7,133-word guide to AI agent orchestration patterns. Use its five names, because shared vocabulary beats proprietary vocabulary.

| Pattern | How it coordinates | Best for | When to avoid | | --- | --- | --- | --- | | Sequential | Linear pipeline. Each agent processes the previous agent's output in a predefined order | Multistage work with clear dependencies and progressive refinement | Stages that could run in parallel, or workflows needing backtracking and dynamic routing | | Concurrent | Fan-out and fan-in. Agents work independently on the same input, results aggregated after | Independent perspectives on one problem, and latency-sensitive work | Agents needing cumulative context, or cases with no clear conflict-resolution strategy | | Group chat | Agents contribute to one accumulating thread, a chat manager sets turn order | Consensus-building, structured review, and maker-checker validation loops | Linear delegation that would suffice. Microsoft advises capping this at three agents | | Handoff | Dynamic delegation. One agent holds control and transfers it when it hits its limit | Cases where the right specialist only becomes clear during processing | Routing that is rule-based and predictable from the initial input | | Magentic | A manager agent builds and revises a task ledger, then delegates against it | Open-ended problems with no predetermined solution path | Time-sensitive work, or any solution path that should be approached deterministically |

The most valuable thing on that page is the complexity ladder that precedes the patterns. Microsoft describes three rungs and tells you to climb as little as possible. A direct model call, where "if prompt engineering can solve the problem, you don't need an agent." A single agent with tools, which the page calls "often the right default for enterprise use cases." Then multiagent orchestration, which "adds coordination overhead, latency, and failure modes."

That advice is correct and widely ignored. Multi-agent orchestration is the most over-adopted pattern in this field. Teams reach for five specialised agents on a problem one agent with six tools would handle, then spend a quarter debugging handoff loops instead of the task. Microsoft deserves credit for leading with the warning rather than burying it. We mapped the frameworks that implement these patterns separately, alongside coordinating models, agents, and the app layer.

Layer three: the app layer, where work stops being reasoning

Look again at the bottom rung of that ladder. The cheapest option Microsoft offers is a direct model call. Plain deterministic software, a function that does the thing without asking a model, is not on the spectrum at all. The ladder starts one rung too high.

In a page about orchestration that omission is fair. It becomes a real disagreement one section later, in the sentence relating sequential orchestration to prior art:

"This pattern resembles the Pipes and Filters cloud design pattern, but it uses AI agents instead of custom-coded processing components."

Read the direction of travel. Custom-coded components are the old thing and agents replace them. That inverts what we think good agent architecture does. A pipeline stage that reliably reformats a payment file, checks it against contract terms, and writes the result to a ledger should stop being an agent and start being code. Swapping code for an agent buys nondeterminism, token cost, and an audit trail you reconstruct from a prompt.

Microsoft's antipattern list gets close and then closes the door. It warns against "using nondeterministic patterns for workflows that are inherently deterministic." The word doing the damage is inherently. It treats determinism as an intrinsic property of a workload, fixed before the system runs. Most real workflows are not born deterministic. They start as judgment, and the tenth time you handle one you know exactly what the steps are.

The closest anyone comes to the idea is a practitioner rule from Amar Gupta's widely cited guide: "If you can draw the decision tree on a whiteboard, don't use an agent." Good advice, and a build-time fork. A human decides once, agent or code, before anything runs. What it lacks is a lifecycle in which work begins as reasoning, gets figured out, and then becomes software. The whiteboard test is right. It should run continuously rather than once.

So the third layer is the app layer: durable software that handles the parts of a task the system has already worked out, holding its own data and its own logs, called by the agent rather than reasoned through by it. The workflow-level view of the same idea is covered in patterns, steps, and limits.

Two honest limits. Not every step can move down: genuinely novel judgment stays in the model permanently, and a system that tries to compile away judgment produces confident wrong answers on exactly the cases that matter. The app layer also is not free. Code an agent wrote still needs versioning, tests, an owner, and a deprecation path when the process changes. You are trading token cost and variance for maintenance surface. For work that runs a thousand times that trade is obviously right, and for work that runs twice it is obviously wrong.

How the architecture changes as the system runs

Microsoft's testing guidance contains the assumption this whole piece is arguing with: "Agent outputs are nondeterministic, so use scoring rubrics or language-model-as-judge evaluations rather than exact-match assertions."

For the reasoning layer that is simply true. You cannot assert exact equality on a judgment call. As a settlement for the whole system it gives up too early, accepting that everything an agent does must be graded rather than tested, forever, in proportions fixed on day one.

A system running for six months should not have the same architecture it had on day one. The steps it has figured out should have moved down, so the share of each run graded by rubric shrinks and the share covered by exact-match assertions grows. That migration is the most interesting property of a running agent system, and none of the reference architectures here can draw it. They describe a standing configuration: components that reason on every invocation, in a topology a human fixed at design time.

Determinism, on this view, is a quantity that grows rather than a label you attach to a workload before you understand it.

Choosing an architecture

Start with the lowest rung that works. Then sort each step of the workflow by how often it needs judgment.

1. Needs judgment every time. Keep it in the model. Ambiguous customer intent, novel exception handling, anything where the right answer depends on context you cannot enumerate. Grade this with rubrics and accept the variance.
2. Needs judgment sometimes. Keep it in the model with a deterministic fast path. The known cases run as code, the rest escalate to reasoning. Most classification and routing lives here.
3. Never needs judgment, and you know the steps. Move it to the app layer. It runs as code, holds its own state, and becomes testable with exact assertions.

Then revisit the sort quarterly, because items move from the first bucket to the third as the system learns the work. Most teams skip that revisit, and it is the difference between an agent system that gets cheaper as it runs and one that gets more expensive. The operational requirements are covered in what enterprise-grade actually requires.

What we're building at Major in response

Every architecture reference on this topic diagrams a system in a fixed state. An architecture that cannot express how a system's steps migrate over its life is describing a snapshot.

So on Major, when an agent works out how to handle a repeatable part of a task, it builds an app for that part and runs the app instead of reasoning through the step again. Three architectural consequences follow, and they are why we think the app layer belongs on the diagram.

Execution. The step runs as deterministic code, so it behaves the same on run 500 as on run 1, and it becomes testable with exact-match assertions rather than scoring rubrics.

State. The app carries its own managed database, storage, and logs, so the state that used to live in a context window persists and the work resumes after an interruption.

Governance. Because the work is code acting through scoped credentials, an action can be inspected, attributed, and controlled rather than reconstructed from a prompt afterwards. That is a different exercise from making agent behaviour inspectable at the reasoning layer, and both are needed.

The model still does real work here. It reasons about the judgment calls, and it writes the apps in the first place. What changes is the proportion, and the direction it moves. Reason once, run forever, and the architecture diagram gets simpler each time it happens.

If you are drawing this diagram for a system you will have to operate, the third layer is the part worth prototyping first: see how Major turns a figured-out workflow into a deterministic app the agent calls.

Related articles

• Model Routing: How to Send Each Request to the Right Model
• Multi-Agent AI: What It Costs and When It's Worth It
• LLM Gateway: What It Governs, and What It Cannot