Major

AI Model Selection: Choose Per Task, Not by Hype

Choosing an AI model is a workload decision, not a leaderboard decision. Use a scorecard for quality, latency, cost, reliability, and governance, then keep the architecture reversible.

Jason BaoUpdated
Team gathered around a whiteboard of sticky notes scoring AI model options against requirements

How to do AI model selection

AI model selection means testing candidate models on your own representative tasks and scoring them against requirements you set in advance: output quality, latency, context, tool-call reliability, data residency, and cost per completed task. Leaderboard rank only tells you which four models are worth testing. Your frozen evaluation set makes the decision.

The second half of the answer is architecture. Models turn over every few months, so the repeatable parts of a workflow should not live inside prompts written against one provider. On Major, agents build those parts into governed apps, so a model swap changes who answers a judgment call and nothing else.

Mozilla's guide to choosing ML models shows why leaderboards mislead. Pegasus and BART both published ROUGE-1 summarization scores above 40. Run on Mozilla's own text, the same models scored roughly 27 to 31. Microsoft's model selection guide goes further and names two factors to exclude on purpose: cultural popularity and who published the model.

Separate model choice, routing, and architecture

Three decisions get collapsed into one, and they have very different lifespans. Model choice is which weights answer a request. It changes on the provider's schedule. Routing is the policy that maps a request to a model, and it changes when your traffic mix or cost target moves. Application architecture is where logic, state, permissions, and audit trail live. It should change only when the business process does.

The failure mode is letting the fastest-moving decision drive the slowest. When prompt text, retries, output schemas, and business rules are scattered through code written against one SDK, swapping a model becomes a migration. Microsoft's guidance points the same way: put an abstraction layer between the app and the model, test candidates in parallel, and avoid opaque routing unless you get observability with it.

Criteria that matter

Quality on your tasks. IBM defines model selection as picking the candidate that generalizes best to unseen data. You rarely retrain a foundation model, so the equivalent is a frozen evaluation set of 30 to 100 graded examples per task type, built once and reused on every candidate. Write acceptance criteria before you read any output. Grading afterward is how a team talks itself into the model it already wanted.

Latency and throughput. Measure p95. An agent that answers in 900 milliseconds most of the time and 14 seconds on the tail feels broken, and the average hides that. Separate time to first token from total completion time, and count rate limits here too. A model that clears your quality bar and then throttles at 40 requests per minute has failed a throughput requirement.

Context and tool use. An advertised context window is capacity, not usable capacity. Test retrieval at 60 to 80 percent of the stated limit with the relevant fact in an awkward spot. For tools, track malformed call rate, wrong-tool rate, and recovery when a tool returns an error. That last one separates models more sharply than any public leaderboard and decides whether an agent can run unattended.

Cost per completed task. A rate card is cost per million tokens. Your invoice is cost per completed task, including retries, rejected outputs, and re-prompts after schema validation fails. A cheaper model that needs two attempts is not cheaper. Microsoft notes that larger context, multimodal input, and reasoning each add cost independent of volume.

Privacy and residency. Region availability, retention terms, and certifications decide eligibility before any quality score matters. Treat them as gates. A model that fails a residency requirement does not get a score. It leaves the pool.

A weighted scorecard

These weights are editorial defaults for a tool-using enterprise agent. Retune them for your workload. A support surface pushes latency up and context down. A contract pipeline does the reverse.

  • Task quality on your eval set (25%): graded pass rate across five task types; below 85% disqualifies.
  • Instruction and schema adherence (15%): valid structured output on the first attempt; below 95% disqualifies.
  • Tool-call reliability (12%): malformed and wrong-tool call rate; above 3% disqualifies.
  • Cost per completed task (10%): total tokens including retries, at your real mix; budget ceiling.
  • Latency, p95 (10%): time to first token and total completion; surface-specific ceiling.
  • Effective context handling (8%): retrieval plus synthesis at 70% of the window.
  • Throughput and rate limits (8%): sustained requests per minute at peak load.
  • Data residency and retention (7%): hard pass or fail gate.
  • Deployment options (3%): hard gate if on-prem is required.
  • Deprecation policy (2%): notice period and version pinning.

Run every candidate through the same five tasks with identical prompts, data, and tool definitions in the same measurement window: structured extraction from a messy real document, code changes to a real file in your repository, support triage against a written policy including cases it does not cover, long-context synthesis across far-apart passages, and multimodal input such as a scanned PDF. Changing any variable between candidates invalidates the comparison.

To apply it, drop every model that fails a hard gate, normalize each criterion to a 0-to-1 range across the survivors, apply the weights, and pick the highest score within your p95 latency and per-task cost ceilings.

This scorecard measures the model. It says nothing about your system. A model that passes at 94 percent inside a system with no retry logic and no durable state will not deliver 94 percent to a user.

One model or routing?

Default to one model. A single default is easier to forecast, debug, and govern. Routing earns its complexity when request difficulty varies widely and cheap requests dominate by volume. Two costs come with it: the router's effective context window is bounded by the smallest window in its pool, and runtime routing makes cost forecasting harder.

Configure fallback and escalation separately. Fallback handles provider failure, so a timeout or 503 sends the identical request to a secondary model. Escalation handles difficulty, so a low-confidence result or a refund above a stated amount goes to a stronger model or a named human queue. Write escalation as explicit conditions in your application and log each one against the record it touched. A model deciding it feels unsure leaves no audit trail. A threshold rule does.

Build a model-agnostic workflow on Major

Take invoice reconciliation as the example. Without an app layer, one agent holds the whole process in context and re-reasons it on every run, so a new model re-opens quality and cost for everything. On Major the split looks like this:

  1. The agent works out the repeatable steps once, such as field extraction rules, matching logic, and approval thresholds, and builds them as an app.
  2. The app holds state in its managed database: invoices, match results, exceptions, and the history of each decision.
  3. Connectors such as NetSuite for the ledger, Gmail for vendor email, and Slack for approvals sit behind scoped credentials, so each action is permissioned and attributed.
  4. A workflow trigger runs the app on a schedule or on an inbound event, and the model is called only for judgment: an ambiguous match, a contract clause, a drafted vendor reply.
  5. Escalation rules live in the app as readable thresholds, with every escalation written to the audit log.
  6. RBAC controls who can edit thresholds, view records, or approve exceptions.

Now your scorecard covers only the judgment calls, a much smaller surface. Swapping the judgment model changes which weights answer a question. It does not change how the invoice reconciles, where the state lives, or who can see it. That is reason once, run forever applied to model churn, and it is also why token cost is front-loaded and then flat rather than climbing with every run.

This does not remove the need to choose well. If your workload is a single stateless call with no repeatable structure, the scorecard and an abstraction layer will serve you. The case for the app layer grows with how repetitive, stateful, and governed the work is. Our guides to LLM routing and LLM cost management cover the mechanics in depth.

Start building on Major

Pick your judgment model with the scorecard above, then put the repeatable half of the workflow in a governed app with its own database and audit log, so the next model release is a config change.

Related articles

Frequently asked questions

How do I choose an AI model?
Start with the workload, not a leaderboard. Write down your requirements for quality, latency, context, tool use, cost, and data residency, then build a frozen evaluation set of 30 to 100 graded examples per task type. Run every candidate on identical prompts, data, and tool definitions, score against weights you set in advance, and treat residency and deployment limits as pass-fail gates.
What factors matter most in AI model selection?
Quality on your own tasks carries the most weight, followed by instruction and schema adherence, tool-call reliability, cost per completed task, and p95 latency. Effective context handling and sustained throughput come next. Data residency, retention terms, and deployment options work as gates rather than scores, since a model that fails a compliance requirement is ineligible regardless of how well it performs.
Should an enterprise use one AI model or several?
Default to one. A single model is easier to forecast, debug, and govern. Routing earns its complexity when request difficulty varies widely and cheap requests dominate by volume, sending simple work to a smaller model and hard or high-risk work to a stronger one. Two costs come with it: a router's effective context window is bounded by the smallest window in its pool, and runtime routing complicates cost forecasting.
How do I compare AI models fairly?
Hold everything except the model constant. Use identical prompts, identical input data, identical tool definitions, and the same measurement window for every candidate. Write acceptance criteria before you look at any output, grade with the same rubric, and record retries and failures rather than only successful runs. Published benchmark scores rarely transfer to your data, so treat them as a screening filter for which models to test.