How should teams route many LLM workflows to the right models?

Define requirements for each workflow

“Right model” is not a single leaderboard position. It is a decision about a particular workload under constraints. A model that is excellent at long-form synthesis may be too expensive or slow for extracting two fields from a form. A low-latency model that handles routine classification may be unsuitable for a tool-using workflow that must reason over a changing codebase. A model that performs well on a public benchmark may be prohibited for data that has a residency, confidentiality, or contractual restriction.

Treat each workflow as a small product with its own contract:

• What job is the workflow meant to do?
• What inputs, data classes, and tools may it access?
• What output or action counts as success?
• What errors are unacceptable?
• What response time and cost are acceptable?
• What human review or escalation is required?

NIST’s framework recommends documenting an AI system’s knowledge limits, intended use, benefits and costs, test sets, metrics, and deployment-context performance. That is a sound basis for routing because it forces teams to compare candidate models against the actual job rather than an abstract score. NIST AI RMF core

A shared routing process can prevent dozens of workflows from independently choosing models, settings, fallbacks, and upgrade dates. Centralize the common decisions, but preserve task-level evidence.

Build a capability registry

A capability registry is an internal record of approved model profiles and the conditions under which they may be used. Do not make it a list of marketing descriptions or a static price sheet. It needs to support a real routing decision and show what changed.

Each profile should have a stable internal ID, such as “extract-fast-v3” or “code-review-reasoning-v2.” The profile points to an exact reviewed model or runtime configuration. Workflows reference the internal profile, not a vendor’s moving “latest” label.

Record the test results and conditions supporting each capability claim. “Supports tool calling” can mean only that a model returned a tool shape in a specific runtime. It does not prove that it chooses the right tool under ambiguous instructions or that it is safe to authorize high-impact actions. The registry should link to the relevant evaluation suites and tool-contract tests.

NIST’s Generative AI Profile notes that behavior can be difficult to attribute when a system combines third-party components and data sources. A registry should therefore also capture the surrounding runtime, retrieval configuration, and tool contracts when they materially affect results. NIST AI RMF Generative AI Profile

Define task-level evaluation sets before routing

A model profile has no useful “quality” score until it is tested on a task. Build one evaluation set per workflow or per tightly related task family. Reuse shared safety and formatting cases, but do not assume that an evaluation for customer-support drafting predicts performance on code modification or data extraction.

An evaluation set should include:

• representative successful cases drawn from approved, privacy-safe historical data or carefully designed fixtures
• difficult but legitimate boundary cases
• cases where the right outcome is to ask for clarification, abstain, or escalate
• regression cases from previous incidents
• tool or retrieval fixtures when the workflow depends on them
• a clear expected outcome, rubric, or invariant
• metadata for data class, language, input length, tenant configuration where relevant, and risk tier

Use a mix of deterministic and human checks. Deterministic checks work for JSON schema validity, citation presence, tool-argument constraints, refusal of an unauthorized action, and known exact data fields. Human review is needed for nuanced usefulness, factual accuracy in context, completeness, and whether an explanation is understandable. An LLM judge can help scale a specific rubric, but it is not ground truth and should be calibrated with human labels for high-impact cases.

Document the test set, scoring method, and measurement uncertainty. NIST explicitly calls for objective, repeatable test, evaluation, verification, and validation processes, with test sets, metrics, tools, and results documented. It also recommends measuring in conditions similar to deployment and monitoring components in production. NIST AI RMF core

Route by policy before optimizing for cost

Routing should be a series of policy decisions, not one opaque score. First eliminate profiles that cannot legally, technically, or safely handle the work. Then choose among the eligible profiles based on evidence and budget.

1. Classify the request. Identify the workflow, data class, required output format, tool requirement, locale, risk tier, and any user-selected service level.

2. Apply non-negotiable policy. Exclude profiles that do not satisfy data residency, confidentiality, access, model-approval, tool-use, or human-review requirements. Do not let an accuracy score override these boundaries.

3. Apply capability requirements. Exclude profiles that lack required structured output, tool calling, context capacity, approved language support, or deployment environment.

4. Apply quality gates. From the eligible profiles, keep only those that meet the workflow’s minimum evaluation score and critical-case pass requirements. A candidate that has an excellent average but fails an unacceptable safety or data-access test does not pass.

5. Choose within the service envelope. Prefer the lowest expected cost profile that meets the latency, quality, and capacity budgets. If several profiles pass, use a deterministic tie-break rule so behavior is explainable.

6. Use a declared fallback or escalation. Fallbacks must be compatible with the workflow contract. When no eligible profile exists, return a safe limited response, queue the work, or escalate to a human. Do not silently route restricted data to a less constrained provider.

This order keeps “cheapest sufficient” from becoming “cheapest regardless.” It also separates policy enforcement from quality selection, which makes later audit and incident review much clearer.

Define cost and latency budgets per workflow

Cost and latency are properties of the full workflow, not just the model request. Context construction, retrieval, retries, tool calls, output length, and fallback attempts can dominate both. Give each workflow a service envelope:

Use measured distributions rather than a single advertised latency or list price. Record input size, output size, retries, cache behavior, tool calls, and failed attempts. A model may look inexpensive in a short benchmark but become costly when an agent repeatedly calls tools or produces long explanations.

Set different objectives for interactive and offline work. A meeting-note draft can tolerate a queue and batch-oriented profile if it has a clear deadline. A customer-facing triage response may need a strict end-to-end latency limit, a short controlled answer, and a clear human escalation path. Do not reserve the strongest profile for every urgent request if a smaller eligible profile meets the tested service envelope.

Pin versions behind internal profiles

Provider aliases, default models, APIs, safety settings, tokenizers, and tool formats can change. A workflow that calls a moving “latest” identifier cannot reliably compare its behavior before and after a provider change. Pin the concrete reviewed version in the model profile, along with the relevant runtime settings and tool schema versions.

Version the routing policy too. A route decision should record the workflow ID, routing-policy version, selected profile, exact model version, prompt or instruction version, retrieval or tool version, experiment variant, and fallback outcome. This supports reproduction without retaining more user content than is necessary.

Treat a model change as a release. Run the affected workflow evaluations, compare the old and candidate profiles, review failures, and publish release notes. Separate a routine patch, such as a transport-client correction, from a material behavior or compatibility change, such as new tool-calling behavior, changed context handling, or a new model family.

Software-supply-chain provenance offers a useful analogy. SLSA describes recording resolved dependencies so a consumer can understand how an artifact was built. Agent routing does not need to implement SLSA, but a route record should similarly identify the resolved model, configuration, and material dependencies that shaped a result. SLSA provenance specification

Design fallbacks as compatible contracts

A fallback is not “try any model that is available.” It is a pretested alternate that meets the same safety and data boundary, or a deliberately reduced mode that tells the user what changed.

For each workflow, define:

• primary profile and an ordered, compatible fallback list
• conditions that trigger a fallback, such as timeout, capacity exhaustion, repeated invalid structured output, provider outage, or a policy-preserving quality failure
• conditions that prohibit an automatic fallback, such as a new data class, an action with side effects, a high-risk decision, or a missing human review
• user-visible degraded behavior, such as a shorter draft, a request to retry later, or escalation to a human queue
• a circuit-breaker and rollback owner for systemic failure

Always validate the fallback result against the original contract. If a profile returns malformed structured output or chooses an invalid tool argument, do not merely retry it with a stronger model and declare success. Record the failure, validate the final result, and investigate whether the workflow’s contract or tool schema needs repair.

Use controlled rollouts, not surprise migrations

After offline evaluations identify a candidate, release it in stages. The exact percentage depends on risk and traffic, but the sequence should be consistent:

1. Run a reproducible offline comparison on the task set, including prior regressions and critical safety cases.

2. Use a shadow or replay stage where the candidate processes representative requests without affecting users. Compare quality, cost, latency, tool behavior, and policy outcomes with the current profile.

3. Start a small canary cohort, assigned deterministically and tagged in telemetry. Do not put high-risk actions into a canary without the same human approval controls as the control route.

4. Promote only if predeclared success criteria hold. Pause or roll back on a critical failure, material quality regression, policy violation, budget breach, or unexplained shift in error behavior.

5. Keep the previous approved profile available for a defined rollback window, then document the decision and retire it according to the lifecycle policy.

Do not update every workflow together merely because a new model performs well on a general benchmark. Group rollout cohorts by task family and risk. A product team may choose a new extraction profile quickly, while a code agent with repository tools needs deeper evaluation and a longer observation window.

Monitor the route and the outcome

Monitor what the router chose, why it chose it, and whether that choice produced the desired user outcome. A model dashboard alone will not show a policy bug, a failing fallback, or a workflow that is no longer using current data.

For each request, log a privacy-conscious route record that includes the workflow, policy version, eligible-profile count, chosen profile and resolved model, fallback reason, request and response size bands, latency, cost estimate, tool outcomes, evaluation tags, and outcome status. Keep raw prompts and outputs separate and subject to the organization’s data policy.

OpenTelemetry semantic conventions exist to give telemetry a shared meaning across systems, and its guidance calls for recording operation target, input parameters, and result properties where appropriate. Use a consistent schema for route records so a model incident can be traced across application, retrieval, tool, and provider boundaries. OpenTelemetry semantic conventions

NIST recommends regular production monitoring, feedback processes for users to report problems, documented incident response and change management, and tracking risks over time. These are useful routing disciplines even when the workflow is low risk. NIST AI RMF core

Example routing decisions

The following examples are hypothetical. The profile names describe an internal registry, not a vendor recommendation.

The important difference is that each choice is supported by a workflow contract and evaluation evidence. The router does not infer that “code is hard” or “summaries are easy” from a generic request. It receives a workflow identity with already-declared constraints.

A practical first 30 days

Start with the highest-volume or highest-cost workflows, not all 54 at once.

• Inventory the workflows, current models, data classes, tools, owners, observed cost, latency, and known incidents.
• Define five to ten internal model profiles and retire direct use of moving aliases in production paths.
• Build a capability registry with links to provider or runtime documentation, known limitations, and the exact versions currently in use.
• Create small evaluation sets for the top few task families. Include golden, edge, abstain, and regression cases.
• Write a simple routing policy with policy exclusions first, then capability, quality, cost, latency, and fallback rules.
• Add route-decision telemetry and a small dashboard for quality, cost, latency, fallbacks, and errors.
• Move one low-risk workflow through shadow, canary, and rollback. Use what that teaches to improve the registry and test format.

This first system can be a version-controlled configuration file and a modest service or library. It does not need a machine-learning router. A rules-based policy is usually easier to audit, change, and explain until teams have trustworthy evidence that a more adaptive approach improves outcomes.

Common failure modes

• One benchmark chooses every model. A broad benchmark hides task-specific errors and data boundaries. Use task evaluations and critical-case gates.

• Routing only by price. A cheap profile can create more cost through retries, corrections, escalations, and missed outcomes. Measure completed-task cost and quality together.

• Silent fallback to an incompatible service. A timeout sends data or a tool-enabled task somewhere that violates policy. Make compatibility and escalation rules explicit.

• A provider alias changes in place. The workflow drifts without a release or comparison. Pin reviewed versions behind internal profiles.

• No route decision is logged. Teams cannot reproduce incidents or tell whether a performance change came from the model, prompt, retrieval, tool, or policy. Record privacy-safe resolved dependencies.

• One rollout affects every workflow. A global migration creates simultaneous failures with no clear attribution. Use task-family cohorts and rollback windows.