Explainer

Model routing: using cheaper models where they work

Learn how model routing sends each request to the cheapest model that clears your quality bar, and how to test, escalate, and monitor it.

By the Cloudverg engineering team · · 5 min read

What it is

Model routing is the layer between your application and the models it calls. Instead of sending every request to one large model, a router looks at the request, picks a model, sends the prompt, and returns the answer. The aim is simple: use a small model for work it handles well and save the large model for hard cases. A password reset question can go to GPT-4o mini. A dense policy comparison can go to GPT-4o or Claude Sonnet.

Routing can happen per request, per step in a workflow, or per field in a structured output. In a claims intake flow, a small model can extract dates and amounts. A larger model can handle the ambiguous narrative and the final review. The router is part of the product behavior, not a hidden infrastructure detail. It changes latency, failure modes, and the prompts you need to maintain.

A router is not a model. It is a policy. That policy can be a few if statements, a classifier, a learned ranker, or a mix. The policy should name each model it can call, pin a version, and define the fallback. Without that discipline, routing becomes a random switch that is hard to debug.

How it works

Most routers start with a cheap signal. Read the task type from your app: support FAQ, invoice extraction, code review, or contract summary. Add metadata such as input length, language, attachment count, customer tier, and risk flag. Then apply rules. A short FAQ with a known intent goes to the small model. A request with a legal flag or a large attachment goes to the large model. Rules are easy to audit and cheap to run.

The next step is validation and escalation. Ask the small model for a structured answer and a confidence field, or check the answer against a schema, a regex, or a unit test. If validation fails, if confidence is low, or if the user asks for a better answer, send the same prompt to the larger model. Cap the number of escalations per request. A router that can loop between models is a budget leak with good intentions.

Log every route decision. Store the model name and version, the prompt version, the route reason, token counts, latency, validation result, and any user feedback. That log is your evidence. It shows which rules fire, which small model calls fail, and where the large model still earns its place. Replay the log against a golden test set before you change a rule. A model update from a provider can move behavior, so pin versions and rerun the tests.

Why it matters for a business

AI features often start with one large model for every call because it is the fastest way to ship. That works for a pilot and gets expensive in production. Routing lets you match spend to task value. The bulk of requests in many products are simple classification, extraction, or short answers. A small model can handle those at lower token cost and lower latency. The large model stays for the work that needs it.

Routing also improves reliability. If one model is rate limited or down, the router can fail over to another model that meets the same data rules. It can keep sensitive prompts on a model in an approved region or on a self-hosted open-weight model, while general prompts go to a hosted model. That is a policy decision, and the router is where you enforce it.

Cost is only part of the business case. Faster answers for simple requests make the product feel better. Clear escalation paths give support and engineering a shared language for failures. And a model registry with named versions makes audits and incident reviews possible. You can answer who called what, with which prompt, and why.

Common mistakes

The first mistake is routing on price alone. A small model that fails and then escalates can cost more than a large model called once. It also adds latency and a second chance for a bad answer. Measure the full path, including retries and human review.

The second mistake is no test set. Teams pick a small model after a few spot checks and then learn about edge cases from customers. Build a set of real prompts with expected outputs, including short prompts that are hard and long prompts that are easy. Test by task, not as one blended score.

The third mistake is a router that is too clever too early. A learned router with many features is hard to explain and harder to debug. Start with two model tiers and a handful of rules. Add features only when the logs show a clear gap. Also watch version drift: a model name without a version is not a stable dependency.

Another common mistake is ignoring the prompt. A small model often needs a shorter, more direct prompt than a large model. If you reuse the large model prompt, you may blame the model for a prompt problem.

How to get started

Pick one task with high volume and low risk. Support FAQ answers and invoice field extraction are good first candidates. Write down the quality bar: what must be exact, what can be paraphrased, and what must escalate. Then collect a test set from real traffic or from realistic samples. Keep it small enough to review by hand and broad enough to include the ugly cases.

Run the test set on your current large model to get a baseline. Run it on a small model such as Claude Haiku or GPT-4o mini. Compare pass rate by task, latency, token use, and the kinds of failures. Add a simple router: small model first, schema check, then escalate on failure. Put a hard cap on escalations and log the reason for each route.

Ship to a small slice of traffic. Watch the escalation rate, the validation failure reasons, and the user feedback. Remove rules that do not earn their keep. Keep the large model path available as a fallback and test it during incidents.

You can run this as an internal spike, hire a contractor, or use a platform team. Cloudverg offers an AI readiness assessment for $5,500/month, a two-week fixed-scope engagement that ranks use cases and builds a token and run-cost model. It is one option among others. The important part is to start with a measured task, not a company-wide router.

Frequently asked questions

What is model routing in AI?

Model routing is a policy that picks which model handles each request. It can send simple work to a small model and escalate hard or risky work to a larger one. The router also handles fallback, logging, and validation.

Does model routing always lower costs?

No. It lowers token spend only when the small model succeeds on most requests and escalation is rare. A bad router adds retries, latency, and review work. Measure the full path before you claim a saving.

How do I test if a cheaper model is good enough?

Build a test set from real prompts for one task. Score exact fields, allowed paraphrases, and required escalation. Run the small model, then compare failures with the large model. Keep the small model only if its failures are acceptable or easy to catch.

Monthly plan

Want help with this? AI readiness assessment

$5,500/month

2 weeks · Starts within 10 business days

Two weeks to rank your AI and agent ideas by value and risk, check your data, and plan a first project with a cost model.

Renews monthly. Cancel any time.
What is included

Related guides

More ai and agents guides   AI and machine learning

Smallest way to start

Get a senior architect on your problem this week

Ninety minutes a month on the cloud, AI or security question that is blocking you, a pre-read of your material and a written summary within two business days.

$450/month

Renews monthly. Cancel any time.