GENERATIVE AI
Cost-tiered Model Routing
A production routing pattern for sending easy requests to cheaper models and reserving frontier-model capacity for the cases that actually need it, without losing visibility into quality.
The problem
An LLM feature is built against the most capable model available, because during development that removes a variable.
It ships.
Usage grows, and the cost per request multiplied by volume becomes a line item somebody wants explained.
The reason this is worth addressing is that the price spread between frontier and small models is large — commonly an order of magnitude or more per token — while a substantial share of production traffic consists of requests that a much cheaper model handles identically.
Classification, extraction into a fixed schema, short summarisation and routine formatting are not tasks where frontier capability necessarily changes the answer.
Paying frontier prices for them is waste, and the waste scales linearly with success.
The corresponding trap is treating this as a switch to flip.
Routing is a quality risk introduced deliberately in exchange for cost.
A routing layer deployed without measurement moves spend down and quality down together, with no way to attribute the second effect.
The published cost reductions in the research literature are real and are conditional on a query distribution that skews toward easy cases and on a routing signal that actually discriminates.
When to use this
- There is meaningful volume — enough that inference cost is a real line item rather than a rounding error.
- The workload is mixed and request difficulty genuinely varies.
- An existing evaluation harness can measure quality per tier. This is a precondition, not a nice-to-have.
- Some paths are latency-sensitive and a smaller model is also materially faster.
- There is a requirement to reduce dependence on a single model provider.
When not to use this
There is no evaluation harness
Without one, routing is an untested change to system behaviour that cannot be validated or reverted on evidence.
Build Pattern 2 — LLM evaluation harness for production systems first.
This is the order.
Inverting it is how teams end up with a cheaper system that nobody trusts.
Volume is low
Below meaningful scale, the engineering cost, added failure modes and ongoing tuning can exceed the saving.
A single well-chosen model is often the right answer for a long time.
Every request is genuinely hard
Where the workload is uniformly complex, routing produces no meaningful saving and adds another component.
Measure the request distribution before assuming it is skewed.
Uniform behaviour is a hard requirement
In some regulated or contractual environments, identical inputs must follow a consistent execution path.
A router that sends similar requests to different models can break that guarantee in a way that is difficult to explain to an auditor.
Cost pressure is really a prompt problem
Before adding routing, inspect context size.
Sending a large static preamble on every request, or retrieving forty chunks where eight would do, is often a larger and much simpler saving.
Prompt caching may resolve the problem outright.
Architecture
Two designs dominate, and they are endpoints of a continuum rather than competing philosophies.
Classify-then-route
Classify-then-route decides before any generation.
A lightweight classifier or embedding-based similarity model predicts difficulty and sends the request directly to the appropriate tier.
It adds negligible latency and never pays twice, but it commits before seeing any output.
Cascade
A cascade sends the request to the cheap model first, scores the response, and escalates to a stronger model when the score falls below a threshold.
It captures more of the available saving when the cheap model is usually right.
The cost is that every escalation pays for two generations.
Which approach is better depends on whether a reliable post-generation confidence signal exists for the task.
Where it does, cascading can capture more of the gap.
Where it does not, pre-routing is safer because it never acts on a signal it cannot trust.
A badly calibrated scoring function turns a cascade into a system that escalates the wrong cases while paying twice.
Policy rules belong alongside the router
Policy rules frequently do most of the useful work.
Route by customer tier, request type, explicit SLA or a hard budget ceiling.
A deterministic rule that sends all bulk classification to the small tier and all interactive customer-facing generation to the large tier is crude, predictable and can capture a large share of the benefit with none of the calibration risk.
Components
| Component | Role | Technology options |
|---|---|---|
| Routing layer | Decide which model serves a request. | LiteLLM, model gateway, application code |
| Difficulty classifier | Predict request complexity before generation. | Small fine-tuned classifier, embedding similarity |
| Scoring function | Judge a cheap-model response and decide whether to escalate. | Model-based scoring, heuristics, self-verification |
| Model pool | Provide two or three capability and cost tiers across providers. | Hosted APIs plus optional self-hosted open-weight models |
| Prompt cache | Reuse repeated context across requests. | Provider-native caching |
| Cost and quality telemetry | Track spend, latency and evaluation score per tier. | Existing observability stack plus the evaluation harness |
| Policy rules | Apply deterministic routing overrides. | Configuration, not model logic |
Tradeoffs
| Choice | What you gain | What it costs |
|---|---|---|
| Classify then route | One generation per request and low latency. | Commits to a tier without seeing the output. |
| Cascade with escalation | Captures more saving when traffic is skewed toward easy cases. | Two generations on escalation and total dependence on scoring quality. |
| Policy rules only | Predictable, explainable and requires no calibration. | Leaves savings on the table within each request category. |
| Learned router | Adapts to the real request distribution. | Training data, maintenance and drift as the model pool changes. |
| Self-hosted small tier | Lowest marginal inference cost and data remains in your environment. | Infrastructure to operate and capacity to plan. |
| Aggressive thresholds | Larger cost reduction. | Quality regressions that only the evaluation harness may detect. |
Implementation notes
Establish the quality baseline first
Record quality on the current single-model configuration before changing anything.
Without a measured baseline, every subsequent quality discussion is unanchored.
The routing layer will otherwise become the suspect for every unrelated complaint.
Start with one deterministic rule
Begin with the most predictable request type, measure it for a fortnight, and expand from there.
A two-tier policy split on a single well-understood category is straightforward, reversible and often captures a large share of the available saving.
A learned router is a reasonable second step and a poor first one.
Instrument cost and escalation rate
Track cost per request per tier.
Where a cascade is used, track escalation rate explicitly.
A cascade that escalates most of its traffic can cost more than not routing at all.
That condition is invisible without the metric.
Set an alert on it.
Pin model versions
Pin the model version used by every tier and re-run evaluation whenever one changes.
A provider update can alter the quality of one tier without any code change on your side.
The resulting movement is difficult to attribute after the fact if version changes are not controlled.
Design provider failure behaviour explicitly
Define fallback behaviour as part of the architecture.
A multi-provider model pool is an availability benefit only if the router actually fails over.
Test that failover path.
In particular, decide what happens when the cheap tier is unavailable and every request begins routing to the expensive tier.
That is a cost incident rather than a conventional outage, but it still requires detection and response.
Treat published savings as upper bounds
Cost-reduction figures in the literature come from specific benchmark distributions and routing policies.
Treat them as evidence that the mechanism works, not as forecasts for your workload.
The magnitude of the saving is a property of your traffic and must be measured on it.
References
-
FrugalGPT: how to use large language models while
reducing cost and improving performance
— Chen, Zaharia & Zou, 2023. Covers prompt
adaptation, model approximation and LLM cascades, and is
the source of widely cited upper-bound cost-saving
figures.
arXiv -
RouteLLM: learning to route LLMs with preference data
— Ong et al., 2024. Routers trained on human preference
data rather than manually labelled difficulty, with
reported cost reductions while maintaining quality.
arXiv -
AutoMix: automatically mixing language models
— explores self-verification as the escalation signal in
a model cascade.
arXiv -
A unified approach to routing and cascading for LLMs
— ETH Zurich SRI Lab work showing routing and cascading
as special cases of a common policy.
GitHub -
LLM model routing: how to cut cost without losing quality
— a practitioner treatment of routing economics and the
limitations of the confidence signal.
Agentropic
EXPLORE PATTERNS
Spend frontier-model money only where frontier capability matters.
Explore practical data and AI architecture patterns built around measurable quality, cost control, operational constraints and production failure modes.