LLMs
Model routing is one of the few cost levers in an agent stack that works without touching your prompts, your tools, or your model quality targets.
The reason devs find it handy is that API pricing is spread widely.
Anthropic currently charges $5 per million input tokens for Opus 5 and $1 per million for Haiku 4.5.
And for the kind of agent workloads we run today, it’s an important thing to consider.
A single coding task can push 400K to 2M cumulative input tokens through the API, because every turn re-sends the full conversation so far.
Most of those turns are not hard. For instance, deciding to run the test suite, reading a file, or checking whether a command succeeded do not necessarily need frontier reasoning.
Technically, a router can sit in front of the model call, score the incoming prompt for difficulty, and then send the easy ones to a cheaper model.
On a workload where 70% of turns are trivial, this can provide huge cost savings:
But the problem is that the cost reduction only holds for single, independent prompts.
It does not work when you are inside a long session. Let’s understand more.
How prompt caching works
When you send a prompt to an LLM, the model has to process every token of that prompt before it can generate anything.
This is the prefill phase, and the intermediate results it produces (the key and value vectors for each token) form the KV cache.
Providers let you keep that cache alive between requests with prompt caching.
When your next request starts with the exact same tokens as the last one, the provider reuses the stored state instead of recomputing it, and bills those tokens at 10% of the base input rate.
The only problem is that the cache belongs to one model. Opus 5 and Haiku 4.5 have different weights and separate cache stores. A prefix that is warm on one of them does not directly transfer onto the other.
So a router that switches models mid-session is discarding the warm prefix and forcing a full cold prefill of the entire conversation on the new model.
This means you don’t pay the cheap model’s rate on the new instruction but rather the whole transcript.
An example
Say an agent has had 14 turns with the user and it’s carrying 60,000 tokens of history, including the system prompt, the tool schemas, a few files it has read, and the back and forth so far.
The new instruction (about 200 tokens) tells it to run a test, which is easy.
Option A, stay on Opus 5:
The 60,000 tokens of history are already warm, so they bill at the cache rate of $0.50 per million instead of $5.00. The 200 new tokens bill at the full rate.
- 60,000 cached tokens at $0.50/M comes to $0.030
- 200 new tokens at $5.00/M comes to $0.001
- Total input cost for the turn: $0.031
Option B, route to Haiku 4.5:
Haiku has never seen this conversation, so all 60,200 tokens are new to it and bill at its full input rate.
- 60,200 tokens at $1.00/M comes to $0.060
The model with 5x cheaper tokens produced a bill roughly 2x larger.
The router did its job, the cheap model also handled the prompt fine, and the whole turn still cost double.
The general rule
Call the history already in the conversation H, and the new tokens in this turn N. Call the strong model’s input price Ps and the cheap model’s Pc.
- Staying on the strong model costs
0.1 × Ps × H for the warm history plus Ps × N for the new tokens.
- Switching costs
Pc × (H + N), because nothing is warm. Setting those equal gives the condition where switching is cheaper:
- The numerator on the right denotes savings per new token by using the cheap model.
- The denominator denotes the loss per history token by re-prefilling it from cold rather than reading it warm.
Together, they tell you how many tokens of history you can afford to re-prefill for each new token you save on.
For Opus 5 and Haiku 4.5, that works out to be 8.
That means switching pays only while the history is under 8x the size of the new input. With a 200-token instruction, the conversation has to be shorter than 1,600 tokens.
An agent session with a system prompt and tool schemas is past that even before the first user turn.
Where routing is still important
In the example, switching cost an extra $0.029, but it saves $20 per million on output, so the cheap model needs to produce more than about 1,450 output tokens before the turn breaks even. That splits agent traffic along a clean line:
- Tool calls, short instructions, and control-flow turns emit a few hundred tokens and lose money on a switch.
- Turns that write a full file, draft a long document, or produce a large diff can benefit from a switch.
Short sessions follow a different rule.
With little accumulated history to re-prefill, the penalty nearly vanishes, and per-prompt routing actually works.
These are also the workloads routers were first measured on, and long agent sessions just took over their conclusion without understanding the conditions.
Where to place the switch
The cheapest moment to switch models is right after a compaction or a context reset. The prefix has already changed at that point, so the cache is invalid whichever model you use next, and the switch is free.
That suggests a different decision procedure.
Rather than scoring each prompt for difficulty and switching whenever one looks easy, find the points where the session’s cache already breaks and make every model change land on one of them.
A router scores the next prompt on its own, while the bill is computed over a prefix that grows for the entire session, and the savings disappear into the gap between those two units of measurement.
Further reading:
👉 Over to you: has your router ever made a session more expensive than the model it was routing away from?
