The key insight: when you swap the model behind an agent, you're not upgrading a text generator — you're replacing the planner. Same prompt, same tools, same task, but a completely different policy deciding how to get there. Turn count, tool-call order, even whether a checkpoint gets hit at all — none of that is guaranteed to survive a migration.
This is the trap: an agent that used to take eleven turns to close a support ticket now takes four. Looks like a win, right? Except it skipped a validation step the old model always ran. Or it takes forty turns and trips a guardrail that was calibrated around the old model's more deliberate pacing. The system prompt didn't change. The tools didn't change. The policy did.
<> Trajectory length was never a proxy for quality. It was a proxy for this particular model's quality, under this particular model's planning habits — and that correlation breaks the moment you swap the backend./>
Why this happens under the hood
Agent planning is emergent, not specified. Your system prompt describes constraints and available tools; it doesn't dictate the internal search process the model uses to satisfy them. Recent work on long-horizon agent planning (splitting agents into an explicit Planner and Executor, with forced dynamic replanning) exists precisely because a fixed initial plan tends to diverge from what the executor actually does step by step. Different models diverge differently.
There's also solid evidence that models vary a lot in plan compliance — how strictly they stick to a given plan versus adaptively overriding it based on perceived task difficulty. A newer, more capable model might decide a step is unnecessary and skip it. An older or more literal model might follow the plan rigidly even when it's suboptimal. Same plan, same tools, wildly different execution traces.
What this looks like in practice
Say you have an agent that resolves support tickets with this rough loop:
1def resolve_ticket(ticket, model, tools):
2 plan = model.plan(ticket, tools)
3 turns = 0
4 while not plan.is_complete() and turns < MAX_TURNS:
5 action = model.next_action(plan, history)
6 result = tools.execute(action)
7 history.append(result)
8 turns += 1
9 if guardrail.triggered(history):
10 escalate(ticket)
11 return
12 return plan.statusBefore migration: MAX_TURNS = 20, typical resolution takes 11 turns, guardrail almost never fires. After migration to a faster, more decisive model: typical resolution takes 4 turns. Great — except your regression suite only checks plan.status == "resolved", and the new model quietly skips the "confirm customer identity" step because it decided, on its own, that the ticket context made it redundant.
Nothing in your test suite catches this, because nothing in your test suite checks how the ticket got resolved — only that it did.
The fix: test trajectories, not just outcomes
Instrument the checkpoints that actually matter, in code, not as an implicit expectation baked into a system prompt:
1REQUIRED_CHECKPOINTS = {"verify_identity", "check_refund_eligibility"}
2
3def validate_trajectory(history):
4 hit = {step.name for step in history if step.name in REQUIRED_CHECKPOINTS}
5 missing = REQUIRED_CHECKPOINTS - hit
6 if missing:
7 raise TrajectoryViolation(f"Missing checkpoints: {missing}")This turns an implicit assumption ("the model will always verify identity because it always has") into an explicit, testable contract that survives a model swap even if the model's path to resolution changes completely.
Metrics that actually catch migration drift
Comparing anecdotes before/after a migration is how these regressions slip through — you eyeball a few transcripts, they look fine, you ship. Instead, run a representative batch of tasks through both models and compare distributions:
- median turns and variance (a shift in variance is often more telling than a shift in the median)
- guardrail-trigger rate
- checkpoint/tool-call coverage (did required tools get called, regardless of order?)
- escalation rate
- repair rate after a failed attempt
If median turns dropped from 11 to 4 but checkpoint coverage also dropped, that's not an efficiency win — it's a silent capability regression wearing a performance improvement's clothes.
Re-tune, don't assume portability
A prompt tuned for a verbose, cautious model often underconstrains a more decisive one — it doesn't need the scaffolding you built for the old model's hesitation, so it starts improvising past your intended guardrails. Conversely, a prompt with heavy step-by-step instructions can force a newer model into inefficient loops it wouldn't otherwise need. Plan reminders help some models stay compliant; they can actively hurt others by over-constraining an already-good policy.
<> Treat prompt calibration as coupled to the specific model version, the same way you'd treat a hyperparameter tuned for a specific dataset. It doesn't transfer automatically./>
Why this matters
If your team ships agents into production, model migrations need a different QA lens than "did it still pass the eval suite." You need trajectory-level regression tests, explicit checkpoint validation in code rather than in prompt text, and distributional comparisons of turn counts, guardrail triggers, and tool coverage — not single-run spot checks. The next time a model upgrade makes your agent "faster," ask what it stopped doing to get there before you celebrate the latency win.
