What a model upgrade actually broke, and why it was not the prompts

Enterprise AI Management · November 2026 · PolyQ Team

What a model upgrade actually broke, and why it was not the prompts

We moved Mila onto a newer generation of models this month. The prompts survived. The plumbing around them — APIs, caching, length limits and one forgotten hard-coded model — did not. It is a useful case study in what model governance has to cover.

The usual worry about a model upgrade is that the prompts will behave differently. Ours mostly did not. What failed was everything that assumed the old model: the interface used to call it, the way responses were cached, and the length of the answers it was allowed to give.

  • A new generation can mean a new interface: Newer models had to go through a different API than the one Mila used. Routing by model generation is now explicit rather than something a model name happened to imply.
  • Caching rules changed underneath us: The property brief that every answer depends on needed explicit cache breakpoints to stay cached on the newer models, and our cache keys had quietly grown past the provider’s 64-character limit.
  • Silent truncation is the dangerous failure: Two background jobs — writing the property brief and extracting entities — were being cut off without any error. They now have room to finish, and an answer that is cut off is flagged instead of passed along as complete.
  • The forgotten hard-coded model: One feature, the portal’s own help assistant, still named an older model directly in code. It now uses the configured answer model like everything else.
  • The governance lesson: An audit trail for prompts is necessary and not sufficient. What a system was instructed to do depends on the prompt, the model, the interface and the limits together, and all four need to be recorded and reviewed as one change.

Models will keep being replaced on a schedule you do not control. The organisations that cope will be the ones that know every place a model is named, called and constrained — before the deprecation notice arrives.

If you are planning a model migration across a portfolio of AI features, we are happy to compare notes on what to inventory before you start.

More from Thinking