Evals stopped being a report and became a merge gate

Platform · October 2026 · PolyQ Team

Evals stopped being a report and became a merge gate

The meaningful change in 2026 is procedural rather than technical. Every pull request that touches a prompt, a model version or a retrieval configuration now triggers an evaluation run, and a regression past threshold does not merge.

For two years evals were something a team ran after the fact, wrote up, and circulated. The output was a document. What changed is that the output became a status check — and a failing status check stops a merge the same way a broken unit test does.

  • The gate is the whole point: An eval that produces a report invites a conversation about whether the regression matters. An eval wired into CI against a golden dataset simply blocks the merge. Only the second one changes behaviour.
  • The production-to-eval loop closed: Platforms now curate sampled production traces back into the evaluation dataset, so the cases that actually broke become the cases you test against. The dataset stops being a thing somebody wrote once.
  • Sampling catches what you did not change: Scoring 5–10% of live traffic with an automated evaluator is the only layer that catches drift — the changes that happen to you rather than the ones you make. A model provider’s silent update does not announce itself in your diff.
  • Version the evaluator alongside the prompt: Keeping prompts, evaluator configurations and test reports under the same history is what makes an audit survivable. Without it you can show what the system was told to do but not what you checked before letting it.
  • Where we sit: This is the discipline our Enterprise AI Management platform was built around. A change goes through review, lands as a recorded version, and is reversible — which is only useful if something was actually checked in between.

Software got reliable when tests started blocking merges rather than describing failures. AI systems are a few years behind on the same lesson, and 2026 is when the industry appears to have learned it.

If your prompts are reviewed by a person reading them carefully, that is a start. The next step is a dataset the change has to pass before anyone reads anything.

Sources

More from Thinking