AI Model Testing: Three New Claude Models In Three Weeks, And Why UK Firms Should Test Before They Upgrade

AI model testing means running a fixed set of your own cases through a new model version and comparing the results with the version you use today, before you switch. It matters more than usual this autumn. Anthropic released three new Claude models between 22 September and 7 October 2026, and OpenAI released new GPT-6 models on 22 and 29 September. Each upgrade can improve results, and each can quietly change them.

What changed in the latest AI models?

Anthropic's release notes list Claude Opus 5.5 on 22 September 2026, Sonnet 5.5 on 28 September and Haiku 5.5 on 7 October. They also list breaking changes. On Opus 5.5 and Sonnet 5.5, asking the model to always use a particular tool now returns an error, and Opus 5.5 rejects an older way of switching on extended thinking. Software written for earlier models can stop working on the day of an upgrade.

Errors are the easy case, because they are visible. The harder case is a model that still runs but answers differently: a different cause for a reconciliation break, or a figure filled in where the data was missing.

Why does AI model testing matter for financial firms?

Because the work is checked by people who trust it to be consistent. On 30 September 2026 the Governor of the Bank of England, Andrew Bailey, wrote that “a sensible starting point is rigorous model testing, conducted before and after deployment.” A wrong cause on a break report wastes a morning. An invented figure on a client document is worse. Testing is how you find those before your team does.

What does a simple AI model test look like?

• **Collect real cases.** 20 to 50 items the agent will see, with the answer a person gave. Include awkward ones where data is missing and the right answer is to say so.
• **Run both models.** Same cases, same instructions, current model and new model.
• **Score two things.** Is the answer right, and did the model invent anything that was not in the input?
• **Block on safety.** If the new model invents a figure on any case where the old one did not, it does not go live.
• **Sign off by name.** A named person approves the switch and the results are kept.

How often should firms test AI models?

Every time something changes: the model version, the instructions, the tools the agent can use, or the software library that calls the model. Add every real mistake to the test set the week it happens. For technical teams, our playbook on LLM evaluation as a release gate builds this test in code.

Where should a firm start?

Pick the agent whose mistakes would cost most, write down 20 cases and run them before the next upgrade. Our AI Automation London team builds this check into every agent, including the daily mismatch report. Our article on the Bank of England's AI warning explains why supervisors care, and the 14-day Proof Run shows the approach on your own data.