Leading AI chatbots give wrong answers to financial questions more often than not, according to a new research report by UK fintech Saturn that tested 18 of the most popular AI models and found an average error rate of 57%, the Financial Times reported on Monday.

The report, titled "Artificial Authority," arrives as millions of consumers increasingly turn to chatbots for guidance on pensions, tax, debt and savings — questions where a single wrong answer can carry real financial consequences. As general-purpose assistants are pushed into regulated territory, studies like this one are becoming a recurring stress test of the industry's consumer promises, and we will keep tracking the latest AI developments as regulators respond.

How the Test Worked

Saturn's domain team put 18 of the most widely used AI models through 121 questions spanning pensions, tax, debt and savings — the core areas where retail consumers most often seek advice. The models tested included ChatGPT, Gemini, Claude and Microsoft's Copilot, according to the report's published summary.

On average, the models made mistakes 57% of the time, Saturn said. The company framed the findings as spanning "concerning mistakes to consumer protection failure," and used them to argue that the UK's advice gap — the gap between people who can access regulated financial advice and those who cannot — will not be closed by unregulated chatbots alone.

Even the Best Model Erred Frequently

The results were not uniform, but even the strongest performer failed often. According to the FT's report, the best-performing model was Anthropic's Claude Opus 5 running in reasoning mode — and it still made mistakes in 39% of answers.

That detail may prove to be the report's most consequential finding. Reasoning modes, which let models deliberate before answering, are the industry's leading answer to hallucination and error, and frontier models routinely post strong results on professional exams in finance and law. A 39% error rate on practical financial questions suggests that exam-style benchmarks and real-world consumer queries measure very different things.

The findings also drew pushback. Commenters on Hacker News questioned the methodology and model selection, noting that several of the report's example errors came from smaller free-tier models. But the 57% average and the 39% best-in-class figure stand as published claims from Saturn and the FT, and the report's central warning — that consumers should not treat chatbot output as regulated advice — was not disputed.

A Warning Regulators Have Heard Before

The report lands amid growing official unease about chatbots in money matters. UK consumers were already warned about AI chatbots giving inaccurate financial advice in a widely covered alert published by The Guardian in late 2025, and financial regulators across Europe and the United States have since flagged AI-generated advice as an emerging consumer-risk area.

Saturn's own position gives the report a particular edge: the company develops AI tools for financial advice, and it published the findings not as a rejection of AI in finance but as a case for getting the guardrails right. Its argument is that poorly governed chatbots are already filling the advice gap by default, and that the choice facing regulators is between supervised AI guidance and unsupervised AI guidance — not between AI and human advice.

Saturn's contribution to that debate is an argument about policy design: the company says a shift from what it calls excessive regulation toward putting practical guardrails on large language models would narrow the advice gap faster while protecting people from unregulated AI guidance. In other words, rather than keeping AI out of financial advice entirely, build supervised, constrained deployments that can safely serve customers whom human advisers are too expensive to reach.

That framing matters because the advice gap is the industry's strongest justification for being in this space at all. In the UK, most households fall outside the threshold at which regulated financial advice is considered economically viable, which is precisely the population now asking chatbots about pensions and debt.

What Consumers Should Take Away

Until models improve or regulators intervene, the practical guidance from the research is straightforward:

  • Treat chatbot answers on tax, pensions, debt and savings as a starting point for research, not as advice.
  • Verify any specific figure — thresholds, rates, allowances — against an official source before acting on it.
  • Be especially cautious with free-tier models, which may be older or smaller than the flagship versions a lab promotes.
  • For consequential decisions, a regulated human adviser remains the only channel with legal accountability attached.

The broader question the report raises is one the AI industry will face repeatedly in 2026: capability that impresses in demos and benchmarks can still fall short of the reliability bar that regulated domains demand. Financial advice is simply the first arena where that gap has been quantified this starkly.

Stay Ahead of AI

Tracking how AI is reshaping finance, research and regulation? Read more AI news on AI Buzz Wire — independent AI coverage, every day.

---

Stay Ahead of AI

Get the latest AI news, analysis, and breakthroughs — all in one place.

Read more AI news →