How Accurate Is AI at Analyzing Data? 2026 Benchmarks

Two years ago the honest answer was “badly, on anything real.” The benchmark scores have since climbed enormously. The reasons to distrust them have climbed just as fast.

When someone asks whether AI can analyse their data, they usually want one number. There isn't one. What the model is asked to do — write a SQL query, write Python, chain eight steps together across four files — changes the answer by a factor of five. And the published figures for all of it rest on benchmarks that a 2026 audit found to be wrong more often than they are right. This is what the evidence actually says, what it cannot tell you, and what reduces the error rate regardless.

The two jobs, and why they score differently

“AI analysing data” almost always means one of two things under the hood, and they fail differently.

Text-to-SQL. The model reads your schema and writes a query. The database does the arithmetic. If the query is right, the number is right — there is no arithmetic left for the model to get wrong. The failure mode is a query that runs cleanly and answers a slightly different question than the one you asked.

Text-to-Python. The model writes code — usually pandas — that loads files, joins them, filters and computes. This handles things SQL cannot, but it adds every failure mode of writing a program, and the model is now responsible for the arithmetic as well as the logic.

Benchmarks exist for both. They disagree with each other, and increasingly with themselves.

Text-to-SQL: the cliff, and the climb back up

Spider 1.0 was the standard academic benchmark for years: small, clean schemas. Models passed 90% on it and vendors quoted the number. Spider 2.0 was then built to look like an actual company — 632 problems drawn from real enterprise systems on BigQuery and Snowflake, databases running past 1,000 columns, answers that often need several queries exceeding 100 lines.

When it launched, execution accuracy fell off a cliff, to roughly 21%. That collapse is the statistic that has circulated for the last two years, and it is the one in most articles written about this — including, until today, one of ours.

It is now out of date. The current Spider 2.0 leaderboard looks like this:

Three things are worth pulling out of that. First, the top scores are not models — they are elaborate purpose-built agent systems, and none of them is something you can buy and point at your warehouse. Second, the splits diverge sharply: the same class of system that scores 96% on the Snowflake split scores 65% on the dbt split, which asks it to write pipeline code rather than a single query. Third, and most important, every one of those numbers is measured against a ground truth that may not be correct.

Text-to-Python: where the wheels come off

DABstep, built by Adyen's data team, is the most useful benchmark I have found here, because it is drawn from a real financial analytics platform rather than an academic set. It has 450+ tasks split into 72 easy and 378 hard, and answers are checked automatically against a known value. Hard does not mean exotic — it means the answer takes several steps, reading documentation to find out what a field means, and combining sources.

The baseline results are the clearest illustration in this entire article of what is going on.

Source: DABstep baseline evaluation, Adyen / Hugging Face. Every model was run through the same wrapper with Python execution, so the comparison is like-for-like across rows.

The finding that matters: the model is not the lever

Read the easy column top to bottom. It runs from 66% to 81%, and it barely tracks model tier at all — GPT-4o-mini scores 69.4% on easy tasks, level with o1 at 69.4% and ahead of Gemini 2.5 Pro at 66.7%. Fifteen of the sixteen models land in the same band.

Now read the hard column. Every model in the table is under 15%. The best of them manages 14.55%. The gap between the best frontier reasoning model and a cheap small one is about eleven points on a scale where both have already failed.

The pattern repeats outside DABstep. A 2026 run of current frontier models against BIRD's Mini-Dev set produced Gemini 3 Flash at 68.2%, Claude Opus 4.5 at 64.9% and GPT-5.2 at 58.9% under strict scoring — a fast, cheap model beating two flagships. That same experiment added column comments to help the models along and gained 1.1 points.

The benchmarks are less reliable than the leaderboards imply

In January 2026 a team at the University of Illinois published an audit of the two benchmarks everyone quotes. Human SQL experts, assisted by a purpose-built review agent, went through the annotations — the question, the “correct” SQL, and the domain knowledge attached to each example — and checked whether they were right.

More than half of BIRD Mini-Dev and nearly two thirds of Spider 2.0-Snow contained annotation errors. Not edge cases either — a swapped longitude and latitude inside a geography function; a gold query using an Enrollment (K-12) column to answer a question about grades 1 through 12, which silently includes kindergarten.

The authors then corrected a sample of BIRD and re-ran all 16 open-source agents from its leaderboard. Scores moved by −7% to +31% in relative terms and positions moved up to nine places in either direction. The decisive result is the correlation: rankings on the uncorrected sample tracked the published leaderboard closely (Spearman's r_s = 0.85), while rankings on the corrected sample did not (r_s = 0.32, p = 0.23 — not statistically significant).

Put plainly: the leaderboard ordering is substantially an artefact of mistakes in the answer key. A separate line of work reached the same place from another direction, finding that BIRD's automatic execution-accuracy scoring agrees with human expert judgement only about 62% of the time.

The scoring problem cuts both ways

It would be convenient if this meant the models are simply worse than advertised. It does not. The errors run in both directions, and the more interesting direction is the one that makes models look worse than they are.

In that 2026 BIRD Mini-Dev run, the same answers that scored 56–68% under strict exact-match scoring scored 94–95% when a judge was allowed to assess whether the query actually answered the question. The gap was not luck. It was 90 cases where the model filtered more sensibly than the gold query, 19 where the number differed within rounding, 14 where an empty result was correct, and 11 where it returned fewer columns but answered what was asked.

What none of this measures

Even a perfectly annotated benchmark would leave out the things that actually break analysis on company data.

• Your vocabulary. Benchmarks ask questions whose terms are in the schema. Your team asks for “bookings,” “net revenue,” “billable,” “closed won” — words that mean something specific at your company and appear nowhere in the column names. The model has to guess, and the guess is invisible.
• Your exports. Real CRM and finance exports carry closedwon, Closed Won and CLOSED-WON in the same column across three years. A filter written against the wrong spelling returns zero rows, and zero rows looks like an answer.
• Structure the schema does not state. One-to-many joins that quietly duplicate revenue. Subtotal rows sitting in the same column as detail rows. Refunds landing in a later month than the sale. The BIRD databases average seven tables with no many-to-many relationships at all; that is not what a real export looks like.
• Repeatability. Every benchmark score is a single run. Nobody reports whether the same question asked next Tuesday returns the same number, which is the property a business report actually needs.

What actually reduces the error rate

The consistent finding across every benchmark family here is that models fail when asked to derive and succeed when asked to relay. Accuracy comes from narrowing what the model is allowed to decide. Concretely that means four things: define the vocabulary up front, constrain the generated query, refuse rather than guess, and make failures visible.

That is a design argument rather than a product one, and it applies to anything you build or buy. Since it is our own tool and I can describe it precisely, here is how each of the four is implemented in Quiriz — less as a pitch than as a worked example of what these guards look like when they are real.

1. A governed layer, so the metric is not re-derived each time

Metrics like win rate or billable ratio are modelled once, with their definition, their grain and their filters fixed. When a question names a modelled metric, it resolves to that definition instead of the model inventing a calculation. The same question next month compiles to the same SQL, because the SQL is not being written from scratch.

The honest limit: this covers metrics somebody has actually modelled. Anything outside that falls back to the general path — and the general path carries the error rates in this article.

2. Company Context, so your words mean your things

An admin writes down what the business means by its own terms — that a booking excludes renewals, that the fiscal year starts in February, which status values count as won — and that text is injected into the prompt behind every answer and report. It is the cheapest guard on this list and the one that addresses the vocabulary gap above most directly.

3. Absolute rules on the query planner

The planner that turns a question into SQL runs under a fixed set of constraints rather than free generation. Column names must come from the dataset's own mapping and are never invented. Datasets are aggregated individually before being joined, which is what stops a one-to-many join from silently multiplying revenue. Denominators are wrapped so a division by zero cannot produce a number. Subtotal rows are excluded from sums so hierarchical exports do not double-count. Every column referenced from an earlier step is checked against what that step actually produced.

None of these make the model smarter. They remove specific ways of being wrong that we have watched happen on real files.

4. Declining, and asking, instead of guessing

If a required column or filter value is genuinely missing, the planner returns an error rather than a best guess. Ambiguous questions are scored before execution and can trigger a clarifying question instead of an answer. A tool that occasionally says “I need to know which of these two you mean” is more useful than one that always answers, because the one that always answers is answering the ambiguous cases too — you just don't see it happen.

Frequently asked questions

Sources

• Jin, T., Choi, Y., Zhu, Y., Kang, D. (2026), “Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards”, University of Illinois — annotation error rates of 52.8% (BIRD Mini-Dev) and 62.8% (Spider 2.0-Snow), and the effect of correction on leaderboard rankings. Also published at CIDR 2026.
• Egg, A., Goyanes, M., et al. (2025), “DABstep: Data Agent Benchmark for Multi-step Reasoning”, Adyen — benchmark design, the easy/hard split and the 16-model baseline table.
• Spider 2.0 — enterprise text-to-SQL benchmark; current leaderboard across the Snow, Lite and DBT splits, and schema complexity figures.
• BIRD — execution accuracy on real databases and the 92.96% human expert baseline.
• MotherDuck (2026), “Your Data Model Is the Semantic Layer” — Claude Opus 4.5, GPT-5.2 and Gemini 3 Flash on BIRD Mini-Dev under strict versus judged scoring, and the column-comment result.
• Quiriz, AI Hallucination Statistics 2026 — hallucination rates across summarization, recall and reasoning benchmarks.

Figures compiled from published benchmarks, leaderboards and papers as of 25 August 2026. Scores are not comparable across benchmarks and should always be quoted with the evaluation that produced them; leaderboard positions change frequently, and the sources cited here document substantial errors in the underlying answer keys. Model names reflect those published by the benchmark maintainers at the time of the run and may not correspond to the current release of a given product.