When we decided to benchmark WisdomAI against Claude Code, we didn't know how it would turn out. The results surprised us.
A vendor benchmark where the vendor wins? We'd be skeptical too. In this post, we'll show you the setup and scoring, so you can judge the benchmark for yourself.
Why compare with Claude Code?
Our team uses both Claude Code and Claude Cowork in day-to-day work. We know the tools well, so we wanted to test how Claude Code handled the analytics questions a business user would ask.
If your company is already using Claude, connecting it to your warehouse and data sources is a natural next step. Before adding another product to your analytics stack, you'd want to know: how far can Claude take us, and what would a dedicated analytics system add?
Anthropic's own account of how its teams use Claude for self-service analytics informed the setup. Their team found that accurate answers depend on well-maintained data models and curated business context, which they supplied through skills. We built our test to understand how a built-for-purpose analytics architecture, like WisdomAI, helps harness that context to improve outcomes.
The test controlled for the same questions, data, schema, and definitions to measure:
Accuracy: did it get the right number?
Consistency: did it get the same number every time?
Completion time: how long until the answer, SQL and chart were ready?
Cost per question: what did each answer cost, across every model call it took?
Experiment setup
Data and questions
The benchmark uses data from LiveSQLBench, developed by the team behind the BIRD text-to-SQL benchmark, because it is designed to reflect the complexity of business analytics. The dataset also includes its own documentation, with schema notes and business definitions. Both systems received those materials exactly as published for the documentation stage.
Most of LiveSQLBench’s public questions are moderately difficult. To broaden the coverage and make the test more aligned with actual enterprise data questions, our context engineers used Claude to write more complex questions. The questions were written in a separate session that never saw either system, how they were set up, or any results.
For each new question, we generated the SQL for the expected answer and ensured it ran correctly against the database. We then locked the question set and answer key, which was not adjusted after testing began.
What we held constant
What | How |
Setup | The same 30 questions, answer key, and Postgres warehouse, with equal access for both systems. The setup stayed fixed throughout testing. |
Documentation | Both systems received the same published files in Stage 2. We checked that each run had only the context allowed for its stage. |
Fresh sessions | Every run started with no conversation history from previous attempts. |
Three runs per question | A question counted as correct only if at least 2 of 3 runs were right. The results held, as repeat runs consistently missed the same questions. |
No clarifying questions | We disabled WisdomAI's ability to ask for clarification, so both systems answered every question as written. |
Equal output | Both returned the answer, SQL, and a chart. We didn't grade chart quality; any chart built from the right data counted. |
Testing occurred across three stages
Stage 1: Cold start | Both systems | Warehouse only; No documentation |
Stage 2: Documentation | Both systems | Warehouse & BIRD’s documentation |
Stage 3: After learning | WisdomAI only | Automated exploration; no reference SQL supplied |
Before Stage 3, WisdomAI's built-in learning loop explored the dataset and saved context without receiving reference SQL. We then reran the same questions in fresh sessions. This stage only exists for WisdomAI because Claude doesn’t have an automated learning equivalent. If we’d built a Stage 3 for Claude Code, it would have been our own engineering — not a real comparison.
How we evaluated the answers
During evaluation, we ran each system's SQL against the warehouse and checked its answer against the answer key. Different queries could still count as correct.
Our evaluation used two kinds of checks:
Deterministic: We compared the returned rows and values with the expected result, allowing cosmetic differences.
Pass: Renaming "Revenue" to "Total revenue"
Fail: Returning 40 customers when the expected result contains 39
LLM classification: We reviewed both system’s generated SQL queries, result summaries, and sample rows, checking to see whether the queries answered the same question: i.e., did they use the right filters and grouping, and did the return match the explanation.
Pass: 0.25 and 25% expresses the same conversion rate even though a direct value comparison fails
Fail: Omitting a filter for paying customers changes the question being answered
Both systems used the same scoring rules and classifier. There was no partial credit.
Results
After learning, WisdomAI reached 93% accuracy at roughly one-third of Claude Code's cost and completion time.
Time and cost are medians per question, comparing WisdomAI after learning with Claude Code with documentation. Time includes full completion of the answer, SQL and chart for both systems.
Unpacking the results
Business context helped both systems
Writing valid SQL is only part of the job; the system also needs to know how the business defines its metrics. Documentation raised accuracy from roughly 40% to roughly 75%. Without human intervention, that’s Claude’s ceiling, underlining that a general-purpose agent isn’t reliable on its own.
WisdomAI starts with relevant context
In our test, Claude Code queried the warehouse and explored gaps as it worked through each question, using tokens and time. WisdomAI preloads relevant table information, definitions and available learned query patterns, then explores on demand when something is missing. Less repeated discovery and less information to process help explain the lower token use, cost and completion time.
Learning builds context for later questions
Given a warehouse connection, business questions and rough feedback about expected answers, WisdomAI's Adaptive Context Engine can explore the data, test calculations, and save useful queries and business rules for reuse. These analytics workflows are built into WisdomAI's Analytics Harness.
For example, in our earlier learning-loop evaluation, WisdomAI initially calculated an average repair cost of $4,850. Feedback placed the expected cost between $1,500 and $2,000. It tested alternative calculations and saved the formula that produced $1,541, without receiving the reference SQL.
While WisdomAI made significant accuracy gains with auto-learning in place, that’s still not viable in an enterprise setting. We'll cover what it takes to close the remaining accuracy gap in production in an upcoming post.
Outside our lab
A large technology customer ran their own independent benchmark study, comparing a Claude instance they’d already tuned to their data vs. Claude paired with WisdomAI. They wanted to decide which approach to use in production.
When both were given access to the warehouse directly with Claude using WisdomAI, they found WisdomAI + Claude was roughly 5x faster and 3x more cost efficient than Claude alone. Today, they use Claude with WisdomAI in production.
Don't just take our word for it. Pick 20 to 30 questions your team actually asks, and have your team write and check the correct SQL answer for each one. Then follow the same rules we used above. Your results will tell you more than ours can because they come from your own data.
Schedule an intro call to get a deeper look at WisdomAI, and see how we can help you deliver higher accuracy at lower cost and latency in your own data analytics use case.
