Scaling Enterprise SQL RAG to ~95% Accuracy

Natural-language analytics looks simple when the database is small. It becomes a very different problem once the system has to reason across dozens of related entities, hundreds of thousands of operational records, ambiguous business terminology, and multi-turn analytical questions.

We built an enterprise SQL RAG system for that environment. The final product reached the mid-90% range on a controlled end-to-end evaluation set, answered most analytical questions in the low tens of seconds, and reduced model cost per correct answer materially compared with the first production prototype.

The difficult part was not teaching an LLM to write SQL. It was teaching the overall system when not to guess.

Where basic text-to-SQL broke

Our early prototype worked well on clean questions. Ask for a simple aggregation, ranking, or time-based comparison and the model could usually produce something plausible.

Real business questions were harder.

A request such as:

Which accounts performed best this quarter?

does not have one universal answer.

“Best” could mean growth, revenue, conversion, activity, margin, or another company-specific metric. Several interpretations can produce perfectly valid SQL, and that is exactly what makes the failure dangerous: the database executes the query successfully even when the business interpretation is wrong.

As the data environment grew, we saw the same pattern repeatedly. Syntax errors were easy to catch. Semantic errors were much harder because the results still looked believable.

That moved the focus of the project from SQL generation to business understanding.

Business context became part of the data layer

A schema tells an LLM which tables and columns exist. It does not explain how an organization defines terms such as active, qualified, completed, recognized, or high value.

We therefore treated approved business definitions, reporting conventions, valid categories, and important analytical terms as first-class context.

The goal was not to give the model more text. It was to give it less, but more relevant, context.

That distinction mattered.

In a wide relational environment, supplying the entire schema increases token usage while also increasing the number of plausible but incorrect joins, filters, and fields available to the model. Narrowing the context to the entities and business concepts relevant to the current question improved both consistency and cost.

Ambiguity also became an explicit product state. If two interpretations could materially change the answer, the system could ask one short clarification instead of silently choosing one.

That small design decision prevented a large class of confident analytical errors.

Cost optimization: from cost per call to cost per correct answer

The first production version was expensive for a simple reason: too much work was being delegated to high-capability models.

A single question could trigger intent understanding, context retrieval, relational reasoning, analytical generation, validation, explanation, and a follow-up response. Using the strongest model with a large context window for every step worked, but it did not scale economically.

We started measuring a different number:

That changed the optimization strategy.

Smaller models could handle lightweight normalization, formatting, and simpler classification tasks. More expensive reasoning was reserved for ambiguous or structurally difficult questions. Stable business context and frequently reused metadata could be reused rather than regenerated repeatedly.

Reducing irrelevant context was especially effective because it improved three things at the same time: fewer tokens, lower latency, and fewer wrong analytical paths.

Across successive iterations, model-side cost per successful answer fell materially while the evaluation score remained within the target band.

The important lesson was that cost optimization was not primarily about finding a cheaper model. It was about spending model intelligence only where it changed the outcome.

Evals changed how we built the system

At first, we measured obvious engineering signals: did the generated request execute, did the result render, and did the answer look reasonable?

Those metrics were too weak.

A query can execute without errors and still use the wrong reporting period, wrong grouping, wrong business definition, or wrong context from a previous turn.

We moved the benchmark to the final business answer.

The evaluation set contained several dozen carefully selected analytical questions covering filtering, aggregation, ranking, period comparisons, multi-condition questions, ambiguous terminology, unsupported values, and conversational follow-ups.

A response counted as correct only when the final interpretation and returned result matched the expected benchmark answer.

That produced a much more meaningful score. The strongest evaluated version landed at roughly 95% end-to-end accuracy on the controlled internal set.

More importantly, every failure was classified.

Instead of recording only “wrong answer,” we separated failures into categories such as business interpretation, retrieval quality, temporal reasoning, grouping, filter behavior, conversational context, and result presentation.

That turned evaluation into an engineering tool.

If accuracy moved down, we could see why. If a new model or retrieval change improved one class of questions but hurt another, the regression suite exposed it before release.

Human feedback became an RLHF-style improvement loop

Offline evals tell you about the failures you already know how to test.

Users find the failures you did not anticipate.

We added lightweight feedback to analytical answers and stored enough surrounding context to make a negative result reproducible: the original question, interpreted intent, important assumptions, resulting answer, system version, and user feedback.

We describe this as an RLHF-inspired loop, not continuous online retraining.

Raw thumbs-up and thumbs-down signals are noisy. A user may dislike a technically correct answer because of terminology, formatting, or a changed business convention. Training directly on every rating would introduce new errors.

Instead, reviewed feedback was used to improve the system in several ways. Real failures were promoted into new eval cases. Repeated preference patterns informed answer behavior. Retrieval mistakes revealed missing or irrelevant business context. Hard examples helped us decide when stronger reasoning was worth the additional cost.

Over time, curated examples can also become useful preference data for supervised fine-tuning or preference optimization.

The key was treating feedback as evaluation data first and training data second.

What the later system looked like

The exact client dataset and commercial metrics are intentionally abstracted, but the operating profile was roughly:

These numbers were not optimized independently.

Lowering inference cost at the expense of semantic accuracy would have created a cheaper but less useful product. Increasing accuracy with unrestricted context and expensive models would have produced a system that was difficult to scale.

The production target was the intersection of accuracy, business understanding, latency, and cost.

What we learned

The most expensive failures were semantic, not syntactic. Invalid SQL is obvious; a valid answer based on the wrong business assumption can pass unnoticed.

More context was not always better. In complex schemas, irrelevant context increased both cost and ambiguity.

Evals were more valuable than repeated prompt tweaking because they gave every engineering change a consistent test. Human feedback became significantly more useful once those eval categories existed, because each production failure could be converted into a measurable regression case.

And RLHF was ultimately a data-quality problem. Collecting ratings was easy. Creating clean, reviewed preference data that represented what a better analytical answer actually meant was the difficult part.