Before AI Acts:
Validate the Data Behind the Decision

A practical guide to connecting business rules, automated tests, and validation evidence

Ai validation shield

Executive Summary

This guide is for data, quality engineering, BI, and governance leaders evaluating the data behind AI models and agents. It follows an illustrative invoice through testing and a rule change, explains how to maintain validation in practice, and provides a six-question buyer checklist. The objective is a process that can explain what was tested and why data was accepted for its intended use.

An AI agent preparing a supplier payment can receive a complete invoice and still use the wrong exchange rate. A recommendation can rely on an outdated inventory balance. The business needs evidence that the information meets the requirements of the decision being made.

That evidence starts with an approved definition of correct: the records in scope, the applicable business rule, the expected result, and the permitted tolerance. Validation turns those expectations into repeatable tests. The surrounding workflow connects failures to a defined response, such as holding a transaction or requesting review.

The work continues when requirements change. A new currency policy or reporting definition can make an existing test obsolete even while it continues to pass. Teams need to identify affected tests, review their expectations, and retain evidence of the results.

QuerySurge supports this process by connecting documented requirements with validation assets. Mapping Studio brings mappings and tests into one environment and integrates with QuerySurge AI to generate tests. Validation Lineage helps teams inspect those relationships and identify assets that may need review after a change.[1][2]

Recent surveys show the concern. Fivetran's 2026 survey of 400 data professionals found that 15% of surveyed organizations were fully prepared to support agentic AI in production.[3] In dbt Labs' 2026 survey of 363 practitioners and leaders, 71% expressed concern about hallucinated or incorrect data reaching stakeholders.[4]

Use Case Video

Step inside the payment scenario to see how QuerySurge catches a payment error before the AI agent acts.

This short video shows you the entire process.

1. Follow a business rule from requirement to evidence

Consider a payment-preparation workflow in which an AI agent uses approved supplier invoices to prepare payment instructions. Finance requires foreign-currency invoices to use its approved exchange rate for the invoice date. The following scenario uses hypothetical records, rules, and results to illustrate a validation design.

Invoice AP1042 is for 10,000 Euros. At the approved rate of $1.08/Euro, the expected amount is USD $10,800. After a pipeline change, the transformation instead uses a current rate of $1.18/Euro and produces $11,800. The load succeeds, the invoice count is unchanged, and required fields are populated.

Illustrative comparison for invoice AP1042

Invoice

Expected USD​

Actual USD

Difference USD​

Result

AP1042

10,800.00

11,800.00

1,000.00

Fail

Document the approved requirement

Record the source amount and currency, target payment amount, authoritative rate table, date used to select the rate, rounding convention, and acceptable difference. Assign Finance as the rule owner. Specify how to handle missing rates and which invoices the requirement covers.

In QuerySurge, a team can create or import this mapping in Mapping Studio and use Mapping Intelligence to generate validation tests from it.[1] Document the other requirements for the process as well, including valid supplier contracts and duplicate-invoice criteria.

Review the test and its expected result

Calculate the expected USD amount from Finance's approved rule and reference data, separately from the production transformation. Review the generated SQL for the rate-date join, currency pair, rounding, and treatment of missing values. Exercise known valid, invalid, and boundary cases before relying on the test.

For AP1042, the test should compare USD 10,800 with USD 11,800 and report a USD 1,000 difference. In this example, differences above USD 0.01 fail. A separately reviewed expectation makes it less likely that the test will reproduce the same mistake as the pipeline.

Connect failure to an operational response

Configure the payment-preparation workflow to hold an affected invoice when the required test fails or cannot complete. QuerySurge's APIs allow an external workflow to execute a test suite and retrieve execution status.[5] The release decision must check the required test outcomes, including failures and execution errors; completion alone does not establish that validation passed.[6] Test the configured response before operational use. Payment approvals and access controls remain part of the surrounding application.

Retain the invoice identifier, requirement and test versions, data snapshot, expected and actual amounts, and resolution. After the transformation is corrected, rerun the relevant checks before releasing the invoice.

Review the tests when the business rule changes

Now suppose Finance approves a contract rate of 1.10 for a defined supplier group, starting on a specified effective date. Other invoices retain the invoice-date rule. A new EUR 10,000 invoice within the revised scope should produce USD 11,000. A pipeline and a test that both continue to use 1.08 could agree on USD 10,800 while missing the new requirement.

Update the mapping with the revised supplier scope, rate source, and effective date. Use Validation Lineage to inspect its connected queries, test suites, and recurring schedules, then decide which assets need review.[2] A displayed relationship helps locate a test; reviewers must still determine whether it covers the revised rule.

Check invoices on both sides of the effective date, suppliers inside and outside the new group, and missing contract rates. Review shared query logic, update affected expectations, and rerun the relevant suites. Retain which rule applied to each tested batch and the evidence supporting the release.

Querysurge validation lineage querypair

Figure 1. Published QuerySurge product view of a selected QueryPair and its connected mapping, reusable query snippet, test suite, and schedule. The screenshot illustrates asset relationships; the invoice scenario above is a separate hypothetical example.[2]

Teams define relationships between mappings and relevant queries. QuerySurge identifies other dependencies using reusable snippets, QueryPairs, suites, and schedules.[2] Review coverage by business requirement, including conditions that a linked test does not exercise.

2. What data validation establishes

A useful validation test connects a business requirement to an expected result. It specifies which records are in scope, how the result should be calculated, which differences are acceptable, and what constitutes a failure. Passing the test provides evidence about those conditions at the time tested.

Source-to-target agreement is one part of that evidence. If an incorrect source value is copied faithfully, reconciliation can pass while the underlying business problem remains. A complete plan therefore includes checks on source data and business meaning, alongside tests of movement and transformation.

Data validation belongs within a broader AI assurance program. Language models can produce incorrect statements even with error-free training data; research identifies training and evaluation incentives among the contributing factors.[7] Teams should evaluate model behavior, retrieval quality, access permissions, and authorization of actions alongside the enterprise data those systems use.

Working with the existing data environment

Governance establishes responsibilities and policies. Lineage helps trace dependencies and origins. Monitoring and observability help teams detect operational and data changes. Validation tests specified expectations. These activities overlap and should share information, including rules, incident records, and affected datasets.

Modern data platforms also provide useful testing capabilities. Snowflake supports custom data quality checks, including comparisons across datasets. Databricks supports SQL-based pipeline expectations that can identify invalid records and influence processing.[8][9]

An enterprise validation program should assess how well those capabilities cover the complete business flow. The need may span a source application, several transformation platforms, a reporting layer, and an AI input dataset. A common testing approach can help teams apply approved requirements consistently across those boundaries.

Independence depends on the expected result and the test design. Reusing the production calculation as the only reference can reproduce its error. Business review, separately derived expectations, and tests that deliberately exercise failure conditions make the evidence more useful, wherever the tests execute

3. Build a practical validation program

Select the business process and its critical data

Select a process where a data error would have a clear consequence. Identify who uses the result, what action it supports, and where an error could be contained.

Trace the required fields and calculations from their authoritative sources through transformation and consumption. Include reference data, filters, joins, refresh schedules, and semantic definitions. Record important handoffs between teams and systems. This establishes a manageable scope for the first validation suite.

Prioritize the requirements that matter most to the decision. A payment workflow may require strict checks on amount, currency, supplier identity, and duplicate status. An optional descriptive field may warrant a different response. Make those priorities explicit before designing the tests.

Agree on the expected result and assign owners

Business owners should approve what the data means. Data engineers and testers should translate those definitions into executable checks. The application or process owner should decide how failed checks affect downstream use.

Translate broad goals into testable requirements. A revenue rule should identify the included transactions, treatment of returns and cancellations, applicable currencies, reporting period, and approved rounding.

For each critical requirement, record the business owner, authoritative reference, population to be tested, expected result, tolerance, execution trigger, and failure response. Assign responsibility for maintaining the test when the rule changes. A documented rule without an owner can become obsolete just as easily as undocumented logic.

Design checks that cover movement and meaning

Use complementary checks to address different ways the data could fail. Counts and totals are useful starting points, but they cannot show that every important value is associated with the right record. Choose comparisons that match the transformation and the business question.

Validation area

Question to answer​

Example check

Completeness

Did the expected population arrive?

Reconcile records by key and period, including rejected and late-arriving data

Transformation logic

Were approved calculations and mappings applied?

Compare converted amounts or classifications with independently calculated results

Business rules

Does each record satisfy the intended conditions?

Validate required relationships, effective dates, permitted values, and duplicate criteria

Freshness and timing

Is the data current enough for the decision?

Check refresh completion, event timestamps, and allowable delay

Consumption

Does the downstream result preserve the intended meaning?

Compare an AI input dataset or supported report result using matched filters and definitions

For aggregations, compare results at the agreed level of detail. An order-line source and a customerlevel target require a defined aggregation before comparison. Align time zones, precision, null handling, and comparison periods so that differences reflect defects rather than inconsistent test assumptions.

Use full-population comparisons where the requirement and consequences warrant them. For frequent or large workloads, design incremental checks around changed records or partitions, supported by broader reconciliation at suitable intervals. If sampling is used, document what it can establish and what remains outside its coverage.

Check the tests themselves. Include known valid cases, deliberately invalid cases, and boundary conditions. Review generated SQL for the correct joins, filters, calculations, and expected failure behavior. Successful execution alone does not establish that a test asks the right question.

Run validation at the points where it matters

Separate checks on software changes from checks on newly arrived data. A deployment test asks whether a changed pipeline still meets its requirements. A recurring data check asks whether the latest load meets them. Many business processes need both.

Run relevant regression tests when schemas, mappings, transformation code, or business rules change. Run operational checks after required refreshes complete and before important downstream consumption. Define how to compare consistent snapshots or processing windows, especially when sources update at different times.

Execution frequency should follow the use case. A daily reporting dataset may be validated after its scheduled load. A live decision service may require fast input checks, supplemented by deeper reconciliation outside the immediate response path. Account for query cost, source-system load, and the time available to act on a failure.

Specify the response to a failed check

Define whether a failure should block publication, hold affected records, trigger manual review, or allow processing with an approved exception. Avoid treating every warning as equally consequential. Separate a data mismatch from a test that could not run because a connection or service was unavailable.

Connect the test result to the workflow that can enforce the decision. A failed test becomes a release control only when the pipeline or application is configured to act on it. Assign an owner, notification route, expected response time, and escalation process.

Test that response before relying on it. Confirm that a critical failure actually prevents the intended release, that authorized users can review exceptions, and that any override records who approved it and why. After correction, rerun the relevant checks and consider whether downstream outputs require rebuilding.

Retain evidence and maintain coverage

Preserve enough information to explain what was tested and what happened. Depending on the use case, this may include the requirement and test versions, execution time, data snapshot or batch identifier, expected and actual results, thresholds, defects, and resolution.

Set retention, access, and protection requirements for test evidence. Detailed results can contain sensitive business data. The evidence needed for investigation may differ from the summary needed by a business owner or auditor. Align the retained records with the organization's applicable control requirements.

Review coverage whenever requirements or data flows change. Confirm that linked tests still check the current rule and relevant conditions. Track unresolved failures and accepted exceptions alongside the results.

4. Apply the framework to AI inputs

The underlying discipline remains the same across AI use cases: define the intended use, establish expectations, and test them at the right point. The questions change with the role the data plays.

Training and evaluation datasets

For models developed or adapted using enterprise data, check dataset assembly, labels, transformations, duplicates, and time boundaries. Verify that joins preserve the intended population and that records in evaluation sets do not improperly overlap with training data.

For predictive tasks, examine whether each input would have been available at the time of the decision being modeled. Information from a later event can make a historical evaluation misleading.[10]

Data science teams should also assess whether the population and labels are suitable for the intended use; matching source records alone cannot answer that question.

Inputs used in production

Check that live inputs follow the definitions and transformations expected by the model. Changes to units, category codes, defaults, or feature calculations should trigger review and appropriate regression tests. Confirm that the same concept is calculated consistently during training and production use.[10]

Freshness requirements should be specific. Define the maximum acceptable age for a balance, inventory quantity, or eligibility status, and decide what the application should do when current data is unavailable. Keep these data checks distinct from evaluation of the model's predictions or responses.

Retrieved information and agent workflows

When an AI application retrieves business information, validate the accessible datasets and reference content that support the intended task. Check identifiers, effective dates, document versions, and important relationships. Also evaluate whether retrieval returns relevant information and whether the answer is supported by that information.[11]

An answer can be supported by retrieved content even when that content is inaccurate. Source validation and response evaluation therefore address different questions. Teams should retain the connection between the source version, the retrieved context, and the response when investigating an error.

For agents, identify the specific data used to select or authorize an action. Define whether a failed check should hold the transaction or route it for human review. Connect those decisions to the application's access permissions, tool controls, transaction limits, and approval rules.

5. Evaluate QuerySurge in your environment

Apply the same expectations to a platform evaluation. Start with a representative data flow and inspect the requirements, test logic, results, and operational response. Mapping Studio and Validation Lineage support the requirement and change-review work illustrated earlier; the following capabilities extend the process across the data environment.

Create tests and compare data across systems

Query Intelligence helps users create and refine SQL through a conversational interface informed by selected schema metadata.[12] Business owners and testers should review the resulting requirements and queries, including how the expected result was derived. AI assistance can reduce the work of authoring tests while keeping that review part of the process.

QuerySurge supports more than 200 data stores through supported drivers and connectors, including vendor and partner connectivity.[13] Teams can use that reach to compare source data, transformed datasets, and downstream outputs across different technologies.

BI Tester extends validation to data retrieved from supported BI reports. For example, its Power BI integration supports retrieving report data and comparing it with another supported source.[14] For an AI workflow that depends on reported business measures, this provides an additional place to test the data and calculations used downstream.

Integrate execution and retain useful results

Use QuerySurge's APIs to connect suite execution and status retrieval to your deployment or data release workflow.[5] During evaluation, exercise both a data mismatch and an execution failure. Confirm that the configured workflow makes the intended decision in each case.

Administrators can configure retention of detailed query results and summary information.[15] Those settings should reflect investigation needs, evidence requirements, and the sensitivity of the data. The surrounding process should connect failed checks to ownership, remediation, and approval of any exceptions.

6. Start with one consequential data flow

Select one AI use case and follow its critical data from source to consumption. Agree on the business rules, assign owners, and create an initial set of reviewed tests. Run the suite against representative data, including known failures, before making it part of a release or operational workflow.

Measure progress against the requirements in scope. Useful measures include the share of approved critical requirements covered by reviewed tests, unresolved failures by severity, time to investigate and resolve defects, and releases completed with approved exceptions. Define each measure consistently; a high pass rate has limited meaning when important requirements remain untested.

Expand the program as the team learns which checks detect meaningful defects and how quickly it can respond. Add new data flows, refine thresholds, and review coverage when the business changes. The goal is a working process that can explain which requirements were tested, what failed, and why the data was released for its intended use.

See the process in QuerySurge

Bring a representative mapping, a supported source and target, and one critical business rule to a QuerySurge demonstration. Use a controlled example to follow the requirement into a reviewed test, inspect the comparison results, and examine connected assets after a rule change.

Ask the team to show the generated SQL, a deliberate failure, and the tests that need review when the mapping changes. Discuss how execution status would connect to your release process and what evidence your team needs to retain. Use the buyer checklist to record the answers and identify the scope of a practical evaluation.

Schedule a QuerySurge demonstration

About QuerySurge

QuerySurge is an enterprise-grade data validation platform that continuously automates the validation of data across your entire ecosystem - from data warehouses and big data lakes to BI reports and enterprise applications.[16]

Audit-ready evidence, down to the row
Audit-ready evidence, down to the row

Buyer checklist for data validation

Use these six questions to evaluate an internal program or a validation platform. Ask to see evidence from one representative business process, including a failure and a rule change. Record gaps, owners, and next steps.

1) Which business requirements are being tested, and who owns them?
Look for an approved requirement linked to each test, with a named owner, the decision it supports, the records in scope, and the permitted tolerance. The rule should be precise enough for someone else to review the expected result.

2) How is each expected result calculated and reviewed?
Inspect the authoritative reference data and calculation. Ask how the expectation was checked separately from production logic. Known valid, invalid, and boundary cases should demonstrate that the test detects the intended errors.

3) Which critical requirements still lack adequate tests?
Review requirements against the tests that exercise them across relevant sources, transformations, and downstream inputs. Check whether coverage includes all required records, changed records, or a sample. Identify exclusions and conditions that a linked test does not cover.

4) What happens when a test fails or cannot run?
See a failure trigger the configured response, such as holding records, stopping a release, or requesting review. Confirm the responsible owner, notification, and escalation route. Distinguish a data defect from a test that could not execute.

5) When a business rule changes, how are affected tests identified?
Start with a rule change and inspect the connected mappings, queries, suites, and schedules. Ask how expected results are reviewed, which checks are rerun, and how the team establishes which rule applies to each dataset.

6) What evidence can the team retain and explain afterward?
Request a record of requirements and test versions, data snapshot or batch, expected and actual results, thresholds, and outcome. Where applicable, include remediation and exception approvals. Confirm retention periods and access permissions.

Sources

Research and documentation accessed November 9 and 10, 2026.