Manual Data Validation
Cannot Keep Up

Technical Challenge

Your test team wrote SQL queries to test data across different legs of the data process. But they need to execute them manually using a SQL editor, export them into Excel, and compare them by eye.​

Or they’re using Minus Queries to subtract one data set from another. ​

Or a homegrown tool with limited features and minimal reporting.

Or a framework from a system integrator.

These methods are time-consuming, resource-intensive, inefficient, and lacking in reporting features.

How can you automate and improve this process?

The QuerySurge Solution​

QuerySurge, the leading AI-powered data quality platform that continuously automates data validation across your entire ecosystem, fits well into any DevOps for Data strategy.

Here are some of QuerySurge’s key features:​

• Connect to 200+ data stores quickly and easily with the Connection Wizard​
• Create tests visually in minutes to compare tables, columns, and row counts with the Query Wizards – no coding needed​
• Leverage Artificial Intelligence to convert all data mappings into tests, and generate individual tests with our chat interface, through QuerySurge AI
• Create custom tests, set thresholds, leverage reusable queries in our Design Library​
• Compare billions of rows of data from source files and data stores to the target data warehouse and big data lakes
• Leverage the Flexible scheduling to run tests immediately, on a specific date & time, or automatically after an event ends
• Produce informative reports, view updated dashboards, and send auto-email results to your team

Now Let’s Automate:​

ETL automation:​

• Import your existing tests with our Import/Export feature​
• Create table-to-table and column-to-column tests quickly and visually with our Query Wizards​
• Create transformational tests — with no programming needed — with QuerySurge AI​
• Schedule a date & time to kick off tests or have your Data Integration/ETL software kick off QuerySurge through our Restful API after the ETL process completes its loading.
• Execute all tests and report all results, including pass/fail results and complete details of each data failure with our Data Intelligence dashboards & Data Intelligence reports.​
• Send an automated email to everyone on the team with the test results.

DevOps for Data, CI/CD, and Continuous Testing:​

• Seamlessly connect with virtually any data integration, CI/CD, automated build/delivery, or test management solution through QuerySurge’s DevOps for Data module.​
• Use our built-in Swagger documentation to explore, test, and validate API calls before using them in production.
• Programmatically create, execute, and update tests and data stores 
• Create or modify source and target queries
• Manage data store connections
• Generate staging tables from various data platforms
• Add custom flow controls based on run-time results
• Send an automated email to everyone on the team with the results.

Webhooks Integration:​

• Utilize QuerySurge Webhooks to provide real-time integrations with your DevOps, CI/CD, and alerting tools. 
• Send a Slack or Teams message when a data test fails
• Kick off other processes automatically (like sending alerts or creating a task)
• Open a ticket in Jira automatically if there’s a problem
• Start a job in Jenkins or Azure DevOps to fix or roll back the data
• Log results in tools like Splunk or Datadog so your team can review

FAQ: Challenges to Automating the Testing Effort