Operations Team

The Operations team uses QuerySurge to perform Operational Monitoring of production ETL runs on a daily basis.

Operational Monitoring

• DevOps automates the execution of a library of SQL pairs on a regular daily cycle that start immediately after the ETL or Build/Release process ends, using QuerySurge’s execution API.
• Reports that monitor the data validations are triggered when the tests complete and are distributed to stakeholders through QuerySurge’s automated email functionality.
• Issues are quickly identified and flagged.

ETL Development Team

ETL Developers uses QuerySurge to perform Unit Testing.

Unit Testing

• The ETL Developers use QuerySurge to build Unit Test QueryPairs as ETL code is developed, for immediate testing as code is committed in the development environment
• Developers build tests ‘on-the-fly’, execute them, and see immediate results from QuerySurge
• Developers execute with large result sets to profile more data than would otherwise be feasible
• Developers catch issues in their code quickly in the Unit Test phase with QuerySurge, reducing remediation costs to the project.

Test Team

Testers use QuerySurge to perform both Functional and Regression testing.

Functional Testing

• Testers create and execute tests during the development cycle to provide the development team with quick feedback on each ETL build deployed during development
• QuerySurge is used for rapid high-volume data testing during the development cycle – for whatever period and at whatever sample size makes sense for the project
• Tests are run when each ETL leg completes, helping to pinpoint where defects appear in the code. QuerySurge’s execution API can be used to trigger execution automatically when each ETL leg completes, for 24x7 continuous testing.

Regression Testing

• Testers execute a full regression test covering 100% of all ETL mappings prior to deployment of the ETL code to production. The team tests a large sample of the data from each source system.
• Test data sample sizes up to 100% (full coverage) for each source system can be validated.
• QuerySurge generates Data Intelligence reports that provide audit-able records of official test cycles for organizations that require downstream audits of testing activities

Analyst Team

Analysts Use QuerySurge to verify that the tests meet specifications and to analyze data.

Data Analysis

• Analysts use Data Analytics Dashboard & Data Intelligence Reports to provide insight into the data, help find problematic areas and provide root cause analysis
• Analysts examine QueryPair assets to verify that the tests match the requirements in the mapping documents
• Analysts examine preliminary QuerySurge Scenario runs to evaluate whether the queries tested a mapping correctly
• Then, Analysts verify whether failures identified by QuerySurge are deemed true failures or whether they are acting according to specifications (i.e. type mismatches, row count difference, etc.)