QuerySurge Partner Atos helps validate a data migration project with over 10 billion records
Atos is a global leader in digital transformation with 111,000 employees and annual revenue of c. € 11 billion. European number one in cybersecurity, cloud and high-performance computing, the Group provides tailored end-to-end solutions for all industries in 69 countries. As a QuerySurge partner, they are skilled in the implementation of QuerySurge, which helped them deploy a robust data testing solution in a short timeframe to validate an extensive database migration project for their client.
A large telecommunications company was planning a database migration from Oracle to a comparable enterprise level full-stack application. Atos was brought on to assist with validating the migration, which consisted of more than 10 billion records related to the company’s backend services, solutions batch processes and configuration servers. In addition to the data base technologies, flat files (CSV format) were incorporated into the validations as part of resource exports generated from the source system.
One of the main challenges was to define the scope of tables to validate, given the substantial number of records and the short project time frame, which was only a few months. Comparing the data points manually would have been inadequate given the time and resources available. An automated data validation and ETL testing solution was necessary to achieve the level of coverage required by the client’s stakeholders and to meet their project timeline.
As a QuerySurge Alliance partner, the Atos team was well equipped to implement and deploy an automated solution centered around QuerySurge. To define the scope of testing, tables included in the migration were categorized based on priority and criticality. The testing effort was then divided into three iterations with a big bang migration to production planned after testing was complete.
Using QuerySurge, connections were established to the source and target systems. A suite of 421 QueryPair tests were created to perform the validations. Issues discovered during the testing effort were resolved by the migration development team and verified by re-running the QueryPairs. At the end of each iteration, QuerySurge provided the confidence to proceed forward with a successful migration.
Average table validation: 351 million records
Largest table validated: 4.8 billion records
Total records validated: 9.2 billion records
Total tests created: 421 QueryPairs
QuerySurge Server Specifications:
- Processor Cores: 13
- RAM: 64 GB
- Hard Drive: 5 TB SSD6
QuerySurge provided a robust testing solution which helped to validate all critical tables included in the migration. It was easy to deploy and integrate with the client’s source and target data sources. Testing was able to ramp up quickly and achieve the required scope coverage established by the client. Without QuerySurge, the team would not have been able to validate the critical tables included in the migration.
The client team was so impressed by the solution that they are now looking to implement it across their other data testing efforts. Utilizing QuerySurge along with the expertise of the Atos, the client is now well positioned to increase data confidence across their platform.
The PMO officers were delighted about the insights we have generated and the security guarantees we were able to provide to the success of the project.
QuerySurge is the smart data testing solution that automates the data validation and ETL testing of Big Data, Data Warehouses, Business Intelligence Reports and Enterprise Applications with full DevOps/DataOps functionality for continuous testing.