Any data migration validation activity is complex which needs full attention because it always has a profound effect on the data which is one of the most valuable assets of an organization. Hence, in order to prevent loss and safeguard an organization’s assets, data verification and validation processes will become essential prerequisites.
The fundamental notion behind performing a data migration task is to conduct an ETL operation which is ‘Extract, Transform, and Load’ in as much as data is extracted from multiple sources, transformed, and loaded into the target data warehouses. This ensures that our testing team is not left out of the critical consideration points for planning their strategy for tests in order to help them data migrate activities.
Below are the steps involved in data migration process:
1.2.1 PRE-MIGRATION TESTING
There is a set of activities that we execute before pre-migration testing and prior to the actual data migration activity. This aids in understanding how the data is supposed to be mapped from its origin to the final location while also bringing to light the inconsistencies. The degree of these inconsistencies is critical in choosing whether to carry out a system design change or business process change in order to effect data migration to the target location.
Pre-Migration Steps are Assessed.
Examine the design document or detailed requirements.
Define the source and scope of data (Extraction Process).
Establish the extraction process, total record count, total table count, and also relationships.
Understand all the data suppliers for the system.
Review ERD (Entity Relationship Diagram), data dictionary, or similar documents available for the current legacy system.
Understand and define the target data-warehouse (Load Process).
Review ERD (Entity Relationship Diagram), data dictionary, or similar documents available for the current legacy system.
Study the data scheme which includes mandatory fields, field names, field types, data types etc. from both the original data source and the destination system .
Clean the data as necessary.
Understand any interfacing with other systems.
After completing the previous steps, we:
Align to the user interface.
Align to the business process.
Set up the software testing application.
Correct generation of scripts using the above mentioned mapping details.
Determine sample subset data (work with the client to gain acceptance and mitigate the risks).
Analyze business cases with user stories and use cases.
Data cleansing is critical during a data migration process and there is a high likelihood of some inaccurate data existing in the migrated data set. We perform data cleansing by validation of data during ‘Field Validation’ and ‘Error Checking’ as it is much less expensive and more efficient.
This set of tasks is not distinct, rather they are overlapping. We execute these tasks during all phases of the migration: prior to the migration, during, and after the migration process.
The main objective of the data cleansing is
1) to find incomplete, incorrect, inaccurate, or irrelevant data,
2) substitute it with the right data,
3) remove unclean data, and
4) ensure uniformity among different data sets which are combined from diverse sources.
Types of Errors
Some of the errors that may arise include blank fields, bad characters or data that is too long for the required length.
Examine the data to find the type of error.
Determine the procedure to validate data.
Check the database using SQL queries.
Apply tools that remove invalid data.
Understand how data relate to each other.
Changing data in one table can damage the data stored in another related table if the updates are not done throughout the entire system.
These procedures are followed in the process of data cleansing
Data Parsing: Parsing data extract one or more individual data elements from source files, which are then filled in the target files.
Data Correction: We fix any data inaccuracies using appropriate tools or secondary data sources.
Data Standardization: We change the data to fit the stipulated formats which could be business standards and/or customized rules.
Data Matching: We perform data matching on processed data to check for any duplication according to the business rules.
Data Consolidation: Merged processed data is analyzed to establish relationships and data which are found to be related is presented as a consolidated representation.
POST-MIGRATION TESTING
After the migration stage is done, black box testing/acceptance testing is done against business cases, use cases, and user stories. The same data subsets that were utilized during pre-migration testing are also utilized during post-migration testing. This activity is done in a test environment as opposed to the production environment.
Post-Migration Activities:
Evaluate how many records can be processed in a set amount of time (the throughput). Through this testing, we aim to verify that the downtime determined in advance will be sufficient.
Check the completeness and the accuracy of the migrated data.
Check the data record counts of the migrated data just to ascertain that all the data is actually migrated. Record count match only displays if data migration is completed or not. It does not disclose internal aspects like redundancy, validation, integrity, and other data attributes.
Check that the migration of contents is done according to the outline. We conduct this type of testing through sampling of data or automation tools.
Along with testing the known business workflows, our testers carry out the following tests. We also employ negative testing approaches primary to ensuring that data cleansing occurs at runtime within the system.
Input invalid data: We attempt to breach the validation rules utilizing boundary value analysis, equivalence partitioning, and error guessing. Validation at the user interface level should corroborate with the imposition of validation at the database level. If no validation is enforced or if the validation between the database and the user interface do not correspond, we treat this as a defect.
Skip over required data: We try to move on to subsequent steps of a use case in the system before all data requirements are completed.
Verifying data access locks: It must not be possible to allow multiple users access to the same new record within the database simultaneously.
The risk might be minimal as normal validation procedures are typically handled by the database system, but this testing still must be conducted anywhere deemed necessary.
Conversions testing: This process entails verifying that the data is accurately displayed upon request from the database(s) according to the requesting user’s access privileges.
Non-functional testing: These are other tests that we conduct outside of functional boundaries which include: security testing, performance testing, usability testing, etc.
USER ACCEPTANCE TESTING
After the post-migration testing is performed, we carry out user acceptance testing. We expect some level of functional weaknesses due to the consolidation of the migrated data during earlier stages, and this mitigation strategy is effective because the previous phases can overlook these subtle differences. It serves to enable interfacing with the legacy data in the claimed system prior to pushing it for production.
PRODUCTION MIGRATION
Even with successful execution of all the above phases, we acknowledge that there is no guarantee that the production process will be seamless. Some of the challenges/areas that we take care of at this stage are:
Procedural slips. Production system configuration errors.
If automated tools for software testing were utilized in the above phases, we prefer to execute those scripts here again. Otherwise, we perform sample or summary verification.
DATA QUALITY TOOLS
Here are some of the areas DQ tools can assist with,
Cost and Scope Analysis
Migration Scenario
Data Quality Governance
Sequence of Activities for Migration Execution
Self-Validation of Migrated Data on the Target System
Continuous Improvement of Data Quality
The following is a partial list of systems used for integration and migration of data:
The LamdaTest Survey Report indicates that 20.6% of professionals anticipate AI to increase testing efficiency,…
Technology Marketing: Strategies for Tech Companies The tech sector is a dynamic, competitive and interesting…
Playwright vs Selenium: Key Differences Between Both Frameworks: How can you choose the best software…
Top 6 Digital Marketing Success Stories To Learn From: Digital marketing is enhancing the standards…
Introduction to Google AdWords Campaign Management Services: Do you wish to increase the traffic to…
5 reasons why digital marketing is important: Companies have to change to remain competitive in…