When performing any type of data migration, validation and testing is imperative to mitigate the risk of unplanned downtime, loss or corruption of data. It is also necessary to ensure functionality of the current system is preserved during the migration process. Cutting corners by failing to perform data migration validation best practices can incur tremendous costs for enterprises.

When Data Migration Goes Wrong

Recently, UK TSB were involved in a disastrous data migration in an effort to upgrade their systems. The technical failures that occurred during and after the migration saw millions of clients locked out of their online and mobile banking accounts. To make matters worse, former CEO of TSB Bank, Paul Pester, who stepped down due to the migration failure, admitted during a hearing in front of members of parliament that the migration failure resulted in 1,300 customers being defrauded. Overall, the migration botch cost the bank over $300 million AUD, and led to over 12,000 customers leaving the bank. The TSB incident shines a spotlight on just how necessary data migration validation best practices are.

Data Migration Validation Using Sampling

The most common process for validating data migration is based on sampling. This means that a random subset of the migrated data is chosen and subsequently inspected. It should be noted that although sampling is an effective method of migration validation, it is by definition based on an inherent level of accepted risk. This means that you cannot ensure that 100% of the data will be migrated without error. When using the sampling method, it is important for migration specialists to meet the statistical standards established for sampling plans and procedures that have been identified by ASQ/ANSI Quality Control Standards Z1.4 and Z1.9.

Automated Testing Tools

Alternatively, for data migrations that pose a significantly high level of business or compliance risk, automated testing tools can be used. These tools can be used on a subset of data or on 100% of the data, depending on what is required based on the risk level of the migration project. The tools themselves however, also need to be configured and tested. There are a few automated testing tools available on the market, and although they are less susceptible to human error, for most organisations’ requirements sample testing is all that is needed.

Crucial Stages in Migration Testing

Pre-Data Migration Testing

During the migration testing process, pre-data migration testing is vital for complex data structures, although it may not always be necessary for simple data structures. Firstly, the source data will need to be tested against the requirements of the destination system, using source-to-destination mappings. The destination system schema will then need to be inspected i.e. mandatory fields, min/max values, field length, field level validation etc.

Migration Testing

The first step in the migration testing stage will be to test the migration script(s) and make sure firewalls and ports are open, and hardware and software configurations are correct. Performance requirements have to be determined to meet migration timing (downtime) requirements. Supplementary systems are identified and verified to work with the destination system, and source system test cases will need to be converted for use with the destination system post migration.

Post Migration Testing

During the post migration testing phase, the number of records in the source and destination systems will first need to be compared. The source system data will then need to be checked to ensure it hasn’t changed during the migration. If any schema changes were required in the destination system this will also need to be validated. Reconciliation and NULL validation checks between the source and destination fields will also need to be undertaken for all fields. Finally, integration testing with the destination system and supplementary systems will be performed.

Final Thoughts

Ensuring that data migration validation best practices are followed is imperative for organisations undergoing data and content migrations. The damage that can be caused by a migration failure is potentially incalculable, as was highlighted in the incident involving UK Bank, TSB.

To ensure your data migration project is successful and doesn’t put your business’s information at risk, ensure you employ a data migration specialist with a proven track record in successfully migrating large amounts of data in complicated environments. Test their expertise and credentials to ensure they are following data validation best practices and have a deep technical understanding of the systems you are migrating to and from.

At Miktysh we have a solid, proven history in successfully migrating enterprise data since inception in 2008. Our vast experience spans numerous enterprise content management systems such as TRIM/Content Manager, SharePoint, OneDrive, RecordPoint, Dropbox, OpenText, Alfresco and many more. Our team of specialists always ensure data migration validation best practices are followed, using Xillio, which allows organisational content to be safely and efficiently migrated to the cloud, or between repositories. If you are considering a move to the cloud or to a new enterprise content management system, please get in touch with our team for a confidential discussion about your unique requirements.