:format(webp))
In this article we describe how data migration takes care of this. Data migration is not a technical trick, it is a functional migration that ensures that the data from the old system (or systems) is adapted to the new way of working and the way in which the new system supports the business products. This is the most important principle of a data migration: continuity is guaranteed, after migration the target system should work correctly.
Why is data migration complex and difficult?
Before we go into more detail about the set of principles that ensure a correct data migration, we first look at why data migration is difficult. As mentioned, data migration is part of the transition to a new system. This can be triggered by the need to modernize the application landscape. Rationalization and consolidation are also often the reason for replacing a system. We then see that several systems are replaced by one system. Data from different systems has to be merged during migration. We often encounter this situation when two organizations are integrated after a merger or acquisition.
Another reason for data migration is the outsourcing of business processes, for example the transfer of back-office activities to a service provider, like a bank that outsources management of mortgage loans. The data from your own system has to be transferred to the service provider and fit into the system of this provider. It must not only fit, the right selections should be made. Which loans are concerned? What are the associated securities? Unbundling is now part of the data migration.
It has already been mentioned that the new system often supports products and processes differently. The service provider that handles mortgage loans slightly different, or the transition from a classic registration system to a case-oriented approach. Often we want to modernize, but we run into hidden functionality, for example fields that are used differently than intended. Or there is simply data pollution, a problem that every organization has to deal with. Difficult for the operation, blocking for data migration.
The data migration must go smoothly, without errors. In various business sectors, both internal and external supervisors will be monitoring. This is the case in finance, healthcare, government and various industrial sectors. One should demonstrate that the data migration is 100% complete and correct. And when it comes to privacy-sensitive personal data, requirements from the AVG/GDPR must be taken into account. During the data migration, production data is converted and this should not lead to a data leak.
Data migration is much more complex than it seems at first glance. In most organizations it is something that only happens occasionally, too little to gain much experience. Therefore it is wrong to consider data migration simply the finalizing step when implementing a new system. It is a main task or even separate project that needs early attention from IT and even more so from the business. We use a set of principles to ensure that the data migration is successful.
Data migration principles
The target system is leading. Only data needed to get the target system up and running needs to be migrated. So start from the target. A data migration is, as it were, from target to source. The target determines which data is needed from the source. And what about the rest? If there is a legal retention requirement, or data must remain retrievable, migrate it to a solution for historical data management.
Migrate 100% correct data. Working with incorrect data should be avoided. It frustrates users. It leads to customer complaints. Low data quality is a high business risk. Use the data migration to put the data quality in order. In many cases the target system will reject bad data and cleansing is necessary. In practice, data quality is always disappointing and resolving this takes time. So start data cleansing as early as possible.
All actions are automated. Manual actions during a migration are a source of errors. Manual actions also make the checks and audit trail even more difficult. A good migration is repeatable. A successful trial migration results in a successful production migration. We automate both the data conversion and all steps around it. Does every exception have to be resolved in software? This is not necessary. Simply put the exceptions on an approved list that is processed automatically.
Checks and audit trail are full proof. We prove completeness and correctness of the data migration. We do this on different levels. Everything is checked within the migration solution, numbers and sums. We use hash totals to ensure that nothing is swapped or changed. Subsequently, there are acceptance and accountability checks that prove functional correctness over the entire chain.
Testing is integrated. Each test and trial migration demonstrates that migration solution is without errors. Our checks prove it. By realizing checks independent of conversion rules, we prevent an incorrect interpretation of a functional conversion requirement from going unnoticed.
Use an agile approach. Often, starting the migration project, the target system is still under development. And one thing is certain: insights do change. The migration solution has to be adapted quickly. After each sprint, a trial migration shows that more and more data can be migrated successfully. During the final trial migration, 100% is migrated without errors. In our migration solution we apply more agile concepts, such as continuous integration. We will discuss this in more detail in one of our next articles.
In short, data migrations are complex. It is necessary to pay real attention to data migration when implementing a new system. That means starting early and not losing sight of the basic principles for data migration. Only then the new system will start with correct data, and continuity is guaranteed.
This is the first in a series of articles on data migration. In our next article, we'll look at choosing the right migration strategy: a big bang or dripping data? For more information about the Ordina Solution for Data Migration, please contact Henk Zwaan, management consultant at the Data Engineering & Data Migration practice: henk.zwaan@ordina.nl.
A Dutch version of this blog is published at LinkedIn.