How to Estimate Data Migration Complexity
Whether through corporate mergers, data interchange or new system deployment, developers seem to spend quite a bit of time moving data. While the task often seems trivial and the job often goes to the least experienced team member, this task is critical to the success of any project. Data migration often hides traps that are seldom anticipated and consumes far more time than the schedule allows. Here are a few factors to consider when measuring the complexity of your next migration project.
Other People Are Reading
Instructions
-
-
1
Measure the size of the current task. Count the number of tables and data elements in both the old and new databases. Also, consider the total number of records to be moved. Compare this to previous migrations to estimate the overall length of the project.
-
2
Analyze the amount of translation necessary. If most elements contain names, addresses, part numbers or similar data, little translation may be needed. However, if many fields hold codes that must be translated to new values, this will take more time. Also consider the interrelation of data elements and the complexity of each translation. If several fields from different tables must be joined to derive a new field, this will take far longer than a field retrieved from a simple look-up table.
-
-
3
Assess the overall quality of the data. Older, legacy data often requires filtering to remove inconsistencies and cleaning to fix business rule violations. Allow time to validate and clean data now instead of tuning and repairing problems in the new system.
-
4
Review the history of the data. Talk with long time developers and users to determine the number of previous conversions. Eliminate tables and fields that are no longer used and adjust translation rules for fields that have changed usage over time. The more volatile the data, the more complex the migration.
-
5
Look for hidden programs or interfaces that rely on the old data structures. Even the best analysts miss processes that were never documented or have been long forgotten. Database triggers may send data to other databases or someone in another business unit may periodically extract data for a combined report.
-
6
Allow time for validation and testing. Create unit tests to verify translation processes and develop comparison reports to ensure that data gets migrated correctly.
-
7
Anticipate risks. After assessing the overall complexity of the project, focus on the critical tasks. Factor in work schedules, other projects, bandwidth restrictions, storage needs and other issues. No project exists in isolation and few developers have the luxury of working on only one project at a time.
-
1
