Legacy data migration fails quietly. The cutover completes, the new system comes up, and the problem surfaces three weeks later when a reconciliation report does not balance or a customer disputes a figure that two systems now disagree about. By then the old system has been decommissioned or the transaction volume since cutover makes a clean rollback impossible.
The engineering work is not the hard part. Extracting, transforming and loading records is well-trodden. What decides the outcome is the discipline around it: knowing what correct looks like before you start, proving equivalence rather than assuming it, and designing a cutover you can reverse.
Key Takeaways
- Define reconciliation rules before writing any migration code. If you cannot state what correct looks like, you cannot verify the migration.
- Profile the source data first. Decades-old systems contain values that violate constraints the new system will enforce, and finding those at cutover is the worst possible time.
- Run the migration repeatedly against production copies. The first run tells you almost nothing; the tenth tells you the runtime and the failure modes.
- Parallel run beats a clean cutover for anything with financial or regulatory reporting attached.
- Keep the source system readable after cutover. Decommissioning early removes your only reference for disputes.
Profile the data before you design anything
Every long-lived system accumulates data that its own documentation says cannot exist. Dates before the company was founded. Customer records with a status code retired in 2011 that thirty rows still use. Amounts stored as text because a field was repurposed. Duplicate identities created when two branches merged and nobody deduplicated.
Profile the source before designing the target schema, not after. Count distinct values in every code field and check them against the reference list. Check nullability against what the new system will enforce. Check referential integrity, which older systems often maintained in application code rather than in the database, meaning it may not hold. Check ranges, formats and character encodings, particularly on anything holding Arabic text, where legacy encodings and mixed collations are a routine source of corruption.
Every anomaly needs a decision recorded before migration code is written: fix at source, transform in flight, migrate as is, or exclude. That decision list is the actual specification. Skipping it means those decisions get made under time pressure by whoever is running the cutover at two in the morning.
Decide what correct means, in writing
Reconciliation rule: a stated, testable equivalence between source and target, such as a control total, a record count by category, or a balance that must match to the currency unit.
Reconciliation cannot be designed after the fact. Before migration, agree the rules with the people who will be accountable for the numbers, which usually means finance and the business owners rather than the technology team.
Useful rules come in three layers. Record counts by entity and by meaningful category catch wholesale omissions. Control totals, meaning summed balances, outstanding amounts or quantities, catch transformation errors that counts miss. Sampled field-level comparison on a statistically meaningful subset catches the mapping mistakes the first two layers cannot see.
Write these as automated checks that run after every migration rehearsal, producing a pass or fail rather than a report someone has to interpret. A reconciliation that requires human judgment will not be run on the night it matters most.
Rehearse until the run is boring
The first migration run against a production copy exists to fail. It will reveal encoding problems, constraint violations, and a runtime estimate that is wrong by a factor you would not have guessed. That is the point.
Run it repeatedly on refreshed production copies until three things stabilise: the reconciliation passes cleanly, the runtime is predictable, and the failure handling has been exercised deliberately by breaking things on purpose. Teams that rehearse twice discover on cutover night that the run takes eleven hours rather than four, and that the window they negotiated is not long enough.
Rehearsals also settle the question of whether the migration can be restarted from a partial state or must run from clean. Restartability is worth building, because a failure four hours in should not mean starting over.
Parallel run, then cut over
For anything carrying financial or regulatory reporting, a period of parallel running is the strongest control available. Both systems process the same inputs, and their outputs are compared daily. Discrepancies are investigated as they appear, when the volume is small enough to trace.
The cost is real. Parallel running means dual data entry or a synchronisation mechanism, and it means people comparing outputs. Two to four cycles, meaning month-ends for most finance systems, is usually enough to build confidence. Longer than that and the organisation starts treating the comparison as routine and stops looking properly.
Where a full parallel run is not viable, a shadow run is the fallback: the new system processes production inputs and produces outputs that nobody acts on, purely for comparison. This works well when the new system can be fed from a change data capture stream, and it pairs naturally with a strangler fig migration, where slices cut over individually rather than all at once.
Design the cutover to be reversible
Reversibility is a design property, not a document. It has to be decided before cutover, because it constrains how the new system writes data during the first hours of live operation.
Three things make rollback possible. A verified point-in-time backup of the source with a tested restore, not just a taken backup. A defined and agreed rollback trigger, so the decision is made against criteria rather than by whoever is most tired. And a plan for reconciling transactions that occurred in the new system before the rollback decision, because those do not disappear when you revert.
Set the rollback deadline explicitly. After a certain point, forward fix is the only option, and everyone should know when that point passes rather than discovering it during an argument.
Keep the source readable
Decommissioning the source system immediately after cutover is a mistake that recurs constantly. Disputes, audits and regulatory queries arrive weeks or months later and reference data as it existed before migration. Without a readable source, those become unanswerable.
Keep the legacy system available in read-only mode, or take a full archived extract in a queryable form, for a period aligned with your audit and retention obligations. Under the UAE’s Personal Data Protection Law, Federal Decree-Law No. 45 of 2021, retention also has an upper bound, so the archive needs a defined disposal date rather than being kept indefinitely by default.
This connects to the wider programme covered in legacy system modernization without the big-bang risk, and to the business case, where the cost of retaining the source in parallel is a line item people routinely forget to include.
Frequently asked questions
How long should we allow for data migration in the plan?
For a mid-sized enterprise system, profiling and cleansing typically takes longer than building the migration itself. A reasonable planning assumption is that data work consumes a quarter to a third of the total programme effort. Projects that budget for it as a technical task at the end are the ones that slip.
Should we clean the data before or during migration?
Clean at source wherever the source will remain in use, because otherwise the same errors regenerate. Clean in flight where the source is being retired and the fix is deterministic. Never clean silently, and always keep a record of what was changed, because someone will ask why a figure differs.
Do we migrate all history?
Rarely. Migrate what the new system needs to operate and what regulation requires you to hold live, and archive the rest in a queryable form. Migrating everything increases risk, runtime and cost for history that is read a handful of times a year, if that.
What about data quality issues we cannot resolve?
Document them, get an explicit business decision on the treatment, and migrate with a marker on affected records so they can be identified later. The failure mode to avoid is a quiet default applied by migration code that nobody agreed to, which surfaces as an unexplainable figure a year later.
Can we migrate without downtime?
Often yes, using change data capture to keep the target current while the source stays live, then switching traffic once the lag is near zero. It costs more to build and needs careful handling of in-flight transactions at the switch point. Whether it is worth it depends on what an hour of downtime actually costs you, which is a question worth answering with a number before assuming the answer is zero.
Migrating decades of data?
Kentro handles profiling, reconciliation design, rehearsal and cutover for enterprise migrations across the UAE and GCC, including parallel run support through the first reporting cycles.
