Learning Objectives
By the end of this lesson you will be able to:
- Define data migration in the PDM sense of the term
- Explain why most data migration projects fail
- Describe the Responsibility Gap and how it forms
- Recognise the risk created by fixed-price procurement models
What Is Data Migration?
Before we can talk about what goes wrong, we need to agree on what data migration actually is. The PDM definition is deliberately precise:
Data Migration is the selection, preparation, extraction, transformation and permanent movement of appropriate data that are of the right quality, to the right place, at the right time, and the decommissioning of legacy data stores - to deliver the business transformation aspirations of the organisation.
Every word in that definition carries weight. Let’s unpack the key ones:
| Term | Why it matters |
|---|---|
| Selection | In modern environments there are often multiple potential data sources. PDM gives you a framework for choosing the right ones - balancing technical, business, and project constraints. |
| Preparation / Extraction / Transformation | Data quality is one of the biggest challenges in any migration. Even data that works perfectly in its current system may not work in the new one. PDM defines how transformation rules are generated with business relevance. |
| Permanent | PDM is for enterprise data migration - the data moves once, for good. This is not cyclical integration between reporting and transaction systems. |
| Right quality | In the tight timescales of modern projects you cannot make all your data perfect. PDM gives you the tools to prioritise what matters most. |
| Decommissioning | This is what separates migration from integration. The legacy data store ceases to support the migrated processes. Decommissioning can be logical as well as physical: the legacy system may still exist elsewhere, but it is no longer available to the business area affected by the transformation. There will be a point of no return, and PDM shows you how to use this to your advantage. |
| Business transformation | You only perform enterprise migrations for business reasons - there is always business transformation involved. The Golden Rules of PDM (lesson 3) are predicated on this truth. |
The Uncomfortable Truth About Failure Rates
Here is the number that tends to silence a room:
Between 40% and just over 50% of data migration projects run over time, run over budget, or fail entirely.
Those figures come from Bloor Research (2011) and the Data Migration Pro practitioner survey (2014). And if you are attempting a large-scale migration for the first time, in-house, without a proven methodology, the same research says your odds are significantly worse.
This is not a technology problem. The root causes are almost always organisational and methodological.
The 7 Factors That Determine Success
The industry research identified seven areas where projects consistently go wrong. Getting these right is what separates the successes from the failures:
-
Data Profiling - You need to know where you are coming from just as much as where you are going. Skipping profiling means discovering problems too late to fix them.
-
Data Quality Management - Poor tooling and a lack of overall management leads to bad data being loaded into production - or to data that simply fails to load at all.
-
Release & Version Management - Across the ETL chain, every party must be working to a controlled and documented version of the data requirements at the same time. When this breaks down, defects multiply.
-
Formal Methodology - The second most significant factor in successful migrations. An improvised approach means no shared language, no repeatable process, and no way to learn from what went wrong.
-
DM Project Independence - Keeping the data migration as a separate, independent track within the wider programme has a significant positive impact on outcomes.
-
Internal Competency - Data Migration Analysts need a specific mix of skills: business analysis, data analysis, technical execution, facilitation, project leadership, and training in a formal methodology.
-
Business Engagement - The single most significant factor. Treating data migration as a purely technical problem is the most reliable path to failure. In PDM, Data Owners are specifically the people who sign off the decommissioning decision, and the DQR team is chaired by a Data Migration Analyst.
The Responsibility Gap
The standard, techno-centric view of data migration looks like this:
- Extract data from source systems
- Transform it according to standard business rules
- Load it into the target
When data quality issues appear - and they always do - they go first to the technologists. The IT team investigates, but many issues cannot be resolved without business knowledge: what does this field actually mean? Which record is the master? What do we do with orphaned records?
These issues get passed to the business. But the business is rarely prepared for the volume of decisions they are being asked to make, on top of their day jobs, with little tooling or support.
The unresolved issues cycle back. The IT team waits. The deadline looms. Both sides grow frustrated. The business complains about the unreasonable volume of requests. IT complains about the lack of engagement from the business. Both sides believe the other should own the data.
This is the Responsibility Gap.
It is not caused by bad intentions on either side. It is caused by a badly constructed project that was never explicit, up front, about the IT-Business dependencies, and that gave neither side the tools to manage them.
PDM prevents the Responsibility Gap from forming by creating a single virtual team across the business and technical domains, not by drawing a clearer boundary between them. This is one of the fundamental points of difference between PDM and every other approach. PDM draws individuals into the team, builds the team, and then delivers the project. It does this through Super-SMART tasks that build the individual while simultaneously delivering the migration. How this works in practice is explained later in the course.
The Perverse Incentive to Fail
One more structural problem worth understanding - especially if you are working on the client side of a procurement.
Most large data migration projects procure supplier assistance under fixed-price contracts. The logic is sound: the client knows their budget up front. But there is a hidden trap.
At a fixed price, the supplier cannot absorb the unknown risk of data quality remediation work. The contract therefore places responsibility for data selection, enhancement, and quality on the client. The supplier then protects themselves with wait-time clauses - provisions that entitle them to additional payment if delays caused by the client hold up the work.
The result: the supplier has a financial incentive to benefit from data quality delays, not to resolve them.
Most suppliers act in good faith and go beyond a strict interpretation of their contract. But when a project gets difficult, the tendency is to retreat to a stricter reading. By that point, the relationship has deteriorated and the project is already in trouble.
Understanding this dynamic is the first step to structuring contracts - and projects - that don’t create it.
Key Takeaways
- The PDM definition of data migration includes decommissioning - there is a point of no return, and that is a feature, not a threat.
- Up to half of all data migration projects run over time, over budget, or fail entirely - and first-time, in-house projects fare worse. The causes are organisational, not technical.
- The Responsibility Gap forms when neither IT nor the business is equipped to own data quality decisions. It is a structural problem, not a people problem. PDM prevents it by creating a single virtual team across both domains.
- Fixed-price contracts can create a perverse incentive for suppliers to tolerate delays rather than resolve them.
Book Reference
Practical Data Migration by Johny Morris (BCS, The Chartered Institute for IT): Chapter 1, “Data Migration - What’s All the Fuss”.