Data Quality Rules (DQR) - Concepts

Learning Objectives

By the end of this lesson you will be able to:

• Explain what DQR is and why PDM uses a structured process rather than an ad hoc approach
• Describe the four components of the DQR process
• List the fix options available within DQR

Why DQR?

Data quality issues are inevitable in any data migration. The question is not whether they will appear, but how the project will manage them when they do.

Without a structured approach, data quality issues become the driver of the Responsibility Gap: IT cannot move forward, the business is overwhelmed with requests, the deadline approaches, and bad data gets migrated.

DQR exists to prevent that. It is a single point of ownership for all data quality and data preparation activity, structured so that the business leads AND the technical data stakeholders meet and prioritise together, with the project team providing the infrastructure for that to happen efficiently.

DQRs manage all issues, not just data quality ones. They are the PDM equivalent of backlogs in other agile methods. Any issue that affects the migration, whether it is a data quality defect, a mapping ambiguity, a missing business rule, or a process gap, is raised and tracked as a DQR. This is reinforced when you come to project planning and release management.

A note on the name. Out in the field you will rarely hear the term “DQR”. People talk about data issues, data quality issues, data defects, or whatever their issue tracker happens to call a ticket. PDM uses Data Quality Rules (DQR) deliberately: the name covers not just the problem but the structured process and repository for resolving it. PDM wants the term DQR to be part of the vocabulary of the project, heard all the time. When you join a project that calls them something else, you are usually looking at the same issues, without the PDM rigour around them.

DQR as a Super Smart Task

DQR is deliberately designed as a Super Smart Task:

Complete the task: DQR items are raised, analysed, prioritised, and resolved. The result is data that meets the agreed quality threshold for migration.

Build the individual: Business Domain Experts who participate in DQR meetings gain recognition for their knowledge and authority to make decisions. They become invested in the outcome.

Build the team: The DQR Board brings IT and business together regularly on shared problems. The relationship built through recurring DQR meetings is the foundation for the cooperation needed at the critical points of the project.

The Four Components of DQR

1. DQR Process

The sequence of activities:

1. Find the issue (through data profiling, mapping, testing, or LA)
2. Raise a DQR (create the form)
3. Quantify (how many records, what percentage, what impact)
4. Bring to the DQR Board
5. Prioritise (business-led)
6. Create sub-tasks (the work programme to resolve)
7. Manage to completion (track progress)
8. Close or escalate

2. DQR Board

The governing body that prioritises and manages the DQR process. Chaired by the Migration Analyst. Members:

• Data Owners (or their BDE deputies)
• Business Domain Experts
• Technical System Experts (legacy and target)
• Other KDSHs as relevant

The DQR Board meets regularly - weekly in high-intensity phases. It is the primary forum for IT-Business collaboration on data quality.

3. DQR Forms

Each DQR is documented on a standard form capturing:

• DQR ID and short name
• LDS and entity affected
• Raised by, date raised
• Priority (business and technical)
• Status
• Description (qualitative and quantitative)
• Method statement (how it will be fixed)
• Sub-tasks (the work programme)
• Metrics (how completion will be measured)

4. DQR Repository

The collection of all DQR forms, maintained under version control. The repository provides:

• A current status view of all data quality activity
• Reporting metrics for programme management
• The basis for migration readiness assessment (“are we ready to migrate?”)

Fix Options

When a DQR issue is analysed, the team must determine how it will be fixed. Options include:

The choice of fix is a business-led decision - the DQR Board approves the fix approach based on business risk, not technical preference.

Starting Early

A critical principle in DQR: start the process before there are significant issues.

The first DQR items raised should be straightforward - easy to resolve, easy to quantify. This establishes the process, builds familiarity with the forms, and creates the habit of DQR participation before the difficult issues arrive.

Projects that start DQR late (when the big issues have already emerged) find that the business is reluctant to engage with an unfamiliar process at a stressful moment. Projects that start early find that the DQR Board is a functioning, trusted forum by the time the hard decisions need to be made.

Key Takeaways

• DQR is a structured process for managing data quality issues - not an ad hoc response
• The four components are: DQR Process, DQR Board, DQR Forms, DQR Repository
• Fix options range from ignoring the issue to complex automated remediation - the DQR Board determines the approach
• Start early: the process must be established before the difficult issues arrive
• DQR is a Super Smart Task - it completes the remediation work while building the IT-Business team

Book Reference

Practical Data Migration by Johny Morris (BCS, The Chartered Institute for IT):

• Chapter 05 - Introducing DQR