Data Quality Improvement: Detection Is the Easy Half

In December 2007, Queensland Health signed a $98 million contract to replace its payroll system. By the time acceptance testing finished, the team had logged over 2,400 defects. Queensland Health’s own rule said the system could not go live with a top-severity defect open. Dozens were reclassified downward instead.

It went live anyway.

Nearly 78,000 staff were paid wrong — overpaid, underpaid, or not paid at all — in errors eventually exceeding $300 million. The project cost $181 million. Repairing, maintaining and running the result came to roughly $1.2 billion over eight years. The commissioner who investigated it wrote that the failure “must take a place in the front rank of failures in public administration in this country. It may be the worst.”

Read that sequence again. The defects were found. Somebody wrote them down, and the process even carried a rule that the worst of them must stop a release. Detection worked. What failed was everything that was supposed to happen next.

The dimensions are a vocabulary, not a programme

Every data quality guide opens with the same list, and it is a good list:

• Accuracy — does the value match reality?
• Completeness — is anything required missing?
• Consistency — do two systems tell the same story?
• Timeliness — is it current enough to act on?
• Validity — does it conform to the rules it claims to follow?
• Uniqueness — is this record here twice?

These six give a team shared words for what “bad data” means, which is genuinely useful — arguments about data quality are otherwise unresolvable.

But notice what the list is. It is a taxonomy of problems. It contains no instruction about what to do when one is found, who is told, or who is allowed to say no. Adopting the vocabulary feels like progress and changes nothing on its own.

A check is worth writing when it has three things

Not one. Three.

A threshold someone agreed to. Not “monitor for nulls” — “customer_id is never null; if it is, the load has failed.” The number has to be written down before the incident, because after it there is always a reason it is fine this time.

An owner with a name. Not a team, a person or a rota. The most common fate of an unowned check is that it fires into a channel where forty people assume one of the other thirty-nine is looking.

A stated consequence. This is the one almost everybody skips, and it is the one Queensland skipped. What actually happens when this fails? The load halts? The dashboard shows stale-and-says-so? Someone is woken? Nothing is a legitimate answer — but then it is a dashboard metric, not a check, and it should not page anyone.

A check missing any of the three will eventually fire, be ignored, and teach the team that the checks can be ignored.

Put the consequence in the code

The useful move is to make severity a property of the test rather than a convention people remember. In dbt that is one line:

models:
  - name: orders
    columns:
      - name: customer_id
        tests:
          - not_null:
              config:
                severity: error        # halts the run
                meta:
                  owner: data-eng
      - name: currency
        tests:
          - accepted_values:
              values: ['EUR', 'USD', 'TRY']
              config:
                severity: warn         # logs, does not halt
                meta:
                  owner: finance

Two tests, two different consequences, both written down. A null customer_id stops the pipeline; an unexpected currency code files a warning for the team that owns currency. Nobody has to remember which is which at 3am, and nobody has to negotiate it during an incident.

The same discipline applies upstream of transformation, where a freshness or schema check belongs, and downstream in how metrics are defined — a number is only as trustworthy as the agreement about what it means.

Where to start

Three moves, in this order.

1. Write down the five numbers leadership actually looks at. Not a data dictionary — five rows. For each: what it means, which table it comes from, who owns it. Most quality work has no anchor until this exists, because “improve data quality” has no edges and “these five numbers must be right” does.

2. Put checks on the sources those five depend on, with all three properties. Threshold, owner, consequence. Start with the tables feeding money figures; they are where a silent error is most expensive and most likely to be noticed by the wrong person.

3. Decide in advance who can stop a release. Write the name down. This is an organisational decision, not a technical one, and it is the only part of data quality that cannot be bought.

Then expand. Automated profiling, anomaly detection, quality dashboards — all of it is useful, and all of it makes the first three matter more, not less.

What to be suspicious of

A data quality score. “We’re at 94%” invites exactly one question nobody can answer: 94% of what, and which 6% is wrong? Aggregate scores hide the thing you need, which is which specific check failed and what it gates.

If a number has to be reported upward, report the count of failing checks against the five anchor metrics. That one is actionable.

The takeaway

Detection is the half you can buy. Tools will find your nulls, your duplicates, your drifting schemas, and they will find them cheaply.

The other half is an agreement: what each check means, who owns it, and what happens when it fails. Queensland had the detection. It had over 2,400 defects in a document and a rule that said the worst of them must stop a release. What it did not have was anyone whose job it was to enforce it.

That is where Datablast puts the emphasis — quality checks that carry an owner and a consequence, defined with the pipeline rather than bolted alongside it, so a failing check does something instead of joining a list.

---

Originally published on datablast.io in February 2024. Rewritten and expanded September 2026.