Data Observability: Why Watching Your Data Isn't Enough

It is 9am. The freshness check is green. Row counts look normal. And the number on the dashboard has not moved since midnight, because the loader died six hours ago and nothing noticed.

Every check passed. That is the problem.

What data observability actually watches

Data observability is knowing the state of your data before someone downstream tells you about it. The standard framing is five pillars, and they are a good map:

• Freshness — when did this table last update, and is that within expectation?
• Volume — did roughly the expected number of rows arrive?
• Distribution — do the values still look like themselves? Null rates, ranges, cardinality.
• Schema — did a column change type, get renamed, or quietly disappear?
• Lineage — what feeds this table, and what breaks downstream if it does?

Every observability platform on the market watches some version of these. They are not wrong. They are just incomplete in one specific, expensive way.

All five watch the data. None watch the pipeline.

Read the list again. Freshness, volume, distribution, schema, lineage — each one inspects what arrived. Not one of them inspects the thing that was supposed to deliver it.

That distinction stays invisible until the night it matters:

• A source system goes quiet at 2am. The volume check expects few rows overnight and receives none. Pass.
• The loader crashes at midnight. The freshness check sees the newest row from the last successful run — and whether that counts as “stale” depends on a threshold someone tuned for a quiet night. Pass, often.
• Nothing looks anomalous, because nothing happened.

Silence looks exactly like success. This is the most common gap in real observability setups: nearly every team measures how old the newest row is, and far fewer prove that the job writing it is still alive.

The fix is a heartbeat, not a tighter threshold

The instinct is to tighten the freshness window. That trades one failure for another — now every genuinely quiet night pages someone.

The actual fix is to check the run, not the rows. Have every load write one row to a run log when it finishes, whether or not it moved any data:

-- "Ran, found nothing" and "did not run" are different facts.
select
  table_name,
  max(finished_at) as last_run,
  timestamp_diff(
    current_timestamp(), max(finished_at), minute
  ) as minutes_since,
  expected_interval_minutes
from pipeline_runs
where status = 'success'
group by table_name, expected_interval_minutes
having minutes_since > expected_interval_minutes * 2

The whole idea sits in that comment. A pipeline that runs hourly and finds nothing is healthy. A pipeline that has not run since midnight is an incident. Without a run log, both produce the same empty table and you cannot tell them apart.

Schema drift: check at the boundary, not downstream

The second quiet failure is upstream renaming a column. Nothing errors. The column arrives as NULL, or the new one is ignored, transformations run happily, dashboards keep rendering — and the number is simply wrong.

In BigQuery this is cheap to catch, because the schema is itself queryable. Snapshot it daily and diff:

-- Anything returned here is a change nobody announced.
select column_name, data_type
from `project.dataset.INFORMATION_SCHEMA.COLUMNS`
where table_name = 'orders'
except distinct
select column_name, data_type
from `project.dataset.schema_snapshots`
where table_name = 'orders'
  and snapshot_date = current_date() - 1

Run it on the tables you do not control, before the transformation layer. Everything computed downstream of a broken schema is wasted spend on a wrong answer.

Which check catches which failure

What goes wrong	Caught by	Silent under
Source sends half the usual rows	Volume	Freshness, Schema
Column renamed upstream	Schema	Freshness, Volume, Distribution
Amounts switch currency	Distribution	Freshness, Volume, Schema
Upstream table deprecated	Lineage	Everything, until something breaks
Loader dead while source is quiet	Run heartbeat	All five pillars

The bottom row is the one most setups are missing, and it is the cheapest to add.

Where to start

Not with ten strategies. With three, in this order.

1. Write the expectation before the check. “The orders table refreshes hourly; older than 90 minutes is an incident, and the data engineering team owns it.” A threshold with no stated expectation behind it is just a number someone will eventually mute.

2. Add the run heartbeat. One table, one query, one alert. It catches the failure class the five pillars structurally cannot see.

3. Put schema checks at the boundary. On the sources you do not own, before transformation. This is the same discipline that keeps a single source of truth trustworthy: guard the entrance, not the exit.

Then expand — distribution checks on the columns that feed money numbers, lineage once you have enough tables that “what breaks if this breaks” has stopped being obvious.

One way this goes wrong

Alerting on everything. A setup that pages someone at 3am over a 4% volume dip teaches the team to ignore the channel, and the real incident arrives to a muted room.

Give every alert an owner and a stated consequence. Checks with neither belong on a dashboard, not in someone’s night.

The takeaway

The five pillars are an accurate map of your data. They are not a map of your pipeline. Add one check that proves the job ran, and the most common silent failure in data engineering stops being silent.

This is why Datablast ships observability as part of the pipeline rather than as a layer bolted on afterwards. Run logs, freshness expectations, and schema checks at every source boundary come standard — so a silent upstream change breaks a check instead of a board meeting, and your ELT pipelines tell you when they are lying.

---

Originally published on datablast.io in August 2023. Rewritten and expanded September 2026.