Lineage Debugger

A number is wrong and you have to find out where. Start at the number and walk upstream one hop at a time, asking the same question at every hop — the first “no” is where the incident lives, and every hop you skip is a hop you will come back to.

Nobody has ever found a data incident by starting at the source. Starting upstream means checking dozens of things that are fine before reaching the one that is not, and it means you have no way to know when to stop. Starting at the number gives you a stopping rule: the first hop where the affected period is not complete.

Lineage is also the only tool that works in the other direction. Upstream answers where did this come from during an incident; downstream answers what breaks if I change this column before one. A platform with only the first has to discover its blast radius rather than know it.

The lab

Click a metric and walk back through the graph. At each node, decide whether the affected period is complete here before you move on.

Lineage debugger
Lineage debugger — walk upstream
A number on a dashboard is wrong. Move one hop at a time toward the source, asking the same question at every node: is the affected period complete here? The hop where the answer flips is the hop that broke it.
what is wrong upstream (normally you would not know this)
Dashboardcannot be asked here1 rows outsim

One number, with the grain now invisible.

↑ reads from
Serving tablenot looked at yet
↑ reads from
Transformationnot looked at yet
↑ reads from
Raw landingnot looked at yet
↑ reads from
Event lognot looked at yet
↑ reads from
Change capturenot looked at yet
↑ reads from
Source databasenot looked at yet
At the dashboard the grain is one number, so “is the period complete here?” has no answer — there is nothing to count. This is why an incident that starts on a dashboard always starts with a walk, and why a platform without lineage answers this question by asking six people.
hop 1/7SIMULATEDRow counts per hop come from the pipeline model for the selected fault.

The walk, as six questions

The same question in six places. Not a checklist to memorise — a discipline that stops you from jumping to the hop you already suspect.

  1. 1
    The tile itself

    Is the tile computing what its title claims, over the period its title claims?

    If yes
    The number is faithfully rendering a model. Go to the model.
    If no
    A filter, a date range or a definition was changed inside the BI tool, where no lineage graph can see it. You are done, and nothing upstream was ever wrong.
    Where Did This Number Come From? →
  2. 2
    The serving model

    Is the affected period complete and unique here — right row count, no duplicates on the business key?

    If yes
    The model faithfully aggregated its inputs. Go to its inputs.
    If no
    The break is in this model: a join that fans out, an incremental run that appended instead of replacing, or a filter that silently excludes.
    Duplicate Rows →
  3. 3
    The transformation that built it

    Did this run succeed, and did it publish — and is “succeeded” the same thing as “published the right rows”?

    If yes
    The logic ran over whatever it was given. Go to what it was given.
    If no
    A crashed or abandoned run leaves the previous output in place, and the dashboard renders yesterday with total confidence.
    The Pipeline Succeeded. The Data Is Wrong. →
  4. 4
    The upstream datasets it reads

    Is each input fresh against its own stated SLO, and complete for the affected period?

    If yes
    The inputs are all present. The break is semantic, not structural — go and compare definitions.
    If no
    The first input that is stale or short is your lead. Follow that one and stop following the others.
    Freshness Monitoring →
  5. 5
    Ingestion

    Did every partition of the affected period land, and does its row count reconcile against the source?

    If yes
    What arrived matches what was sent. Go to the source.
    If no
    A gap here is where the incident lives. Establish its bounds before you fix anything, because the fix usually destroys the evidence.
    Missing Rows →
  6. 6
    The source system

    Did the business event actually happen at the volume you expected?

    If yes
    The data is right and the pipeline is wrong. Everything above is now in scope again, with a much narrower question.
    If no
    Nothing is broken. Revenue really is down, and the platform did its job by telling you so — which is the outcome nobody rehearses for.
    Debugging a Data Incident →

Where to go next

Run the walk under pressure →

Revenue is down 40% and the evidence is contradictory. Diagnose before the answer is shown.

The same path, forwards →

What each hop promises the next, which is what makes the upstream question answerable.

The decision tree →

From the shape of the wrongness to the mechanism that produces it.