Data lineage that survives an audit

Every data platform we inherit has a lineage diagram. It is usually beautiful, usually in a catalogue tool, and usually useless in an audit — because it describes the pipeline as it is configured right now, and an auditor is asking about a number that was reported nineteen months ago.

The distinction is worth being precise about. A catalogue answers how does this column get populated. An audit asks prove that this figure, on this date, was produced by the logic you documented, and show me who could have changed it. The first is a graph. The second is an append-only record of events.

What is actually asked for

Cambridge Mutual's regulatory reporting review ran for eleven weeks. Stripped of formality, the questions were these:

Four of those five demand a point-in-time answer. If your lineage is derived by parsing the code in your repository today, you cannot answer any of them, and you will spend three weeks reconstructing the past from Git history and Slack.

Emit lineage as run events, not as documentation

The design that survives contact with an auditor treats lineage as telemetry: every job run emits an event describing its inputs, its outputs, the exact code version, and the column-level mapping in force for that run. Events are appended, never edited. The graph in your catalogue becomes a view over the events rather than a separate artefact that drifts.

The OpenLineage schema is a reasonable default here, and being an open standard rather than a vendor format matters more than it sounds — your catalogue will be replaced within the retention period.

{
  "eventType": "COMPLETE",
  "eventTime": "2025-03-14T02:41:07.412Z",
  "run": {
    "runId": "0192d1c3-8f4a-7c21-9a55-2b6e5f0c7d31",
    "facets": {
      "parent": { "job": { "name": "reg_daily_close", "namespace": "prod" } },
      "sourceCodeVersion": {
        "type": "git",
        "url": "git@internal:reporting/close.git",
        "version": "e7b41c0d9a2f5b83c1e4d0a7f2913b6c8d5e4a91"
      }
    }
  },
  "job": { "namespace": "prod", "name": "reg_daily_close.exposure_agg" },
  "inputs": [
    {
      "namespace": "warehouse",
      "name": "risk.positions_eod",
      "facets": {
        "dataQualityMetrics": { "rowCount": 4182233, "nullCount": 0 },
        "version": { "datasetVersion": "2025-03-14T02:15:00Z#snap_7741" }
      }
    }
  ],
  "outputs": [
    {
      "namespace": "warehouse",
      "name": "reg.exposure_by_counterparty",
      "facets": {
        "columnLineage": {
          "fields": {
            "gross_exposure": {
              "inputFields": [
                { "namespace": "warehouse", "name": "risk.positions_eod", "field": "notional" },
                { "namespace": "warehouse", "name": "risk.positions_eod", "field": "fx_rate" }
              ],
              "transformationType": "AGGREGATION",
              "transformationDescription": "sum(notional * fx_rate) group by counterparty_id"
            }
          }
        }
      }
    }
  ]
}

That single object answers four of the five questions in the table above, in a form you can hand over without a human interpreting it. The fifth — who could have altered it — comes from snapshotting effective grants on the same cadence, which is a nightly query and a versioned table, not a governance programme.

Three details that decide whether it holds up

Column level or nothing. Table-level lineage tells an auditor that reg.exposure_by_counterparty derives from risk.positions_eod. They already assumed that. The finding lands on a specific column, so the evidence has to be about a specific column.

Capture failed runs. The most common real finding is not wrong logic, it is a partial load that was silently re-run with different inputs. If your lineage only records successes, the interesting event is the one you deleted.

Retain the events for as long as you retain the reports. Seven years, typically. That means the lineage store is a warehouse table with a partition key, not a feature of a SaaS catalogue on an annual contract. We keep the raw events in object storage as the system of record and let the catalogue index them.

The test

Pick a figure from a report filed at least a year ago. Give an engineer who did not build the pipeline two hours to produce the code version, the input dataset versions, the row counts, and the access list in force on that date. If they can, your lineage will survive an audit. If the answer involves reading a diagram and inferring, it will not — and you will find that out under a deadline set by someone else.