Making data lineage legible

A data team's warehouse is a web of dependencies. A raw events table feeds a dbt model, which feeds another model, which feeds a Looker dashboard the CFO reads every Monday and a Hightouch sync pushing segments into the marketing tool. Change one column at the top — rename it, recalculate it, drop it — and the blast radius runs through everything below.

Lineage is the map of that web. Datafold builds it by parsing every SQL statement that hits the warehouse, down to the column, across source data, dbt models, and the BI and reverse-ETL tools downstream. That parsing was engineering's hard-won achievement. The jobs the map has to serve are concrete:

Engineering had built something that technically worked. It listed every table, every column, and every connection between them — a complete map of the warehouse. What it didn't do was let anyone use it.

Performance was the first wall. The backend returned an enormous payload — everything at once — and the view could take several minutes to render. Once it was up, any interaction (a pan, an expand, a filter change) triggered another long wait. In practice the tool was unusable for the day-to-day question — "if I change this column, what breaks?" — because the answer arrived too slowly to be part of a code review.

The second problem was signal. Not everything on the map mattered equally to the person looking at it. To make lineage usable, we had to decide what to not show by default — how to limit what's on screen so the view renders fast, is navigable, and reads at a glance: where to move, what to open, what to ignore. The design work started there.

The redesign ran through the classic NN/g design thinking arc — used less as a linear checklist and more as a way to keep asking which phase needed more evidence next.

Empathize. The company had no product analytics wired to lineage, so I couldn't tell what was working from a dashboard. I watched Datadog session replays to see how people actually moved through the graph — where they zoomed, where they scrolled off-canvas and never came back, which filters they touched, and which they stared at without touching. I paired the replays with shadowing data engineers on impact-analysis walks. That was the ground truth.

Define. One pattern sat under everything: the moment the graph got bigger than a screen, people stopped navigating and started giving up. That named the crux — never render the hairball; reveal the graph one deliberate step at a time. Every subsequent decision had to earn its place against that one line.

Ideate. The redesign moved on three fronts in parallel:

Prototype. The card became a node system before it became a shape — collapsed and expanded states across every asset type. The graph mechanics settled into max-depth plus click-to-expand with the edge counters. The bridges started with a Diff → Lineage jump-in from any changed column.

Test. Some concepts made it to a validation round and didn't survive. Grouping was the biggest: bundling schema-mates or dozens of downstream syncs into a single group with in-place search and sort. On paper it beat starburst. In practice, when we walked users through it, the mental model didn't hold — nobody could predict what was inside a group without expanding it, and the ceremony of grouping got in the way of the actual question. We killed it and doubled down on search instead.

Implement. Table-level and column-level both shipped; the filter panel and run/popularity enrichments followed after my time.

The shipped view keeps the graph collapsed, the card informative, and every path — from a column, from a search, from a Diff — one click away.

At the level of a single card, this is what a user gets in one glance: the data source (Snowflake, Hightouch, Looker, Tableau), the fully-qualified path with the table name emphasized and the database.schema. prefix dimmed, the asset type, a column count with an expand affordance, and a small external-link icon that opens the asset in its native tool. Run (freshness) and popularity (a four-bar meter of downstream demand) were designed alongside these and shipped after my time.

Expand a card and the graph switches to column-level: each field is typed, and the edges redraw column-to-column so you see exactly which field feeds which.

There was no formal metrics infrastructure on lineage when I redesigned it, so I couldn't quote a hard A/B result. What I could do was watch: Datadog session replays made it obvious how the shipped surface was being used differently. The order-of-magnitude signal was a ~40% lift in lineage usage in the months after the redesign, alongside consistent qualitative feedback that the graph was finally usable inside a code review.

The stronger signal is what the surface became internally: the view the lineage value prop is now sold on. Datafold's public writing about lineage — impact analysis, PII tracing, upstream discovery, incident debugging — walks through the shipped UI. The Data Diff → Lineage bridge became a headline pairing in the product story.