The Data Catalog
Four questions it must answer, and the honest failure mode: a catalog nobody populates is worse than no catalog, because it looks authoritative.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Which dataset contains customer revenue, who owns it, can I trust it, and how fresh is it — and where does someone go to find out?
A new analyst in week one, who will otherwise ask in a chat channel and be given whichever table the first responder happens to use. Also a privacy reviewer enumerating personal data, a producer checking who depends on them before a change, and an on-call engineer who needs an owner at 03:00 (Impact Analysis).
One catalog entry is one queryable dataset — the unit a consumer can actually select from. Not one file, not one partition, not one column: those are attributes hanging off the entry. Getting this wrong in the direction of files produces a catalog with hundreds of thousands of entries and no navigability; getting it wrong in the direction of schemas hides the column-level facts that governance needs (Data Classification).
Buy or deploy a catalog product, point it at the warehouse, let it crawl, and announce it to the company. The crawl succeeds, thousands of tables appear with real schemas and real row counts, and the launch demo is genuinely impressive.
Three weeks later the analyst searching for "revenue" gets forty results, none with a description, and picks the one with the most familiar name. It is a deprecated staging model. Nothing in the catalog said so, because "deprecated" is a human-written field (Data Discovery).
- Three weeks later the analyst searching for "revenue" gets forty results, none with a description, and picks the one with the most familiar name. It is a deprecated staging model. Nothing in the catalog said so, because "deprecated" is a human-written field (Data Discovery).
- The
ownercolumn is populated from whoever created the table in the warehouse, which is a service account belonging to the orchestrator. Every dataset appears to be owned byairflow_svcand the ownership field is now noise (Data Ownership). - A team asks whether they can drop a column and the catalog offers no downstream information, because nothing emitted lineage. The catalog knows what exists and nothing about how anything relates (Data Lineage).
- Someone finds a table with a beautiful description and builds a quarterly report on it. The description was written eighteen months ago; the model was rewritten twice since. The dashboard is wrong in a way nobody will discover for a quarter (Dataset Documentation).
- Adoption stalls, so registration is made mandatory. Now every dataset has a description, and the descriptions read "orders table" and "customer data", because a required field with no reviewer produces the shortest string that passes validation.
What is actually happening
- A catalog is not a database of documents. It is an index over three feeds: harvested schema, emitted operational facts, and a small authored layer — and its usefulness is almost entirely decided by how much of it comes from the first two (Metadata: Technical, Operational and Business).
- The four questions in the title are not interchangeable and are answered by different feeds. *What contains customer revenue* is a search problem over names, columns, descriptions and usage. *Who owns it* is authored. *Can I trust it* is operational — freshness, test results, incident history. *How fresh is it* is computed. A product that is strong at one and weak at the others will feel broken.
- The failure the guide asks to be taught honestly: an empty catalog is worse than no catalog. Not equal — worse. No catalog leaves a consumer knowing they must ask a human. A catalog with forty undescribed
revenuetables presents itself as the authority, invites a decision, and provides no signal about which entry is safe. It has converted "I do not know" into "I am confident and wrong". - The forces that empty a catalog are structural. Registration is a cost paid by the producer and a benefit received by a consumer they will never meet; the field a producer least wants to fill in is the one a consumer most needs; and nothing in the system ever tells the producer their description became false.
- The escape is to make the catalog earn attention rather than demand it. If it is the fastest way to find a table, see who else queries it and check whether it is fresh, people will use it — and the ones who use it are the ones most likely to fix an entry that is wrong (Data Discovery).
The four questions, and which feed answers each
A catalog is worth building when four specific questions are being asked repeatedly in a chat channel and answered inconsistently. Naming them precisely is what stops a catalog project turning into a metadata-modelling exercise.
They are: which dataset contains customer revenue; who owns it; can I trust it; how fresh is it. Every one of them has a different answer source, and a catalog that treats all four as "fields on a page" will be strong at whichever one its product happened to prioritise.
The important observation is that three of the four can be answered from harvested and emitted metadata. Only ownership requires a human — which is a strong argument for making ownership the one thing you insist on, and letting everything else arrive automatically (Data Ownership).
Which of the four questions is currently costing the most, and what actually answers it?
when Analysts repeatedly ask in chat, or build duplicate models because they could not find the existing one.
cost A search index over names, column names, descriptions and — critically — usage. Ranking by usage is what makes this work; ranking by text similarity alone reproduces the problem (Data Discovery).
when Incidents stall while someone works out whom to ask, or a producer ships a breaking change because they did not know who consumed them.
cost One mandatory authored field, plus a process for keeping it true through reorganisations. Cheapest to build, hardest to keep accurate (Data Ownership).
when Consumers cannot distinguish a production model from an abandoned experiment, and both look equally official in the warehouse.
cost Wiring test results, incident history and deprecation state onto the dataset page. Requires the quality layer to exist first (The Data Quality Dashboard).
when People are making decisions on data that stopped updating, and finding out days later.
cost A computed freshness per dataset against a declared expectation. Cheap if the pipelines emit; expensive if freshness has to be inferred by querying each table (Freshness Monitoring).
when One team, under a hundred datasets, everyone knows the tables.
cost None. A README and a naming convention are genuinely the better answer, and will keep being the better answer until a second team starts consuming.
Why an empty catalog is worse than none
This is the part usually left out of catalog advocacy, and it is the part that decides whether the project is worth starting. The failure mode of a catalog is not that it goes unused. It is that it gets used while being wrong.
Consider two analysts asked for last quarter's revenue by country. The first works somewhere with no catalog. She has no choice but to ask, and the person who answers names the model finance actually uses and mentions the caveat about refunds. The answer takes a day and is right.
The second works somewhere with a crawled, undescribed catalog. She searches "revenue", finds forty tables, and picks revenue_daily_v2 because the name is the most plausible and it has the most rows. It is a superseded model that double-counts refunds. Nothing lied to her; the catalog simply presented a list with no signal, and a list with no signal in an authoritative-looking interface reads as an endorsement.
The design consequence is that a catalog must be honest about ignorance. An entry with no description should say so loudly, deprecated datasets should be visibly demoted, and no field should render in a way that implies more confidence than its origin supports (Metadata: Technical, Operational and Business).
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| The harvest schedule is never set up after launch. | Entries describe tables that were dropped months ago; new tables are missing. | The crawl was run manually for the demo and treated as a one-off migration rather than as a pipeline. | Schedule the harvest, monitor it per source, and show harvest age on every page so decay is visible before it is total (Freshness Monitoring). |
| Registration is made mandatory to raise coverage. | Every dataset has a one-word description; coverage reports look excellent; nobody finds anything. | A required field with no reviewer optimises for the shortest string that passes validation. | Require one field that is checkable (team owner) and make the rest earn their way in through review on the transformation pull request (Dataset Documentation). |
| Owner is auto-filled from the warehouse creator. | Everything is owned by a service account; ownership queries return nothing useful. | A harvested value was used for a field that only a human can answer. | Leave it empty rather than filling it wrongly. An empty owner is an actionable gap; a service account is a closed ticket (Data Ownership). |
| Search ranks by name similarity only. | Deprecated and staging models outrank the curated model people should use. | Text relevance is the default and usage data was never wired in. | Rank by distinct recent readers and downstream dashboard count first, text second (Data Discovery). |
| A consumer builds on a beautifully described model whose logic changed. | A quarterly report is wrong; the description matches nothing in the current SQL. | The description lives in the catalog, the logic lives in the repository, and no change to one requires touching the other. | Move descriptions into the transformation repository so a logic change and a stale description appear in the same diff (Dataset Documentation). |
Crawl every source, publish every object, make description optional, and run a campaign asking teams to fill in the gaps. Coverage is reported as the success metric and reaches a high number quickly.
Crawl everything, but rank and render by evidence: datasets with an owner, recent usage, passing checks and a current description surface first; undescribed and unused objects are visibly marked as unverified and pushed down. Coverage is not the metric; the share of *queried* datasets that are trustworthy is.
Search results are read as recommendations. When an interface cannot distinguish a maintained production model from an abandoned experiment, the consumer supplies the distinction from the name — and names are the least reliable signal available. Ranking by evidence puts the burden of proof on the platform rather than on the reader's intuition.
Where the content comes from
A catalog page looks like one document and is assembled from at least four independent sources, each with a different owner and failure mode. Drawing that assembly is the fastest way to work out why a given field on a given page is wrong.
Notice which arrows are automatic. The harvester and the run emitters run without anyone deciding to; the query log arrives as a by-product of people doing their jobs; only the authoring path requires a person to choose to act, and it is the path carrying the field consumers most need (Metadata: Technical, Operational and Business).
Notice also what has no arrow at all: the shadow platform. Extracts in spreadsheets, a team's own database, a notebook that reads production directly. The catalog cannot see any of it, and its confident completeness is therefore always slightly overstated (Data Platform Anti-Patterns).
Whether a given catalog product ingests query history, emits or merely displays lineage, and supports column-level tags varies by product and by version, and these are exactly the capabilities vendors change most often. Verify current documentation rather than assuming from an architecture diagram.
How to build it
Most important first.
- Harvest first and launch on harvested content only. Schema, freshness, size, and usage are enough to be useful on day one and none of them can rot (Metadata: Technical, Operational and Business).
- Make usage a first-class, visible field: how many distinct people queried this in the last period, and which dashboards read it. It is harvested, it never lies, and it answers "can I trust it" better than most descriptions do (Data Discovery).
- Require exactly one authored field to publish — a team owner — and enforce it at the point where a dataset becomes visible to consumers, not by a retroactive campaign (Data Ownership).
- Render provenance on every field. A description shows who wrote it and when; a row count shows the harvest time. Users calibrate correctly when you tell them what they are looking at, and stop trusting everything equally.
- Mark deprecation loudly and support it mechanically: a deprecated dataset should be visibly demoted in search, and its remaining consumers should be enumerable from lineage rather than guessed at (Impact Analysis).
- Wire the catalog to the incident process. A dataset that failed its freshness or quality checks this morning should say so on its page — that single feature converts the catalog from a directory into something an engineer opens during an incident (Data Observability).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- A catalog guarantees that an object was observed to exist at harvest time with the described shape. It does not guarantee the object still exists, that you may read it, or that reading it will answer your question.
- It guarantees nothing at all about the authored fields, and the absence of a guarantee there is the single most important thing to communicate in the interface.
- Completeness is guaranteed only for sources that are actually connected. Every catalog has a shadow platform behind it — spreadsheets, notebook extracts, a team's own database — and the catalog cannot know it exists (Data Platform Anti-Patterns).
- Search recall is guaranteed only over indexed fields. A dataset whose description is empty is effectively invisible to any query that is not its exact name, which is why undescribed datasets are also undiscoverable ones.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Check adoption, not registration: distinct users who opened a dataset page in the last period, and the ratio of catalog searches to questions asked in the chat channel it was meant to replace. Registration counts measure compliance; adoption measures whether it works.
- Check the top-queried datasets specifically. A catalog where the hundred most-used tables all have owners and current descriptions is succeeding, even if the long tail is bare — and the reverse arrangement is a catalog that has optimised for a coverage metric.
- Both checks miss correctness of content entirely. A catalog can have full adoption, full coverage and uniformly wrong descriptions, and every number on this list will look excellent (Dataset Documentation).
- Catalog freshness is two numbers, not one: how recently the harvest ran, and how recently each authored field was reviewed. Averaging them into a single "last updated" is how a fresh crawl makes a stale description look current.
- The freshness shown *for a dataset* — its data freshness — must be computed rather than harvested. Reading the table's modification time is a trap: a job that runs and writes zero rows updates the modification time while the data goes stale (Freshness Monitoring).
- A catalog cannot be fresher than its slowest source connector. Publishing one platform-wide "last synced" hides the connector that has been failing for a week, which is the same mistake as publishing one platform-wide data freshness (The Freshness SLO).
- Renames are the hard case. If entries are keyed by qualified name, a rename destroys the entry's history and orphans its authored metadata; keying by a stable identifier the platform assigns is more work and is the only thing that survives a refactor (Schema Evolution).
- The catalog's own model will gain fields — classification, SLO, data product boundaries, agent-readable descriptions. Each addition is empty for everything already registered, so plan the backfill as part of the change rather than after it.
- As the platform adds source types — a lake, a streaming topic, a vector index — the notion of "dataset" stretches. A topic has a schema and no rows; a vector index has vectors and a source corpus. Forcing them into a table-shaped entry loses exactly the attributes that matter (Vector Data Engineering).
- Rebuild the harvested layer by re-crawling; it is a cache and should be operable as one, including the ability to wipe and re-run without ceremony.
- The authored layer has no upstream to recover from unless it is versioned outside the catalog. Keeping descriptions in the transformation repository turns catalog loss from a data-loss incident into a re-import (Dataset Documentation).
- A catalog that has drifted badly recovers faster by re-harvesting and deliberately clearing stale authored fields than by attempting to reconcile entry by entry. Displaying "unknown" is a recovery; displaying a wrong answer is not.
What can go wrong
- The launch-and-decay pattern: high engagement in the first month, no re-harvest schedule, and by the second quarter the catalog is a museum whose exhibits are labelled with the wrong dates.
- Mandatory registration without review, producing complete coverage of meaningless content and a dashboard that says the programme succeeded.
- Ownership auto-populated from the warehouse's creator field, which is the orchestrator's service account, making every dataset appear owned and none of them actually owned.
- A crawler with production-level access whose credentials are broader than any human's, because it needed to read every schema. The catalog is now the most privileged identity in the platform (Least Privilege).
- Search that ranks by name similarity only, so the newest, best-modelled table loses to a deprecated one with a shorter name (Data Discovery).
- "The catalog is a documentation project." It is a search and trust product whose content mostly comes from harvesters. Staffing it as a documentation project produces a wiki with a crawler attached (Metadata: Technical, Operational and Business).
- "Coverage is the metric." Coverage measures how many entries exist. Adoption measures whether anyone uses them, and it is entirely possible to have the first without the second — which is the state most catalog programmes are in when they are declared successful.
- "A catalog gives us lineage." Only if something emits or parses it. A catalog is a place to display lineage, not a source of it (Data Lineage).
- "If people would just fill it in, it would work." The people who most need to fill it in receive the least benefit from doing so. Any design that depends on that asymmetry resolving through goodwill has already failed once (Who Owns Data Quality).
- The catalog is where classification is applied and where it becomes enforceable. Tagging a column as personal data is only useful if masking and access policies read that tag rather than a separate list (Row and Column Security).
- A catalog with column-level statistics and sample values is a disclosure surface: sample values from a column of personal data are personal data, wherever they are displayed (PII in Pipelines).
- The crawler's identity is a governance problem in its own right. It needs metadata access across the platform and should be scoped to metadata — read schemas, not rows — which many deployments quietly skip in order to enable profiling (Least Privilege).
Operating it
- Weekly distinct users of the catalog, split by role. If producers use it and consumers do not, it has become a registration system rather than a discovery system.
- Search queries returning zero results, kept as a list. It is the cheapest possible backlog of what the platform is missing or has named unfindably.
- Per-source harvest success and lag, exposed on the source's own page rather than only in an internal dashboard.
- The share of queries in the warehouse that hit datasets with an owner and a current description. This is the number that says whether the catalog covers the part of the platform people actually use (Data Discovery).
- Below roughly a hundred datasets and a couple of teams, a maintained README and a naming convention outperform any catalog, because the coordination cost the catalog solves does not exist yet.
- Between hundreds and thousands, the catalog becomes load-bearing and search quality dominates. Nobody browses; ranking is the entire product (Data Discovery).
- Above that, the catalog needs its own governance: a model for what a dataset is, who may register one, what a published dataset promises. This is where the data-product framing starts to earn its keep (Data Products).
- Crawl cost scales with object count and with how deeply each object is profiled. Column-level profiling — distinct counts, null rates, top values — is a full scan per column and is the single most expensive optional feature a catalog can enable (Scan Cost).
- Storing usage history is cheap; querying it for ranking on every search is not, unless it is pre-aggregated. Discovery ranking is a small analytical workload sitting inside a documentation tool.
- The dominant cost remains human. Every required field multiplies by the dataset count, and the cost is paid by producers who receive none of the benefit — which is why the field list should be argued down rather than up.
- Harvest-only launches fast, is always true, and does not answer the question people came with. You get credibility and no depth, and depth only arrives through human effort you have to fund.
- Mandatory fields raise coverage and lower content quality, and the two move in opposite directions reliably enough to plan around. Pick the smallest mandatory set you can defend.
- A catalog that is deeply integrated — blocking deploys on registration, driving access policy — becomes far more valuable and becomes a production dependency with an availability requirement it was never designed for.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- ORG-SPECIFICThe catalog solves a coordination problem between teams, so its value tracks team count and dataset count rather than data volume. A petabyte platform run by four people who talk daily needs one far less than a terabyte platform spread across fifteen teams.
- TOOL-SPECIFICProducts differ on the axis that matters most here — how much they harvest without configuration versus how much they expect humans to enter. Evaluate a catalog by what its demo shows before anyone has typed anything, not by its field list.
- SCALE-SPECIFICSearch ranking is irrelevant below a few hundred datasets, where browsing works, and becomes the entire product above a few thousand, where nobody browses. Advice about catalog design that does not name a scale is advice about a different platform.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns the service-catalog equivalent of this — a registry of services with owners and on-call rotations. The two catalogs answer the same organisational question about different objects and are usually built twice, in isolation, by different teams.