Data Engineering Roadmap
Ten levels, each defined by what you can actually build once you have it rather than by what you have read. The order matters: every level assumes the failure modes of the one before it, and the last one is where a number becomes explainable.
The shape of the problem
You can explain why analytics moved off the production database, and answer an analytical question without taking anything down.
Where data lives
You can choose between a lake, a warehouse and a lakehouse for a real workload, and say what each one costs you.
Getting data in
You can build an ingestion path that survives a failure without losing or duplicating a day, and explain the predicate it extracts on.
Modelling for questions
You can design a model whose grain is declared, whose history is preserved where it needs to be, and whose measures do not multiply when joined.
Transformation and orchestration
You can express a platform as a tested dependency graph, and re-run any part of it without corrupting what is already correct.
Physical layout
You can make a query read a fraction of what it used to, and explain exactly why — the highest-leverage skill in analytics.
Change and streams
You can capture changes from a database log, publish them to a replayable stream, and reason correctly about event time and lateness.
Distributed compute
You can read an execution plan, find the shuffle, spot the skew, and know why adding workers will not help.
Trust
You can say whether a number is trustworthy and prove it — and find out that it is not before a finance team does.
Production data engineering
You can run a platform: recover it, govern it, pay for it, and explain any number on it back to its source.