Cloud Data Services
A data platform is assembled from about seven primitives. Every cloud sells all seven under different names, and the names are the least interesting part of the comparison.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Your platform runs on one provider and someone asks what the equivalent service is on another. What does "equivalent" have to mean before that question has an answer?
Two audiences with incompatible needs. An architect choosing a stack wants to know which capabilities exist and what each promises. A migration team wants to know which assumptions in the existing code stop holding — and that second question is never answered by a service-name mapping, because the code does not depend on names, it depends on behaviour.
One primitive: a capability a data platform cannot be built without, such as durable object storage or a replayable log. The unit is deliberately not "a service", because a single managed service often realises two primitives badly and one provider splits across two services what another bundles into one.
Keep a two-column table: "on the old cloud we used X, on the new one we use Y". It is genuinely useful for planning — it makes the shape of the migration visible in an afternoon, it lets a finance conversation happen, and for the primitives that really are close it is all you need.
The old platform published a table by writing files under a temporary prefix and then renaming the prefix. The new object store has the same "rename" in its SDK and it is a copy-then-delete loop rather than a metadata operation, so the publish is no longer near-atomic and readers start seeing half-published partitions (Atomic Publish).
- The old platform published a table by writing files under a temporary prefix and then renaming the prefix. The new object store has the same "rename" in its SDK and it is a copy-then-delete loop rather than a metadata operation, so the publish is no longer near-atomic and readers start seeing half-published partitions (Atomic Publish).
- A consumer relied on per-key ordering from the old event log. The mapped service orders per shard by the same partitioning rule, so the tests pass — until the shard count is changed to absorb growth and every key that moved shard loses order against its own history (Event Keys and Partition Assignment, CDC Ordering and Transaction Boundaries).
- The CDC connector on the new provider does not propagate DDL. The pipeline does not fail; it keeps streaming the columns it knew about, and the column added last Tuesday is simply absent from the warehouse (CDC and Schema Drift).
- Row-level security was expressed in the old warehouse. On the new stack the equivalent capability lives in the identity layer instead, so the policy does not migrate with the SQL and the table is briefly readable by everyone who can reach the dataset (Row and Column Security).
- The catalog on the old side was read by the query engine at plan time; the mapped product is a human discovery surface. Tables "migrate" and then no engine can find them, because the thing that resolved a name to a location was never in the mapping (The Data Catalog).
- Egress becomes a first-class term. Reads that were free inside one provider now cross a boundary, and a job whose shape nobody changed becomes the most expensive thing on the platform (Egress: Moving Data Costs Money, Not Just Storing It).
What is actually happening
- Underneath the branding, a data platform is assembled from a small set of primitives: durable object storage, an analytical query engine, a replayable log, managed distributed processing, an orchestrator, a catalog, and an identity and policy system. Every provider sells all of them. That is why the mapping table exists and why it is seductive.
- A mapping row is a claim about *category*, not about *behaviour*. Two services in the same category answer the same question — "where do durable bytes live?" — and answer it with different guarantees, and it is the guarantees your code was written against (The Data Loop).
- This is why
deCloudMapin this domain carries a mandatorydifferscolumn. The provider columns tell you what to procure. Thedifferscolumn tells you what to re-test, and it is the only column that changes what you build. - The differences that actually bite cluster into four kinds: atomicity (which multi-object operations are all-or-nothing), ordering (what the ordering unit is, if any), where policy lives (identity layer versus data layer), and what is managed (the cluster, the job, or the query). Each maps to a specific class of pipeline bug (The Pipeline Succeeded. The Data Is Wrong.).
- A managed service is also a shared-responsibility split, not an absence of operations. The provider takes the parts that fail loudly — hardware, patching, replication — and leaves you the parts that fail quietly: layout, schema, cost, and whether the data is right (Shared Responsibility).
A data platform is seven primitives wearing different names
Strip the branding off any cloud data platform and the same shapes are underneath. Something holds durable bytes. Something runs SQL over large scans. Something keeps an append-only, replayable stream. Something runs distributed jobs. Something schedules work by dependency. Something maps a name to a location and a schema. Something decides who may read what.
Every provider sells all seven, which is exactly why a mapping table looks so convincing: it is category-complete. It tells you nothing false. It also tells you nothing about the properties your existing code was written against, and those properties are the entire content of a migration.
The table below deliberately names no products. Each row is a primitive, what a data platform needs from it, and the one question that reliably exposes the difference between two providers' versions of it. If you can answer the third column for both sides, you have done the comparison. If you can only answer the second, you have a shopping list.
| Primitive | What a data platform needs from it | The question that exposes the difference |
|---|---|---|
| Durable object storage | Bytes under a key, listable, cheap at rest, readable by many engines | Which multi-object operations are atomic, and is a rename a metadata change or a copy? |
| Analytical query engine | SQL over large scans with a planner, governance and workload management | Is compute provisioned by you or allocated per query, and can one bad query degrade everyone else? |
| Replayable log | Durable, ordered-somewhere, re-readable from an arbitrary position | What is the ordering unit, and where does a consumer's position live — in the client or in the broker? |
| Managed distributed processing | Run a big job without operating the cluster underneath it | Is the managed unit the cluster or the job, and what happens to a running job when a node disappears? |
| Orchestration | Run tasks by dependency, retry them, and record what ran with what inputs | Is it a general task orchestrator, or a data-movement product with a scheduler attached? |
| Catalog and metadata | Resolve a name to a location, a schema and an owner | Is it read by engines at plan time, or by humans at discovery time? Very few do both well. |
| Identity and policy | Decide who may read which rows and which columns | Does a row filter or a column mask live in the identity system or inside the warehouse? |
Which concrete services occupy each row, and which of them a given provider currently bundles together, changes with every product cycle — services are renamed, split, merged and deprecated. deCloudMap in src/data/dataeng/catalog.ts holds the current per-provider mapping with its differs column; treat the service names there as a snapshot and verify current documentation before procuring anything.
The `differs` column is the only load-bearing one
Read a cloud mapping table left to right and it reads as a translation dictionary. Read it right to left — start at differs — and it reads as a list of the tests you have not written. The second reading is the useful one, because your pipeline does not call a service name, it depends on a behaviour.
Take the object-storage row. Both sides store durable bytes under a key; that is the category. The behaviour that a hundred pipelines depend on is subtler: whether writing a set of files and then making them visible in one step is possible, and if so, which operation gives you the step. Some stores let you swap a prefix cheaply; on others the same SDK call is a copy of every object followed by deletes, which is neither atomic nor fast. Nothing in the two-column mapping says this. The publish just stops being a publish.
The same reading applies everywhere. On the log row, the question is what the ordering unit is called and what happens to it under rescale. On the warehouse row, it is whether a heavy query steals capacity from an interactive one. On the identity row, it is whether a data-level policy is even expressible. Each of those is a specific test you can write in an afternoon, and each is a specific outage you can have instead.
Build a two-column table of old service to new service, hand it to the platform team, and let each pipeline owner swap connection strings and SDK calls until their DAG is green again. Cut over when everything is green.
Build a three-column table — primitive, service, and the behaviour this pipeline relies on — then write one executable check per behaviour: publish observed atomically by a concurrent reader, per-key order preserved across a partition-count change, an added upstream column arriving downstream, a restricted user actually being restricted. Cut over when the checks pass on the new side, and keep the checks in CI afterwards.
A green DAG proves the code ran against the new services; it says nothing about atomicity, ordering, policy or DDL propagation, because every one of those failures produces a successful run and different data. The assumptions are what the code was written against, so the assumptions are what has to be re-verified — and a behaviour that is not in an executable check is a behaviour nobody will re-verify at the next release either.
Whether a specific store's rename or prefix-swap is a metadata operation, and whether a listing immediately reflects a just-completed write, are per-service properties that have genuinely changed over the years — at least one major object store strengthened its consistency model after years of pipelines being designed around the weaker one. Verify current documentation for the exact store and API you are using rather than relying on what was true when a blog post was written.
Where equivalence breaks, concretely
The failures below are not exotic. Each one is a pipeline that worked, was moved to a service in the same category, and kept succeeding while producing different data. That combination — green and wrong — is what makes provider differences a data-engineering problem rather than an infrastructure one (The Pipeline Succeeded. The Data Is Wrong.).
Notice the shape of the response column. Almost every response is a check that could have been written before the migration, and almost none is a fix applied afterwards. That asymmetry is the argument for the conformance suite: it is cheap in advance and archaeology in arrears.
Notice also what is missing from the table: performance. Performance differences between providers are real, are workload-specific, and are the thing everybody argues about — but they announce themselves. A slow query gets attention. A publish that stopped being atomic does not (Benchmark Fallacies: Confident Numbers That Are Wrong).
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| A publish pattern that renamed a staging prefix is moved to a store where rename is copy-then-delete. | Readers intermittently see a partition with some of its files, and the numbers for that period are plausible but low. | The atomicity the pattern depended on was a property of one store's metadata layer, not of object storage in general. | Assert atomicity with a concurrent reader in the conformance suite, or move to a table format whose commit is a single pointer swap (Open Table Formats, Atomic Publish). |
| A consumer that assumed per-key ordering is moved to a shard-based log and the shard count is later increased. | A small number of entities end up with a stale "current state" — an old update applied after a newer one. | Ordering is guaranteed within the ordering unit; changing the number of units moves keys between them, and order is only defined per unit. | Test ordering under a rescale, not just at steady state, and make the merge idempotent and version-aware so out-of-order updates cannot regress state (Upserts and Merges, Event Keys and Partition Assignment). |
| A CDC connector is swapped for the same-category service on another provider. | New source columns never appear in the warehouse; no error is raised anywhere. | DDL propagation is a connector feature, not a CDC primitive. Some connectors surface schema changes, some ignore unknown columns. | Compare source and destination column sets on a schedule and alert on divergence — a schema-drift check, not a row check (CDC and Schema Drift, Schema Registry). |
| Row-level policies expressed in the old warehouse have no direct equivalent on the new stack. | The dataset is readable in full by everyone who can reach it, and nobody notices because reads succeed. | Where a data-level policy is expressed — identity layer or data layer — is a platform design choice, not a portable feature. | Include a negative access test in the conformance suite: a restricted principal must fail to read restricted rows. Absence of an error is not evidence of a policy (Row and Column Security). |
| A job that read from a bucket in the same provider now reads across a boundary. | The pipeline is unchanged and correct, and the platform cost profile changes shape entirely. | Byte movement across a boundary became a billed, latency-bearing operation instead of an internal one. | Instrument cross-boundary bytes per job before the migration and set an expectation, so the change is a number rather than a surprise (Cost Attribution, Egress: Moving Data Costs Money, Not Just Storing It). |
| The catalog is "migrated" to a product in the same category. | Engines cannot resolve tables; humans can browse a beautiful catalog of things nothing can read. | Technical schema registry and human discovery surface are two different products that share a word. | Establish which component resolves a name at plan time before moving anything, and keep engine-facing metadata separate from discovery metadata (The Data Catalog, Metadata: Technical, Operational and Business). |
Connector capabilities — which sources are supported, whether DDL is captured, what happens on restart — are the fastest-moving items in this entire domain and change several times a year. Nothing in this lesson should be read as a statement about what a specific connector does today; check its current documentation and, better, test it against your own source.
Deciding how portable to be
Portability is not a virtue with a fixed price. It is a spectrum, and each step along it buys optionality and gives up capability. The honest version of the decision names what you give up, because the managed features that are pleasant to use are precisely the ones that do not travel.
The one asymmetry worth knowing: data portability is much cheaper than platform portability. Keeping raw history in an open format on object storage costs you almost nothing extra and preserves the ability to rebuild elsewhere. Running two full platforms costs you a second on-call rotation forever and insures against an event most companies never experience (Keeping Raw History: The Recovery Position and the Liability).
There is no winner below. The criteria are the lesson: how likely is a move, what would it cost, and how much capability are you willing to leave unused in order to make it cheaper.
How much of your platform should be expressible on more than one provider?
when One cloud contract, no migration mandate, a small team, and capability matters more than optionality. Managed ingestion, the native catalog and the governance surface are all fair game.
cost A future move is a rewrite, not a migration. Every behaviour you depend on is one you did not choose deliberately, and the export path out of a proprietary storage layer is the one nobody has rehearsed.
when The common answer. Raw and curated data live in an open file format on object storage under a table format; compute, orchestration and governance are whatever the provider does best.
cost You give up some performance and some convenience that a proprietary storage layer buys, and you take on the operational work of compaction and table maintenance yourself (File Compaction).
when A genuine multi-provider requirement — regulatory, contractual, or a large acquisition — and enough platform engineering to maintain the abstraction.
cost You are limited to the intersection of what both providers do, which is always smaller and usually less pleasant than either. Abstractions over managed services leak exactly where the providers differ, which is where you needed them not to.
when Availability during a whole-provider outage is a stated, funded requirement — rare outside regulated or very large operations.
cost Two of everything: identity models, on-call rotations, cost models, quirks. Plus the hardest problem in the list — deciding which side is authoritative when they disagree (Source of Truth).
How to build it
Most important first.
- Write the mapping as three columns, not two: primitive, service, and the assumption your code makes about it. The third column is the migration plan; the first two are a shopping list (Data Architecture Patterns).
- Depend on the primitive at the boundary of your code. Reading and writing through a narrow storage interface, and expressing transformations in SQL a planner can execute anywhere, is what makes the third column short (SQL Transformations).
- Put every behavioural assumption into an executable check rather than a document: a test that asserts a publish is observed atomically, a test that asserts per-key ordering under a partition-count change, a test that asserts a new upstream column appears downstream (Contract Enforcement).
- Treat identity and policy as part of the dataset, not part of the environment. If a table's access rules are a page in a wiki, they do not move with it and they will be reconstructed from memory (Data Access Control).
- Decide the openness question deliberately and early: data in an open file format on object storage is portable in a way that data inside a proprietary storage layer is not, and that is a real property with a real cost in performance and convenience (Open Table Formats, The Lakehouse).
- Model the network. On a single provider, "which region is this in" is a performance detail; across providers it is the dominant cost and latency term of the whole design (What Actually Drives Data Platform Cost).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Every provider guarantees durability of stored objects, and takes it seriously. This is the one primitive where the mapping is close to honest.
- None of them guarantees cross-object atomicity. Publishing a dataset as many files is many independent events everywhere; the difference is only in which single-object operations you can lean on to fake a commit (Atomic Publish).
- Ordering is guaranteed only within whatever the provider calls its ordering unit, and the unit differs. There is no primitive anywhere that gives global ordering across a log without giving up throughput (Topics and Partitions).
- No managed service guarantees anything about your data's completeness or meaning. Managed availability and correct data are separate properties and only one of them is somebody else's job (Data Quality).
- Nothing guarantees that a policy expressed in one provider's vocabulary has an expressible equivalent in another's. Access models are the least portable part of a data platform (Data Access Control).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check that makes a mapping honest is a behavioural conformance suite: for each primitive, a small test that asserts the property your pipeline depends on — atomic publish observed by a concurrent reader, per-key order preserved across a partition-count change, a new upstream column surfacing downstream, a deleted row disappearing from the serving table.
- It misses anything about scale and anything about cost: a suite can pass on ten thousand rows and the same code can be unusable on the real volume, because listing behaviour, throttling and planner choices only appear under load (Load Test Shapes: The Shape Is the Hypothesis).
- It also misses semantics. Two providers can both preserve every byte and still deliver a differently-typed timestamp, and no conformance test you did not think to write will catch a field that is now UTC instead of local (Semantic Changes).
- Choosing a provider does not set freshness. The schedule, the ingestion mode and the transformation cadence set it; the provider decides only which of those are convenient to build (Batch vs Streaming Ingestion).
- Cross-provider hops add a real latency term and a real failure surface. A pipeline that reads from one cloud and writes to another has a stage whose availability is the product of two providers' availability, and it is the stage that will be blamed last.
- Managed services often hide their own internal latency behind an API that returns immediately. A write that is "accepted" is not always a write that a subsequent read will see, and any pipeline whose next stage lists what the previous stage wrote is depending on that (Ingestion Failure & Recovery).
- Managed services change under you, and unlike a library there is no version you can pin forever. A behaviour you depend on that was never in the contract is a behaviour that can be improved away.
- The stable parts are the primitives: object storage will still be a durable keyspace, a log will still be append-only and partitioned, a warehouse will still separate a planner from an executor. Anything narrower than that is a note to re-verify (Data Platform Engineering).
- Write down which provider behaviours your design depends on, next to the code that depends on them. This is the artefact that makes a future migration a project instead of an archaeology dig (Dataset Documentation).
- Provider-level recovery is about being able to rebuild. If raw data is retained in an open format and transformations are deterministic, a provider outage is a delay and a provider migration is a re-run (Keeping Raw History: The Recovery Position and the Liability, Reprocessing vs Retrying).
- If the only copy of a dataset lives inside a proprietary storage layer, recovery is bounded by that product's own export path, and export paths are slow, rate-limited and rarely rehearsed. Rehearse one before you need it (Validating a Backfill Before You Publish).
- A partially-completed migration is the dangerous state: two platforms both accepting writes, both feeding dashboards, and no single answer to "which one is authoritative". Name the authoritative side explicitly and put it in the catalog before the first table moves (Source of Truth).
What can go wrong
- A behavioural difference discovered in production because the mapping table implied equivalence and nobody wrote the conformance test.
- Rename-as-copy: an atomic-publish pattern that silently degrades into a non-atomic one on a store whose rename is not a metadata operation.
- An ordering assumption that survives the migration and dies at the next rescale.
- Policy loss: access rules that lived in a product-specific feature and were not expressible on the other side, so they were "temporarily" replaced with broader grants (Data Access Control).
- The mitigation failing too: a conformance suite written once, never run in CI, and now asserting the behaviour of a service version nobody uses (Data Tests).
- Cost inversion — a workload whose shape was tuned for one provider's cost model becomes the worst-shaped workload on the new one, because the driver changed from bytes scanned to hours held (Cost vs Freshness).
- "These two services are equivalent." They are in the same category. Equivalence is a claim about guarantees, and the mapping table is not making it — that is what the
differscolumn exists to say. - "Managed means we do not have to think about it." Managed removes the operations that fail loudly and leaves you the ones that fail quietly. Layout, schema, freshness and correctness never became somebody else's job (Data Observability).
- "We should be multi-cloud so we are not locked in." Lock-in is real, and running everything twice is a much larger and more certain cost than the risk it insures against. Portability of *data* — open formats, retained raw — buys most of the benefit at a fraction of the price (On-Premises vs Cloud).
- "The migration is done, the tables are there." Tables being present is the easy half. The half that fails is the behaviour the code assumed and the policy the wiki described (Data Governance).
- "Picking the cloud is the architecture decision." It is a procurement decision that constrains the architecture. Grain, freshness, contracts and layout decide far more about whether the platform works (Physical Data Layout, Grain: What Does One Row Represent?).
- Copying data to a second provider copies every obligation and none of the enforcement. Classification, retention and deletion have to be re-expressed in a different vocabulary, and "re-expressed" is where they get weaker (Data Classification, Data Retention).
- Residency is a provider-and-region property, and a managed service can move bytes between regions for its own reasons — replication, backup, a global control plane. If residency is a legal requirement, it is a question to ask about each specific service rather than about the provider (Data Governance).
- A deletion request must be satisfiable on every copy, including the one on the platform you migrated away from and never decommissioned (Deletion Requests).
Operating it
- A dependency inventory per pipeline: which primitives it touches and which behaviour of each it relies on. It is metadata, it belongs in the catalog, and it is the first thing anyone asks for during a migration (Metadata: Technical, Operational and Business).
- Cross-boundary byte movement per job. This is the metric that turns an architecture diagram into a cost conversation and it is almost never instrumented until it is a problem (Cost Attribution).
- Conformance-suite results per environment, dated. A green suite from eight months ago is a claim about a service that has since shipped forty releases.
- Per-service quota and throttling errors, which are the earliest signal that a design that worked on one provider is being shaped differently by another (Pipeline Metrics).
- At 10x, the primitives are unbothered and the differences start to show: listing a very large prefix, throttling on a hot partition, planner behaviour on a wide table. These are where providers stop looking alike.
- At 100x, the platform is shaped by whichever primitive is weakest for your workload, and that is rarely the one named in the architecture decision record.
- Consumer count scales the identity and policy problem faster than the technical one. Ten datasets and one team need no policy model; five hundred datasets and forty teams need one that is expressible in the platform they actually have (Data Governance).
- Bytes crossing a boundary — between regions, between providers, out to the internet — is the driver that surprises people, because it is invisible in the architecture diagram and proportional to traffic (Egress: Moving Data Costs Money, Not Just Storing It).
- Retained bytes accumulate on every provider and nothing deletes them by default. Lifecycle policy is a design decision, not a cleanup task (Storage Lifecycle).
- Request and listing volume dominates for datasets made of many small files, and that is a layout decision rather than a provider one (File Size and the Small-Files Problem).
- Idle managed capacity: anything provisioned by the hour bills for being available, so the cost driver is *hours held*, not work done (Compute Waste, Idle Capacity: Headroom or Waste?).
- Migration itself is a cost driver with no ongoing benefit — every byte read once, written once, and validated twice (Backfills).
- Depending only on portable primitives buys optionality and costs capability. The managed features that make a provider pleasant to use — a governance surface, an integrated catalog, a streaming ingest path — are exactly the ones that do not port.
- A conformance suite is real engineering time spent on a migration that may never happen. It is worth it in proportion to how much a wrong assumption would cost, which in the atomicity and ordering cases is a great deal (Atomic Publish).
- Running on two providers doubles the operational surface, the identity model and the on-call rotation for a benefit — availability during a provider outage — that most platforms never collect on (Multi-Cloud, Taught Cautiously).
Dataset review questions
This lesson uses the shared review exercise.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- CLOUD-SPECIFICEvery claim about a specific service belongs to that service and that release: atomicity of multi-object operations, listing consistency, whether DDL propagates through a CDC connector and where a row-level policy is expressed all differ between providers and have changed over time within a single provider.
- GENERALThe primitive set itself — durable object storage, an analytical engine, a replayable log, managed processing, orchestration, catalog, identity — is stable across providers and across decades of on-premise systems before them. What varies is packaging, not the list.
- ORG-SPECIFICWhether portability is worth paying for depends on procurement reality rather than engineering: a company with one cloud contract and no migration mandate is buying insurance against an event it has decided cannot happen, while a regulated multi-region business may have no choice.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns why cross-object atomicity is hard in the first place, and what a consistency model actually promises a reader who lists a prefix immediately after a write. This lesson takes those results as given.
- — DevOps / Production Engineering owns the migration mechanics: infrastructure as code across two providers, environment parity, cutover strategy and rollback. Here we only care which data assumptions have to survive the cutover.