Managed Streaming Platforms
Managed Kafka, Kinesis, Pub/Sub and Event Hubs are all realisations of the same primitive — a durable, replayable log. The axis on which they genuinely differ, and the one that changes your design, is ordering.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Four managed services all give you a durable stream that many consumers can read independently. Which of their differences will actually force you to change your pipeline?
Everything downstream of the log: a stream processor maintaining state per key, a sink writing to a warehouse, a service reacting to events, a backfill job replaying six months. What they need is not "messages" but a position they can return to and an ordering they can reason about — and those two things are exactly what the four services define differently (Retention and Replay).
One record on one ordering unit. The ordering unit is called a partition, a shard, or nothing at all depending on the product, and it — not the topic and not the message — is the grain that matters, because every ordering, scaling and replay property in this lesson is defined per unit (Topics and Partitions).
Pick whichever streaming service your cloud provides, produce events to it with a sensible key, and consume them. This is right far more often than architecture discussions suggest: all four are durable, all four fan out to independent consumers, and for a pipeline that is genuinely order-insensitive the differences never surface.
The pipeline maintains "current state per order" by applying updates in arrival order. It was built on a per-partition-ordered log and moved to a service with no ordering guarantee unless an ordering key is set — which nobody set — so a SHIPPED event occasionally lands before the PAID that preceded it (CDC Ordering and Transaction Boundaries).
- The pipeline maintains "current state per order" by applying updates in arrival order. It was built on a per-partition-ordered log and moved to a service with no ordering guarantee unless an ordering key is set — which nobody set — so a
SHIPPEDevent occasionally lands before thePAIDthat preceded it (CDC Ordering and Transaction Boundaries). - Traffic grows and the partition or shard count is increased. The key-to-unit mapping changes, so records for a key now land on a different unit than their own history did, and for a window there is no ordering between a key's old and new records (Event Keys and Partition Assignment).
- A consumer falls behind over a long weekend and its position is past the retention horizon. The records are not late; they are gone, and the only recovery is from whatever else retained them (The Raw Landing Zone).
- A backfill needs to replay from three months ago. The chosen service tracks consumer position as per-message acknowledgement rather than as a rewindable cursor, so "start again from a point in time" is a subscription-level administrative operation with different semantics than seeking an offset — and the runbook assumed the offset model (Replay from the Log).
- Retries produce duplicates and the sink is not idempotent, so a warehouse table gains extra rows on every consumer restart. Every one of these services delivers at least once by default; deduplication was always the consumer's job (Deduplication).
- A single very hot key concentrates most of the traffic on one unit. Ordering is preserved perfectly and one consumer is saturated while the rest idle, because the ordering unit is also the parallelism unit (Hot Keys: When Aggregate Metrics Hide a Saturated Node, Data Skew).
What is actually happening
- The primitive is a durable, append-only, replayable log with independent readers. Records are appended, retained for a while, and read by any number of consumers each holding their own position. Everything else in this lesson is a variation on that (The Event Log, Kafka as a Log, Not a Queue).
- The log is split into ordering units so that throughput can exceed one machine. Order is defined within a unit and undefined across units — that is not an implementation detail, it is the price of horizontal scale, and it is identical in every one of these products that has units at all (Topics and Partitions). A record is assigned to a unit by hashing a key. Same key, same unit, therefore same order — as long as the number of units does not change. Changing the unit count changes the mapping, which is why rescaling is an ordering event and not merely a capacity event (Consistent Hashing).
- The Kafka-family model exposes the unit and gives the consumer an offset: a durable, addressable position it can commit and seek. Because the position is a coordinate on the log rather than a state of the broker, rewinding is a normal operation and replay is a first-class feature (Offsets and Commits). The shard model is the same idea with different vocabulary: shards instead of partitions, sequence numbers instead of offsets, and resharding by splitting and merging rather than by changing a partition count. Ordering is per shard, and a split or merge moves keys between shards.
- The subscription-and-acknowledgement model hides the unit entirely. The service tracks, per subscription, which messages have been acknowledged, and redelivers the ones that have not. That buys effortless scaling — there is no unit count to choose — and it means ordering is not offered by default; where it is available it is per explicitly-set ordering key, which reintroduces the same throughput ceiling a partition has (Message Brokers: Log-Shaped and Queue-Shaped).
- Delivery is at-least-once in all four unless you build more. Stronger claims exist in some products, and any exactly-once claim has to name whether it covers input consumption, state update or output write, and which assumption — a transactional sink, an idempotent write, deterministic replay — buys it (Exactly-Once: Input Consumption, State Update, Output Write).
- Retention is a time or size horizon, and it is what makes the log a data-engineering asset rather than a queue: replayability for the retention window is the feature the whole domain leans on (Retention and Replay).
One primitive, three ordering models
All four services deliver the same primitive: a durable append-only log, retained for a window, readable independently by many consumers. If that were the whole story they would be interchangeable and this lesson would be a paragraph. The differences that matter cluster in two places — what the ordering unit is and where the consumer's position lives — and both of them are load-bearing in a data pipeline (Kafka as a Log, Not a Queue).
The Kafka family exposes partitions and hands the consumer an offset it commits and can seek. The shard model is the same shape with different words — shards, sequence numbers, and resharding by split and merge instead of a partition count. The subscription model hides units entirely, tracks acknowledgements per subscription, and offers no ordering unless you attach an ordering key to a message, at which point that key is serialised.
Notice what the third model trades. Removing the visible unit removes the capacity decision, the rescale event and the hot-partition failure — genuinely valuable, and the reason it is pleasant to operate. It also removes the default ordering guarantee, and a pipeline built on per-key ordering has to opt back in per key, accepting the same per-key throughput ceiling a partition would have imposed. There is no model here that gives ordering for free; there are models that make the price explicit and one that makes it optional.
| Concern | Partitioned log (Kafka family, Event Hubs) | Shard-based (Kinesis-style) | Ack-based subscriptions (Pub/Sub-style) |
|---|---|---|---|
| Ordering unit | Partition. Order is total within it, undefined across partitions. | Shard. Order is total within it, undefined across shards. | None by default. With an ordering key set, order is per ordering key. |
| Key to unit | Hash of the record key over the partition count. | Hash of the partition key over the shard key ranges. | Not exposed. The ordering key, if any, is what serialises. |
| Consumer position | An offset the consumer commits — a coordinate on the log it can seek to. | A sequence-number checkpoint the consumer stores. | Per-message acknowledgement tracked by the service per subscription. |
| Rewinding | Commit an earlier offset. An ordinary, per-consumer-group operation. | Restart from an earlier sequence number or timestamp. | A subscription-level seek to a timestamp or snapshot, with its own redelivery semantics. |
| Scaling unit | Partition count. Increasing it changes the key mapping. | Shards, changed by splitting and merging ranges — which moves keys. | Invisible. The service scales without a decision from you. |
| What rescaling does to order | A key can move partition, so it has no order against its own history for a window. | Same: a split or merge moves keys between shards. | Nothing, because there was no cross-key ordering to lose. |
| Delivery floor | At least once. Stronger patterns exist and must be scoped to consumption, state or output. | At least once. | At least once. |
| Retention | Time or size horizon, configured per topic. | Time horizon, configured per stream. | Time horizon per subscription, plus snapshots. |
Which managed offering exists on which provider, whether a Kafka-protocol-compatible endpoint is available, how throughput is provisioned or auto-scaled, and the maximum retention configurable on each are product facts that change and differ by region and tier. This lesson states none of them; verify current documentation before designing around a limit or an availability claim.
Ordering is the axis that changes the design
The reason ordering is the load-bearing difference and throughput is not: throughput problems announce themselves. A saturated consumer has lag, an under-provisioned stream throttles, and somebody gets paged. An ordering violation produces no error at all. A SHIPPED applied before the PAID that preceded it leaves one order in a wrong state, no exception is raised, and the discovery happens when a customer complains (The Pipeline Succeeded. The Data Is Wrong.).
The failures below are all versions of the same mistake: taking a guarantee that is defined per unit and treating it as though it were defined per key or per topic. Read the response column and notice that the robust answers do not depend on the broker at all. Version-stamping records at the producer and refusing to apply an older version downstream makes a consumer correct under any of the three ordering models, which is why it is the recommendation regardless of product (CDC Ordering and Transaction Boundaries).
The exception worth naming is the hot key. Ordering unit and parallelism unit are the same thing in every one of these products, so per-key ordering has a per-key throughput ceiling by construction. No configuration removes it — the only remedies are to shrink what must be ordered, or to accept disorder and reconcile downstream (Hot Keys: When Aggregate Metrics Hide a Saturated Node).
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Partition or shard count increased to absorb growth. | A small number of entities end up in a stale state; everything else is fine and nothing errors. | The key-to-unit mapping changed, so a key's new records have no ordering relationship to its old ones. | Version-stamp per key at the producer and make the sink refuse to apply an older version. Treat rescaling as a contract event and announce it (Upserts and Merges). |
| Pipeline moved to a service with no default ordering. | Intermittent, unreproducible state corruption on a small fraction of entities. | The design assumed per-key ordering that the new product only provides when an ordering key is set. | Set the ordering key explicitly and accept its per-key throughput ceiling, or make the consumer order-independent through versioning (Event Keys and Partition Assignment). |
| One key carries most of the traffic. | One consumer is saturated, the rest are idle, and lag on that unit grows without bound. | The ordering unit is also the parallelism unit, so a dominant key pins throughput to one consumer. | Reduce what must be ordered — order per sub-entity rather than per tenant — or split the key and reconcile downstream (Salting a Skewed Key, Data Skew). |
| Consumer down over a long weekend. | On restart the consumer's position is before the oldest retained record. | Retention expired. The records are gone rather than late, and the log has nothing to give. | Alert on distance-to-retention-cliff rather than on lag, and land raw records in object storage so slow replay is always possible (The Raw Landing Zone). |
| Two producers stamp records with their own wall clocks. | "Newest wins" downstream picks the wrong record for entities touched by both. | Timestamps from different machines are not a total order, and the merge assumed they were. | Use a monotonic per-key version from a single authority — a source database sequence or log position — rather than a clock (Processing Time, What a CDC Event Contains). |
| Consumer group rebalances repeatedly during a deploy. | Duplicate rows appear in the sink on every release. | At-least-once delivery plus a rebalance means records processed but not yet committed are redelivered. | Make the sink idempotent on a producer-supplied key. This is required everywhere and is not a property any of these products provides for you (Idempotent Data Pipelines). |
Where the position lives decides whether you can replay
The second difference is quieter and matters most during an incident. In an offset model, a consumer's position is a coordinate on the log: it is durable, it is inspectable, and moving it backwards is an ordinary operation. That is what makes replay a first-class data-engineering tool — reprocessing three months of history is committing an old offset and letting the consumer run (Offsets and Commits).
In an acknowledgement model, the service tracks which messages a subscription has acknowledged and redelivers the rest. Rewinding exists, but it is a subscription-level administrative operation with its own semantics about what is redelivered and to whom, and it is bounded by a configured retention on the subscription rather than on the topic. Neither model is better; they support different runbooks, and the mistake is writing a runbook for the one you do not have (Replay from the Log).
Whichever you have, the log is a bounded recovery mechanism. Retention is the length of your fast-replay window, and every data platform eventually discovers a bug older than it. That is the argument for landing raw records in object storage in parallel with the stream: the log gives fast replay for a window, and the lake gives slow replay forever (The Raw Landing Zone, Keeping Raw History: The Recovery Position and the Liability).
| Check | Expresses | Catches | Still misses |
|---|---|---|---|
| Consumer lag per ordering unit, with an alert on the maximum rather than the average. | Every unit is being consumed, not just most of them. | A stalled partition, a consumer instance that died without its group noticing, a unit whose key became hot. | Correctness entirely. A consumer perfectly caught up on every unit can be writing wrong rows, and out-of-order processing leaves lag looking perfect (The Average Was Fine and Users Were Not). |
| Distance between the oldest retained record and the slowest consumer position. | There is still room before data is lost rather than delayed. | The retention cliff, days before it is reached — the only warning available for a failure that is otherwise instantaneous and irreversible. | Anything about a consumer that is keeping up while processing incorrectly, and it says nothing about producers that stopped sending. |
| Per-key version monotonicity in the sink: alert when an applied version is lower than the stored one. | Records for an entity are being applied in a defensible order. | Cross-unit disorder after a rescale, a mis-set ordering key, a clock-based merge picking the wrong record. | Missing records — a version that never arrives at all leaves the stored version simply stale, with nothing out of order to detect (Missing Rows). |
| Records produced versus records processed per interval, reconciled against the source system. | Completeness end to end, which no broker metric expresses. | A producer that silently stopped, a dropped batch, a topic nobody is consuming. | Any interval not yet closed, and duplicates that offset losses — the standing blind spot of every count-based reconciliation (Reconciliation). |
Only the first two are streaming-specific. The last two are the ordinary quality portfolio, and a streaming platform needs them more than a batch one does, because there is no run boundary at which someone would otherwise look (Data Quality).
Set a long retention, treat the stream as the durable record of everything that happened, and plan to replay from it whenever a bug is found. Consumers write directly to the warehouse and nothing else retains the raw records.
Set retention from the recovery requirement, land every raw record to object storage as it arrives, and keep two replay paths: seek the log for anything inside the window, and reprocess from the lake for anything older. Alert on the slowest consumer's distance to the retention cliff, not on average lag.
Retention is a horizon, and a horizon is exactly the wrong shape for an archive: the data you most need after discovering a subtle bug is the data closest to expiry. A log is optimised for sequential replay at high throughput within a window, and object storage is optimised for retaining bytes indefinitely at rest — using each for what it is good at costs one extra sink and removes the entire class of incident where the records are simply gone.
Retention maxima, snapshot and seek semantics, ordering-key behaviour, consumer-group rebalance protocols and the exact guarantees of any transactional producer or consumer feature are all version- and product-specific, and several have changed materially. Test the replay operation you intend to rely on against the product and version you actually run, and verify current documentation before writing the runbook.
How to build it
Most important first.
- Decide what your ordering requirement actually is before choosing anything, and write it as a sentence about a key: "updates to one order must be applied in the order they were produced" is a requirement; "we need ordering" is not (Event Keys and Partition Assignment).
- If you need per-key ordering, pick a product whose ordering unit is explicit, key on the entity, and treat the unit count as a semi-permanent decision. Rescaling is a controlled operation with an ordering consequence, not a slider (Topics and Partitions). If you do not need ordering, say so explicitly and take the operational simplicity of a model with no units to size. Choosing partitions you do not need is choosing a hot-key problem you did not have (Hot Keys: When Aggregate Metrics Hide a Saturated Node).
- Make every consumer idempotent regardless of product, keyed on a producer-supplied identifier. At-least-once is the floor everywhere and duplicates arrive on restart, on rebalance and on retry (Idempotent Data Pipelines, Deduplication).
- Do not let the log be the only copy. Land raw records in object storage as they arrive, so a consumer that falls past the retention horizon has somewhere to recover from (The Raw Landing Zone, Keeping Raw History: The Recovery Position and the Liability).
- Set retention from your recovery requirement, not from your consumer lag. The question is "how far back must we be able to replay after discovering a bug", and the answer is usually longer than anyone's default (Replay from the Log).
- Design the rescale before you need it: drain, or dual-write, or accept a bounded window of cross-unit disorder with a version-aware merge downstream that cannot regress state (Upserts and Merges). Version-stamp records at the producer. If every record carries a monotonically increasing version per key, a consumer can be correct even when ordering is not guaranteed — which is the only defence that survives a rescale (CDC Ordering and Transaction Boundaries).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Durability of an acknowledged record, replicated within a region. This is the strongest and most portable guarantee in the set (The Event Log).
- Ordering within an ordering unit only, and only for as long as a key maps to the same unit. There is no global ordering across a topic in any of these products, and there is no ordering at all in a subscription model without an explicit ordering key (Topics and Partitions).
- At-least-once delivery by default everywhere. A consumer will see duplicates, and the only question is whether it was written to tolerate them (At-Least-Once Delivery).
- Replayability bounded by retention. Beyond the horizon there is no guarantee whatsoever, and the failure is silent from the log's point of view — it simply has nothing to give you (Retention and Replay).
- No completeness guarantee end to end. That a producer intended to send a record is not something the log can tell you; only reconciliation against the source can (Reconciliation).
- No schema guarantee. The log carries bytes, and whether they still mean what a consumer thinks is a contract question the broker has no opinion about (Schema Registry).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check that matters is a per-unit lag and gap audit: for each ordering unit, the consumer's position versus the latest record, plus a sequence-continuity check on the records actually processed. It catches a stalled unit, a consumer that silently stopped, and a rebalance that skipped records — none of which show up in an average.
- It misses everything about content. A consumer perfectly caught up on every unit can be writing wrong rows, and duplicates that a downstream merge silently absorbs never appear in a lag metric (Duplicate Rows).
- It also misses the failure this lesson is really about: records processed in the wrong relative order still make a lag audit look perfect. Detecting that needs a per-key version check downstream — a row whose version went backwards (CDC Ordering and Transaction Boundaries).
- All four make records readable within a very short time of acknowledgement, so the log is almost never the source of staleness. The delay a consumer feels is its own lag, not the platform's (Depth Is Not an Emergency; Age Is).
- Consumer lag is the freshness metric that matters and it is a per-unit quantity: an average lag of nearly zero across sixteen units with one unit hours behind is a broken pipeline that looks healthy (Percentiles: Which One, and How Many Users Is That?).
- Ordering-key constrained delivery in a subscription model serialises records for that key, which bounds throughput per key. That is a freshness ceiling you have chosen, and it is the same ceiling a partition imposes — the difference is only that one is visible in the topic configuration and the other in the message attributes.
- The log is schema-agnostic, so schema evolution is entirely a producer-and-consumer contract question enforced by a registry or by nothing at all (Schema Registry, Backward Compatibility).
- Because records are retained, a consumer replaying history will encounter several schema versions in one stream. Any consumer that can replay must be able to read every version still inside the retention window, which is a stronger requirement than being able to read the current one (Schema Evolution).
- Changing the unit count is a semantic change to the ordering guarantee even though nothing about the data changed. Treat it as a contract event and tell consumers (Data Contracts).
- Migrating between these products is not a connector swap. The ordering model, the position model and the replay operation all differ, so consumer code that survives the migration unchanged is consumer code that was probably relying on something it should not have (Cloud Data Services).
- Replay from a position is the recovery mechanism, and it is why a log is worth having: a bug in a consumer costs a re-read rather than lost data, provided the window is long enough (Replay from the Log).
- Replay is only safe if the sink is idempotent. Re-reading a month of records into a non-idempotent sink turns a recovery into a much larger incident (Idempotent Data Pipelines, Upserts and Merges).
- In an offset model, recovery means committing an earlier position — a small, well-understood operation. In an acknowledgement model, it means a subscription-level seek with its own semantics for what gets redelivered to whom. Rehearse the one you actually have, because the runbooks are not interchangeable.
- Past the retention horizon there is no recovery from the log at all. This is the single strongest argument for landing raw records in object storage in parallel: the log gives you fast replay for a window, the lake gives you slow replay forever (The Raw Landing Zone).
What can go wrong
- An ordering assumption that holds in testing and breaks at the first rescale, because rescaling is the only event that moves a key between units.
- A consumer past the retention horizon: not late, gone.
- A hot key saturating one unit while the others idle, with ordering perfectly preserved and throughput collapsed (Hot Keys: When Aggregate Metrics Hide a Saturated Node).
- Duplicates on every restart, absorbed silently by a sink that overwrites, and not absorbed at all by one that appends (Duplicate Rows).
- The mitigation failing too: a version-aware merge downstream that compares timestamps assigned by producers whose clocks disagree, so "newest wins" picks the wrong record (Processing Time).
- An average consumer-lag dashboard that is green while one unit is hours behind (The Average Was Fine and Users Were Not).
- "The log guarantees ordering." It guarantees ordering within an ordering unit, and only while a key stays on the same unit. Across units, and across a rescale, there is no order at all (Topics and Partitions).
- "These services are interchangeable, they all do streaming." They realise the same primitive with three different ordering models and two different position models. A consumer written against one is not portable to another without re-examining both (Cloud Data Services).
- "We have at-least-once, so we might see a duplicate occasionally." You will see duplicates routinely — on restart, on rebalance, on retry, on deploy. Idempotency is a design requirement, not a defensive nicety (Deduplication).
- "Retention is a cost setting." It is the length of your recovery window. Choosing it from a storage budget rather than from "how long might a bug go unnoticed" is how a replay becomes impossible (Replay from the Log).
- "More partitions means more throughput." More units means more parallelism only if the key distribution spreads across them. With one dominant key, every additional unit is idle (Data Skew).
- A log is a copy of production data with a retention window and, usually, a much coarser access model than the warehouse downstream of it. Topic-level access is not row-level access (Data Access Control).
- Deletion obligations are awkward here: records are immutable and addressed by position, so satisfying a deletion request generally means letting retention expire or re-keying the design so that a subject's records are separable. Decide which at design time (Deletion Requests, PII in Pipelines).
- Long retention extends the window during which regulated payloads exist in a system whose primary purpose is availability rather than governance. Minimising what is put on the log is cheaper than governing it afterwards (Data Minimization).
Operating it
- Consumer lag per unit, not averaged. This is the single most important operational signal a streaming platform has (The Backlog Arithmetic: Four Levers and a Drain Time, Percentiles: Which One, and How Many Users Is That?).
- Records in and records out per consumer group, reconciled per interval — the completeness signal that lag does not give you (Reconciliation).
- Oldest retained record versus the slowest consumer's position: the distance to the retention cliff, which is the metric that turns a sudden data loss into an alert days earlier (Retention and Replay).
- Per-unit throughput distribution, which is where a hot key announces itself long before anyone calls it skew (Data Skew).
- Rebalance and redelivery counts, because a consumer group that is constantly rebalancing is a consumer group that is constantly reprocessing (Consumer Groups and the Parallelism Ceiling).
- At 10x throughput, unit count is the lever, and using it is an ordering event. Planning the unit count with headroom is cheaper than rescaling under load (Headroom: The Capacity You Deliberately Do Not Use).
- At 100x, key distribution decides everything. Ordering unit and parallelism unit are the same thing, so a skewed key space caps throughput no matter how many units exist (Salting a Skewed Key).
- Consumer count scales read bytes and fan-out cost linearly, and scales the coordination problem more than linearly in models with consumer groups — every additional group is another position to track and another rebalance to survive (Consumer Groups and the Parallelism Ceiling).
- Retained bytes multiplied by the retention window and by the replication factor. Retention is the most direct cost lever the log has and it is also the recovery lever, which is the tension (Cost vs Freshness).
- Provisioned throughput or unit count, which is capacity held rather than work done in most of these products (Idle Capacity: Headroom or Waste?).
- Bytes read by consumers, multiplied by how many independent consumers read the same records — fan-out is a cost driver as well as an architectural benefit.
- Cross-zone or cross-region traffic between producers, brokers and consumers, which is invisible in the architecture diagram (Egress: Moving Data Costs Money, Not Just Storing It).
- Reprocessing: every replay re-reads and re-computes, so a long retention window is also a large potential compute bill (Compute Waste).
- Explicit ordering units buy per-key ordering and a rewindable position, and cost you a capacity decision that is awkward to change and a hot-key ceiling you cannot design around (Hot Keys: When Aggregate Metrics Hide a Saturated Node).
- A unit-free subscription model buys effortless scaling and costs you ordering by default; turning ordering on per key reintroduces the throughput ceiling you were trying to avoid.
- Long retention buys recovery and costs storage and — much more importantly — a longer window during which regulated data sits in a system whose access model is coarser than the warehouse's (Data Retention).
CDC ordering lab
Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.
| Row | Final state after applying | Truth at the source |
|---|---|---|
| A | shipped | returned (lsn 8) |
| B | cancelled | cancelled (lsn 6) |
| C | shipped | shipped (lsn 7) |
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- BROKER-SPECIFICOrdering is per-partition in the Kafka family and in Event Hubs, per-shard in Kinesis, and absent in Pub/Sub unless an ordering key is set — at which point it is per ordering key. A design assuming per-key ordering is not portable between these without changing the design, not merely the client library.
- GENERALThe primitive underneath — a durable, replayable, unit-partitioned append-only log whose readers hold independent positions — is identical across all of them and across self-hosted deployments, so the reasoning about ordering, replay and retention transfers even where the vocabulary does not.
- CLOUD-SPECIFICWhich of these exists on a given provider, whether a managed Kafka offering sits alongside a native service, and how throughput is provisioned or auto-scaled are provider-and-product properties that have changed repeatedly and should be verified rather than remembered.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns why a partitioned log can offer total order within a unit and nothing across units, what replication promises an acknowledged write, and why consensus is what makes a broker durable.
- — DevOps / Production Engineering owns operating these platforms — topic and subscription configuration as code, capacity changes as controlled operations, and the deployment practices that stop a rolling restart from becoming a rebalance storm.