Events & MessagingdebuggingIntermediate
OrderUpdated arrived before OrderCreated
Symptoms
- The order search index shows orders with a status but no items, and some orders are missing entirely.
- The indexer’s error rate went from 0 to ~4% ("order 5510 not found"); the dead-letter topic is filling at 60 msg/min.
- Consumer lag is near zero — the consumers are keeping up.
- It started at 10:55, right after order-svc deployed a change titled "spread producer load across partitions".
# order-indexer consumer log (group=indexer, 6 consumers, topic orders: 12 partitions)
11:02:44.120 p7 off=88213 OrderUpdated order=5510 status=PAID → ERROR order 5510 not found → DLQ
11:02:44.131 p2 off=91004 OrderCreated order=5510 items=3
11:02:44.140 p11 off=87730 OrderUpdated order=5510 status=CONFIRMED
11:02:44.152 p7 off=88214 OrderUpdated order=6120 status=PAID → ok (6120 created 40 s ago)
# order-svc producer diff, deployed 10:55
- await producer.send({ topic: 'orders', messages: [{ key: String(order.id), value }] })
+ await producer.send({ topic: 'orders', messages: [{ key: null, value }] }) // round-robin partitioner
# partition skew before the change (bytes/s, 12 partitions): max/mean = 1.1Investigate
Inspect areas in any order (0/7 inspected). When you think you know the root cause, commit to it.
Broker health
Consumer lag
The producer change
Consumer group rebalancing
Producer idempotence and retries
The indexer’s handling of unknown orders
Partition skew before the change