Events & MessagingdebuggingIntermediate

OrderUpdated arrived before OrderCreated

Symptoms

  • The order search index shows orders with a status but no items, and some orders are missing entirely.
  • The indexer’s error rate went from 0 to ~4% ("order 5510 not found"); the dead-letter topic is filling at 60 msg/min.
  • Consumer lag is near zero — the consumers are keeping up.
  • It started at 10:55, right after order-svc deployed a change titled "spread producer load across partitions".
# order-indexer consumer log (group=indexer, 6 consumers, topic orders: 12 partitions)
11:02:44.120  p7   off=88213  OrderUpdated  order=5510 status=PAID       → ERROR order 5510 not found → DLQ
11:02:44.131  p2   off=91004  OrderCreated  order=5510 items=3
11:02:44.140  p11  off=87730  OrderUpdated  order=5510 status=CONFIRMED
11:02:44.152  p7   off=88214  OrderUpdated  order=6120 status=PAID       → ok (6120 created 40 s ago)

# order-svc producer diff, deployed 10:55
- await producer.send({ topic: 'orders', messages: [{ key: String(order.id), value }] })
+ await producer.send({ topic: 'orders', messages: [{ key: null, value }] })   // round-robin partitioner

# partition skew before the change (bytes/s, 12 partitions): max/mean = 1.1

Investigate

Inspect areas in any order (0/7 inspected). When you think you know the root cause, commit to it.

Broker health
Consumer lag
The producer change
Consumer group rebalancing
Producer idempotence and retries
The indexer’s handling of unknown orders
Partition skew before the change