Distributed DatadebuggingAdvanced
Forty-one orders shipped without a successful payment
Symptoms
- Finance reconciliation found 41 orders in state
SHIPPEDwhose payment record isfailed(card declined). The goods left the warehouse; nobody was charged. - All 41 fall in a two-hour window on one morning; outside that window, failed payments never ship.
- A separate, older problem surfaced in the same query: ~1,900 orders since launch sit in
PENDINGforever with stock still reserved. - No errors in any service; every handler ran and returned 200.
# events for order 7712 (broker timestamps)
10:14:02.010 orders OrderCreated 7712
10:14:02.940 payments PaymentFailed 7712 reason=card_declined
10:14:06.480 inventory InventoryReserved 7712 (warehouse API took 4.3 s — degraded 09:50–11:55)
10:14:06.502 orders OrderConfirmed 7712 ← emitted by order-svc
10:14:06.610 shipping ShipmentRequested 7712 ← shipping-svc on OrderConfirmed
# order-svc/handlers.ts (choreography: each handler reacts to one event)
onPaymentCaptured: order.paymentStatus = 'captured'
if (order.inventoryReserved) confirm(order)
onPaymentFailed: order.paymentStatus = 'failed' // that's all
onInventoryReserved: order.inventoryReserved = true
if (order.paymentStatus !== 'pending') confirm(order) // "payment already came back"
# a normal day: InventoryReserved after ~120 ms, Payment* after ~900 ms
# → declined orders: PENDING forever, stock reserved, no email, no cancelInvestigate
Inspect areas in any order (0/7 inspected). When you think you know the root cause, commit to it.
Payment provider
Warehouse API latency
The confirmation condition in order-svc
Compensation on payment failure
Broker ordering guarantees
Recent deploys
Duplicate delivery