Distributed DatadebuggingAdvanced

Forty-one orders shipped without a successful payment

Symptoms

  • Finance reconciliation found 41 orders in state SHIPPED whose payment record is failed (card declined). The goods left the warehouse; nobody was charged.
  • All 41 fall in a two-hour window on one morning; outside that window, failed payments never ship.
  • A separate, older problem surfaced in the same query: ~1,900 orders since launch sit in PENDING forever with stock still reserved.
  • No errors in any service; every handler ran and returned 200.
# events for order 7712 (broker timestamps)
10:14:02.010  orders      OrderCreated        7712
10:14:02.940  payments    PaymentFailed       7712   reason=card_declined
10:14:06.480  inventory   InventoryReserved   7712   (warehouse API took 4.3 s — degraded 09:50–11:55)
10:14:06.502  orders      OrderConfirmed      7712   ← emitted by order-svc
10:14:06.610  shipping    ShipmentRequested   7712   ← shipping-svc on OrderConfirmed

# order-svc/handlers.ts (choreography: each handler reacts to one event)
onPaymentCaptured:    order.paymentStatus = 'captured'
                      if (order.inventoryReserved) confirm(order)
onPaymentFailed:      order.paymentStatus = 'failed'                      // that's all
onInventoryReserved:  order.inventoryReserved = true
                      if (order.paymentStatus !== 'pending') confirm(order)   // "payment already came back"

# a normal day: InventoryReserved after ~120 ms, Payment* after ~900 ms
#   → declined orders: PENDING forever, stock reserved, no email, no cancel

Investigate

Inspect areas in any order (0/7 inspected). When you think you know the root cause, commit to it.

Payment provider
Warehouse API latency
The confirmation condition in order-svc
Compensation on payment failure
Broker ordering guarantees
Recent deploys
Duplicate delivery