Async ProcessingoptimizationIntermediate

The notification queue has been "almost caught up" for four days

Symptoms

  • Queue depth is 2.4 M and the oldest message is 7.4 hours old; both numbers have risen every day since Monday.
  • Order confirmation emails arrive hours after the order; customers call to ask whether the order went through.
  • Worker CPU is low (~9% per worker); adding CPU to the worker nodes changed nothing.
  • It started on Monday at 09:00, the morning the "order timeline" feature launched.
# queue: notifications  (per-minute rates, daily averages)
day    enqueue/min   dequeue/min   depth        oldest msg age
Fri    3,900         4,100         1,200        2 s
Mon    8,700         4,150         290,000      41 min      ← order-timeline launch 09:00
Tue    8,900         4,180         980,000      3.2 h
Wed    9,100         4,160         1,700,000    5.9 h
Thu    9,000         4,190         2,400,000    7.4 h

# consumers: 8 worker processes × concurrency 1  (prefetch 1)
# handler, average 112 ms:
#   load user prefs (SELECT, indexed)     3 ms
#   render template                        6 ms
#   sendgrid.send()                       98 ms   ← network wait; the worker does nothing else meanwhile
#   ack                                    5 ms
# ceiling: 8 workers × 1 × (60 000 / 112) ≈ 4,285 msg/min

# producer change (order-svc, Monday): emits NotificationRequested on every status change
#   before: OrderPlaced only            (1 per order)
#   after:  Placed, Paid, Packed, Shipped (4 per order) — prefs default: email for all four

Investigate

Inspect areas in any order (0/7 inspected). When you think you know the root cause, commit to it.

Worker CPU
The feature launched Monday
Consumer count and concurrency
SendGrid latency
Database
Broker health
Dead-letter queue