Application ArchitecturedesignExpert

Nine services, one database, one deploy train

Symptoms

  • Releases are a four-hour coordinated event every second Tuesday in which all nine services deploy together; the last one that shipped a single service alone (November) broke pricing.
  • A column rename on customers broke four services at once; the migration was rolled back and rescheduled for the next train.
  • Any one service being down returns 503 for checkout; last quarter’s availability was 98.7% against a 99.9% target.
  • Checkout p99 is 1.9 s across 11 synchronous hops; a developer needs all nine services running to test a change to any of them.
# one checkout request (all hops synchronous and required)
client → gw → checkout-svc → customer-svc ─────────────────────────► shop.customers
                          → cart-svc ──────────────────────────────► shop.carts
                                        → pricing-svc ─────────────► shop.products, shop.customers   ← reads customer tier directly
                          → inventory-svc ─────────────────────────► shop.stock
                          → payment-svc ───────────────────────────► shop.payments, shop.orders      ← writes the orders table
                          → order-svc ─────────────────────────────► shop.orders
                                        → notification-svc → email provider (sync, 400 ms)

# database 'shop': one Postgres · 63 tables · 9 services connect as the same role 'app'
# tables written by ≥ 2 services: orders (3), customers (2)   · read by ≥ 3 services: customers (7), products (6), orders (5)
# shared package shop-common@3.41: DTOs mirroring table rows; every service pins the exact version

# availability arithmetic: 11 required hops at 99.9 % each → 0.999^11 ≈ 98.9 %
# release train v3.41: 9 services, 4 h 10 min, 1 rollback · v3.40: 3 h 55 min

Investigate

Inspect areas in any order (0/7 inspected). When you think you know the root cause, commit to it.

Infrastructure cost and cluster health
Database ownership
Communication style
The shared shop-common package
Team structure
Network latency
Test suite