← PracticeIntermediatePools

The App Is Slow and the Database Is Bored

Pull up the evidence one item at a time, commit to a diagnosis, and only then see the schedule that actually ran.

What was reported

Checkout p99 is 30 seconds, up from 300ms, since we launched in a new region. The DBA has been telling us for two days that the database is idle: 4% CPU, no slow queries, no lock waits, 20 connections open out of a max of 500. Our application logs show requests spending all their time somewhere before the query even runs. We raised the connection pool from 20 to 100 in staging and it did not help, so we reverted it.
1@Transactional                                  // borrows a connection here
2public Order checkout(Cart cart) {
3    Order order = orderRepo.save(newOrder(cart));           // ~4ms
4    PaymentResult r = paymentGateway.charge(cart.total());  // external, p99 6s
5    order.markPaid(r.id());
6    orderRepo.save(order);                                  // ~4ms
7    return order;                                           // connection released here
8}

Evidence

Nothing here is labelled as relevant. Some of it is not.

What is actually happening?