← PracticeIntermediateAsync

Every Endpoint Got Slower At Once

Pull up the evidence one item at a time, commit to a diagnosis, and only then see the schedule that actually ran.

What was reported

At 09:00 every weekday our Node API's p99 goes from 40ms to 1.4s and stays there for about ten minutes. It is not one endpoint — it is all of them, including /healthz, which returns a hard-coded object and touches nothing. The box has eight cores and overall CPU is 12%. The database is bored. We have doubled the replica count twice and the spike is identical.
1// added three weeks ago
2app.get('/reports/daily', async (req, res) => {
3  const rows = await db.query(dailyReportSql)      // ~40MB of objects
4  const csv  = rows.map(toCsvLine).join('\n')      // synchronous
5  const body = JSON.stringify({ csv, meta: buildMeta(rows) })  // synchronous
6  res.type('application/json').send(body)
7})

Evidence

Nothing here is labelled as relevant. Some of it is not.

What is actually happening?