intermediate
Every Core Is Busy
Read the evidence before you read the options. The signals are presented the way a dashboard would present them — nothing is labelled with the answer.
The report
Support says the product feels sluggish. It started sometime this week — nobody can say exactly when. The infrastructure channel has been pointing at a CPU graph that is "basically flat at 100%" for two days.
The system
Catalog API — service dashboard, 48 h windowILLUSTRATIVE
| Signal | Value | What it tells you |
|---|---|---|
| CPU utilisation | 97–100% (all 12 instances) | Every instance is at or near its core allocation, continuously, for two days. |
| Request rate | 1,180 req/s (last week: 1,140) | Traffic is within 4% of the same window last week. |
| p50 latency | 210 ms (was 95 ms) | The typical request takes roughly twice as long as it did. |
| p99 latency | 1.9 s (was 640 ms) | The tail has grown by about the same factor as the median. |
| Run queue length | 8.4 average | On average eight threads are runnable but waiting for a core. |
| DB query time (p99) | 31 ms (was 29 ms) | Database spans are essentially unchanged. |
| Cache hit rate | 94% (was 94%) | Unchanged. |
| Memory / GC time | 46% heap, 1.8% of wall clock in GC | Heap is comfortable and collection accounts for under 2% of time. |
CPU profile, 60 s sample taken during the busy period (self time)
55.1% catalog/pricing.applyRules | └─ 52.7% regexp.Compile | └─ 51.9% regexp.parse 9.8% encoding/json.Marshal 7.2% net/http.(*conn).serve 5.4% catalog/handler.Search 4.1% database/sql.(*Rows).Scan 3.0% runtime.gcBgMarkWorker 15.4% (everything else, none above 1.2%)
git log --oneline --since="4 days ago" -- services/catalog
a91f2ce feat(pricing): per-region promotional rules 7d4e118 chore: bump json library 2b09aa4 fix(search): trim whitespace in query terms
What is the constraint?