Concurrency Practice

We count feature-flag evaluations in memory and scrape the map every 15 seconds. In the test suite the count is exact. In production it is always low and never high — yesterday the request log had 1,204,338 evaluations and the counter reported 1,199,871. That is a 0.37% shortfall. Nothing throws, nothing is logged, and the gap gets wider on our busiest hosts.

A customer had £220 and made two £200 withdrawals from two devices, four milliseconds apart. Both were approved. £400 left the account. The account balance now reads £20 — not negative, which is why none of our balance alarms fired. Support only found it because the customer told us. We are single-threaded Node, so several people on the team think this is impossible.

The ledger service stopped. Not slowed — stopped. It happened at 03:14, and the first symptom was two requests that never returned. Over the next forty minutes every other worker thread was consumed one at a time until the health check timed out and the orchestrator killed the pod. The pod that replaced it ran fine for six hours and then did the same thing.

At 09:00 every weekday our Node API's p99 goes from 40ms to 1.4s and stays there for about ten minutes. It is not one endpoint — it is all of them, including /healthz, which returns a hard-coded object and touches nothing. The box has eight cores and overall CPU is 12%. The database is bored. We have doubled the replica count twice and the spike is identical.

The ingest service runs out of memory every six to nine hours and restarts. We have been hunting a leak for a week. Heap dumps are dominated by event objects, but every one of them is legitimately referenced — nothing is held by a listener that forgot to unsubscribe, no closure captures anything odd, and the profiler shows no growth in any cache. We raised the heap from 4GB to 12GB and it now OOMs every eighteen hours instead of every seven.

Checkout p99 is 30 seconds, up from 300ms, since we launched in a new region. The DBA has been telling us for two days that the database is idle: 4% CPU, no slow queries, no lock waits, 20 connections open out of a max of 500. Our application logs show requests spending all their time somewhere before the query even runs. We raised the connection pool from 20 to 100 in staging and it did not help, so we reverted it.

Our nightly reconciliation job logs "processed 240,000 records", exits 0, and the scheduler marks it green. Between 3% and 8% of the records are not in the database the next morning. No errors, no warnings, no failed batches — the log is completely clean. The missing records are different every night. Downstream reporting has been quietly wrong for about two months.

Every hour, at exactly :00, our database CPU goes to 100% for about forty seconds and API p99 goes from 60ms to 9s. Then it recovers completely on its own. Nothing is scheduled at :00 — we checked every cron in the estate. Traffic at :00 is identical to traffic at :59. The cache hit rate chart has a vertical cliff at the top of every hour.

Inventory reservation. At 12:00 sharp, when the daily drop goes live, all eight cores pin at 100% and completed reservations per second falls to about four. Not zero — four, from a normal three hundred. Nothing is blocked: every thread dump we take shows every thread RUNNABLE. It clears itself after six to ten minutes, which is roughly when the popular item sells out. We have tried more instances; it gets worse.

Our config service holds a rate-table snapshot behind a read/write lock. Reads run at about 40,000 per second across 64 threads. A config push used to apply in under a millisecond. Since the traffic doubled last month, pushes sometimes take ninety seconds and sometimes never complete at all — the deploy tool times out at five minutes and reports a failed rollout, even though the service is serving reads perfectly the whole time. Restarting the pod applies the config instantly.

Roughly one deploy in thirty, one of our four ingest workers comes up and never processes anything. It does not crash, does not log, does not appear unhealthy — the process is alive and the other three workers carry the load, so we only noticed because throughput was 25% low for a week. Restarting the pod always fixes it. We cannot reproduce it locally, and it happens more often on the faster hosts.

Our pricing service holds a rate table in memory and republishes it every thirty seconds from the upstream feed. About once every few hours a quote goes out with the new currency list and the previous set of rates — a EUR line priced at yesterday's USD number. It has cost us real money twice. We added a reader-side assertion that the two arrays are the same length and it fires roughly once in forty million quotes. When we added logging around the publisher to catch it, it stopped happening for two days, then came back.

Our report generator runs a twelve-step pipeline, several steps of which call an LLM provider. Users press Stop on long reports fairly often — the UI clears immediately and the request shows as cancelled. Our provider bill this month is 4.1x what our usage dashboard says users consumed. We also have a support queue full of people saying cancelled reports appeared in their export folder up to twenty minutes later, sometimes with the wrong data.

Our dashboard endpoint used to take 6.2 seconds — a loop of 60 sequential awaits, one per widget. We changed the loop to Promise.all and it now takes 190ms. Everyone was delighted. Two days later, during Monday peak, the database started refusing connections, and three other services that share that database began timing out. Our endpoint is still fast. We have rolled forward, not back, because nobody wants to give up the 30x.

Our rate limiter had one atomic counter and it was a bottleneck at high thread counts, so we sharded it: eight independent counters, one per worker thread, summed on read. No lock, no atomic contention, each thread touches only its own slot. Throughput dropped by a factor of three. The results are correct every single run — it is purely a performance problem, and it gets worse the more threads we add, which is the opposite of what sharding is supposed to do.

Two rare startup problems that we assumed were unrelated until this week. First: some pods show 80 database connections when the pool is configured for 40, and the DBA sees two distinct application_name sessions from one pod. Second: about one request in ten million, very early in a pod's life, segfaults dereferencing a member of our config singleton that the constructor definitely sets. Both got roughly eight times more common after we migrated to Graviton instances.