The question this answers
A hundred concurrent callers want the same value — how do I make that one fetch, and what do they all get when it fails?
100 concurrent requests for the same user profile arriving within 40 ms of a cache miss, against an origin that takes 200 ms to answer.
A Map<key, Promise<T>> of in-flight fetches. Every caller reads it and the first caller writes it; the entry is the coordination point, and its lifecycle — created on the first miss, removed on settlement — is where all the bugs live.
At most one fetch is in flight per dedup key at any moment, every caller receives the result of a fetch that started at or after its own arrival, and the map holds no entry for a key whose fetch has already settled.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
Fifteen lines, and the two that matter
The mechanism is a map from key to in-flight promise. The first caller finds nothing, starts the fetch, stores the promise, and awaits it. Every later caller finds the promise and awaits the same one. When it settles, the entry is removed so the next miss starts a fresh fetch. That is the whole thing, and in the good case it turns 100 origin requests into 1.
The two lines that decide whether it works in production are the key and the cleanup. The key must include everything that changes the result — the user id, the locale, the API version, the tenant, the authorisation scope. A key that omits the tenant will serve one tenant's data to another, which is a security incident rather than a caching bug. The cleanup must run on rejection as well as fulfilment, and it must be conditional on identity, or a slow caller's cleanup deletes a newer caller's in-flight entry (Futures & Promises).
The failure policy is a genuine design decision with no default answer. Sharing the error means one transient blip fails a hundred callers at once. Not sharing it means the hundred all retry independently, which is the herd you were preventing. The usual compromise is to share the failure but keep the window short — remove the entry immediately on rejection, so the very next caller starts a new attempt rather than joining a doomed one.
1type Entry<T> = { promise: Promise<T>; subscribers: number; startedAt: number }2 3class SingleFlight<T> {4 private inFlight = new Map<string, Entry<T>>()5 6 // key MUST include everything that changes the result: tenant, locale,7 // api version, auth scope. Omitting the tenant is a data-leak bug, not a8 // cache-hit-rate bug.9 async do(key: string, fn: () => Promise<T>, signal?: AbortSignal): Promise<T> {10 const existing = this.inFlight.get(key)11 if (existing) {12 existing.subscribers += 113 // Each caller may give up independently WITHOUT cancelling the shared fetch.14 return this.withSignal(existing.promise, signal)15 }16 17 const entry: Entry<T> = { promise: null as never, subscribers: 1, startedAt: Date.now() }18 19 entry.promise = (async () => {20 try {21 return await fn()22 } finally {23 // Conditional on identity: if a newer entry replaced ours, leave it alone.24 // Deleting unconditionally evicts someone else's in-flight fetch and the25 // coalescing silently stops working under exactly the load it exists for.26 if (this.inFlight.get(key) === entry) this.inFlight.delete(key)27 }28 })()29 30 // Set BEFORE any await, so concurrent callers in the same tick find it.31 this.inFlight.set(key, entry)32 return this.withSignal(entry.promise, signal)33 }34 35 // A caller that aborts stops waiting; the shared fetch continues for everyone else.36 private withSignal(p: Promise<T>, signal?: AbortSignal): Promise<T> {37 if (!signal) return p38 return Promise.race([39 p,40 new Promise<never>((_, reject) => {41 if (signal.aborted) return reject(signal.reason)42 signal.addEventListener('abort', () => reject(signal.reason), { once: true })43 }),44 ])45 }46 47 get inFlightCount() { return this.inFlight.size }48}49 50// Usage in front of a cache. Note the double-check INSIDE the flight:51// between the miss and the fetch starting, someone may have filled the cache.52const sf = new SingleFlight<Profile>()53 54async function getProfile(tenant: string, userId: string, locale: string, signal?: AbortSignal) {55 const key = tenant + '|' + userId + '|' + locale + '|v3'56 const hit = await cache.get(key)57 if (hit) return hit58 return sf.do(key, async () => {59 const second = await cache.get(key) // filled while we were waiting?60 if (second) return second61 const fresh = await origin.fetchProfile(tenant, userId, locale)62 await cache.set(key, fresh, jitteredTtl(300)) // jitter, or you rebuilt the herd63 return fresh64 }, signal)65}The two schedules that break it
Both failures below come from the entry's lifecycle, and both are invisible in tests: the first needs a rejection under concurrency, the second needs a caller to arrive in a specific window. The first is the more dangerous, because its symptom is not an error — it is that the coalescing quietly stops working, which looks exactly like ordinary load.
The second schedule is the staleness question, and it is the one that has no purely technical answer. A caller that arrives 190 ms into a 200 ms fetch joins it and receives data that was read from the origin before that caller existed. For a profile that is fine. For a value the caller just wrote, it is a lost update from the reader's point of view. If read-your-writes matters, the caller must bypass single-flight after a write, or the key must incorporate a version that the write bumps (Optimistic Concurrency Control).
| # | Caller A (first miss) | Caller B (joins) | Caller C (arrives after) | In-flight map | State |
|---|---|---|---|---|---|
| 1 | miss; creates entry E1; starts origin fetch | · | · | · | entry=E1 originCalls=1 subs=1 |
| 2 | · | miss; finds E1; subscribes | · | · | entry=E1 originCalls=1 subs=2 |
| 3 | E1 rejects (origin 503); A's finally deletes the map entry | · | · | · | entry=none originCalls=1 subs=2 |
| 4 | · | · | miss; finds nothing; creates entry E2; starts a fresh fetch | · | entry=E2 originCalls=2 subs=1 |
| 5 | · | B's continuation also runs its cleanup: `map.delete(key)` — unconditional | · | · | entry=none originCalls=2 subs=1 ✕ B deleted E2, which belongs to C. The map is now empty while a fetch is in flight, so the next caller starts a third one. Under sustained failure this degenerates to one origin call per caller — the coalescing stops precisely when the origin is already struggling. |
| 6 | · | · | · | with the identity check `if (map.get(key) === entry)`, B's delete is a no-op and E2 survives | entry=E2 originCalls=2 subs=1 |
| 7 | · | · | later scenario: a caller arriving 190 ms into E2's 200 ms fetch joins it | · | entry=E2 joinedAt=190ms dataAsOf=0ms ✕ The caller receives a value read from the origin before it arrived. Harmless for a profile; a read-your-writes violation if that caller just performed a write. |
finally so rejections clean up too. For staleness, decide the join policy deliberately: unconditional joining is right for read-mostly data, and a caller that just wrote must bypass single-flight or use a key that its write bumped.Designing the key and the failure policy
These two decisions carry the whole design, and both have a wrong answer that looks reasonable. Under-specifying the key is a correctness and sometimes a security bug; over-specifying it means every caller has a unique key and coalesces with nobody, so the machinery does nothing while looking like it works. The measurement that tells you which you have is the coalescing ratio: origin calls divided by callers. It should be far below 1, and if it is 1 you have either an over-specified key or the cleanup race above.
The scope decision matters at least as much. This is a per-process map, so twenty instances means twenty flights — a 20× reduction, not a 100× one. Making it cross-process needs a distributed lock or a shared coordination service, with all the correctness warnings in A Mutex on Server A Does Nothing About Server B; the honest position is that per-process single-flight plus jittered TTLs is usually enough, and reaching for distributed coordination for a cache fill is rarely worth what it costs.
| Decision | Option | Consequence | Fails as |
|---|---|---|---|
| Dedup key | Under-specified (omits tenant / locale / scope) | Callers coalesce who should not | One tenant receives another's data — a security incident, not a cache bug |
| Dedup key | Over-specified (includes request id, timestamp) | Nobody ever coalesces | Machinery in place, coalescing ratio stays at 1, no benefit and extra code |
| Dedup key | Correct: every input that changes the result, nothing that does not | One flight per distinct result | Correct — verify with the coalescing ratio |
| Failure policy | Share the rejection with all subscribers | One blip fails 100 callers at once | A correlated error spike; usually acceptable if the entry is removed immediately |
| Failure policy | Retry inside the flight before rejecting | Subscribers wait longer but usually succeed | The shared fetch now holds a resource much longer; needs a hard deadline |
| Failure policy | Do not share failures — each caller retries alone | The herd you were preventing, at the worst moment | Origin overload during exactly the incident that caused the failure |
| Cleanup timing | Delete on settle, unconditionally | A late continuation evicts a newer entry | Coalescing silently degrades to 1:1 under sustained failure |
| Cleanup timing | Delete on settle, only if identity matches | Newer flights survive older cleanups | Correct |
| Scope | Per process | N instances means N flights | Fine at 20 instances; a 20x reduction, not 100x |
| Scope | Cross-process via a distributed lock | Exactly one flight in the fleet | A lock that can be lost or expire mid-fetch — see A Mutex on Server A Does Nothing About Server B |
Key points
- A map from dedup key to in-flight promise: the first caller fetches, everyone else subscribes to the same handle.
- The key must include every input that changes the result — omitting a tenant or a scope is a data-leak bug, not a cache-hit-rate bug.
- Over-specifying the key is the silent failure: the machinery runs and nobody ever coalesces.
- Cleanup must run in a
finallyso rejections clean up, and must be conditional on entry identity so a late continuation cannot evict a newer flight. - The failure policy is a real decision: share the rejection and remove the entry immediately, so the next caller starts fresh rather than joining a doomed flight.
- Set the map entry before any
await, or two callers in the same tick both start a fetch. - It is per-process. Twenty instances means twenty flights, and making it fleet-wide costs far more than it usually saves.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • A caller computes the dedup key from every input that affects the result.
- • It looks the key up in the in-flight map. A hit means an identical fetch is already running; it subscribes to that promise and returns.
- • A miss means it is first: it constructs the fetch promise, stores it in the map *before* awaiting anything, and awaits it.
- • Storing before the first suspension is what makes the check-and-set atomic on an event loop — there is no yield point between the lookup and the write (Event Loops as a Concurrency Model).
- • When the fetch settles, a
finallyremoves the entry — but only if the map still holds *this* entry, so a newer flight is never evicted. - • Every subscriber's continuation runs with the same value or the same error; the promise's broadcast semantics do the fan-out for free.
- • A caller may abandon its wait (an abort signal) without cancelling the shared fetch, because the other subscribers still need it.
- • A misses and creates E1; B and C arrive 5 ms later and find E1; the origin sees one request and three callers are served — the intended schedule.
- • A misses, awaits the cache a second time *before* setting the map entry; B misses in the same window and also finds nothing; two flights start for one key. Set the entry before any await.
- • E1 rejects; A's cleanup deletes the entry; C creates E2; B's cleanup runs unconditionally and deletes E2; the next caller starts a third flight — coalescing degrades to 1:1 exactly when the origin is failing.
- • The same schedule with an identity-checked cleanup: B's delete is a no-op, E2 survives, and the origin still sees one flight at a time.
- • A caller arrives 190 ms into a 200 ms fetch and joins it, receiving data read before it arrived — fine for a profile, a read-your-writes violation if it just wrote.
- • The origin hangs; the shared promise never settles; 100 subscribers wait forever and the map entry is permanent. A deadline inside the flight is not optional.
- • Twenty instances each run their own single-flight; the origin sees twenty concurrent fetches, not one. Correct, and a 20× reduction rather than the 400× the local metric suggests.
- • Guaranteed: at most one fetch in flight per key per process, for as long as the entry exists.
- • Guaranteed: every subscriber receives the same value or the same error — the promise settles once and broadcasts.
- • Guaranteed: a subscriber that gives up does not affect the shared fetch or the other subscribers.
- • NOT guaranteed: freshness. A late joiner receives data read before it arrived, which breaks read-your-writes if that caller just wrote.
- • NOT guaranteed: fleet-wide deduplication. This is a per-process map and nothing more.
- • NOT guaranteed: liveness. If the fetch never settles, every subscriber waits forever and the entry is permanent — put a deadline inside the flight.
- • NOT guaranteed: isolation of failures. One rejection fails every subscriber at once, by design.
- • NOT guaranteed: correctness with a bad key. Under-specify it and you serve one caller's data to another with complete confidence.
- • The map is the coordination point and is uncontended on one event loop; across threads it needs a lock, and that lock is held only for the lookup and insert.
- • The real contention it *removes* is at the origin: N concurrent identical fetches become one.
- • It concentrates risk instead: one slow fetch now holds N callers, so the shared fetch's p99 becomes every subscriber's p99.
- • Settlement wakes every subscriber in the same tick — a small, bounded wake burst that is fine at 100 and worth thinking about at 100,000 (Thundering Herd).
- • Under sustained origin failure with a broken cleanup, contention returns to the un-coalesced level at the worst possible moment.
- • Data leak from an under-specified key: callers coalesce across a boundary they should not cross.
- • No coalescing at all from an over-specified key, with the ratio stuck at 1 and nobody noticing.
- • Cleanup race: an unconditional delete evicts a newer in-flight entry and coalescing degrades to 1:1.
- • Cached rejection: an entry not removed on failure means every later caller joins a promise that is already rejected — a poisoned key until restart.
- • Permanent entry from a fetch that never settles, holding every subscriber indefinitely.
- • Stale read for a caller that joined a flight started before its own write (Optimistic Concurrency Control).
- • Correlated failure: one origin blip fails 100 callers simultaneously, which shows up as a spike rather than a trickle.
- • False confidence at fleet scale: per-process metrics show 100:1 coalescing while the origin sees one flight per instance.
- • Read-heavy caches with hot keys, where a miss is expensive and many callers want the same value at once.
- • Cache-fill paths, as the direct answer to a stampede (Thundering Herd,
cache-stampede). - • Expensive idempotent computations — a rendered page, an aggregate, a model inference — where duplicating the work is pure waste.
- • Rate-limited third-party APIs, where N identical calls consume N units of a budget that one call would have used.
- • Any fan-in where the callers genuinely want the same answer and none of them needs a private one.
- • When the callers do not actually want the same value, and the key was written to make them look like they do.
- • When read-your-writes matters, because a joiner can receive data read before its own write.
- • When failures should be independent — a single rejection failing 100 callers may be worse than 100 independent retries.
- • When keys are naturally unique, in which case the map is overhead that never pays for itself.
- • When the value is cheap to compute, and the coordination costs more than the duplication it removes.
- • Coalescing ratio: origin calls divided by callers. Far below 1 is working; stuck at 1 means an over-specified key or the cleanup race.
- • In-flight map size over time. A monotonic climb is a leak — entries that never settle or never get removed.
- • Subscriber count per flight as a distribution; a long tail is exactly the hot-key concentration this exists to serve (Hot Keys: When Aggregate Metrics Hide a Saturated Node).
- • Shared-fetch latency at p99, because it is now every subscriber's latency, not just the first caller's.
- • Rejection fan-out: errors returned to callers divided by origin errors. It should equal the average subscriber count, and that is your correlated-failure exposure.
- • Origin request rate divided by *instance count* to check the fleet-wide effect, rather than trusting a per-process ratio (A Mutex on Server A Does Nothing About Server B).
- • A shared mutable map with a lifecycle you own: created on miss, removed on settle, and never removed by the wrong owner.
- • The dedup key becomes a contract that must be updated whenever a new input starts affecting the result — and forgetting is a data-correctness bug.
- • The failure policy and the deadline are additional decisions with no defaults, and both are invisible until an incident.
- • It interacts with cancellation: an aborting caller must not cancel the shared work, which is a special case in every cancellation design.
- • Testing requires forcing concurrency and forcing failures together, because the cleanup race needs both.
- • A cache with a short TTL and jitter. If the value can be slightly stale, caching removes most duplicate fetches without any coordination at all (Thundering Herd).
- • Serve stale while revalidating in the background, so there is no simultaneous miss to coalesce.
- • Batching: collect requests for a few milliseconds and issue one combined call, which also deduplicates and additionally amortises across *different* keys.
- • A concurrency limit on the origin, which caps the damage without deduplicating anything — weaker, but it needs no key design (Bounding Concurrency, Semaphores: Counting Permits as a Resource Limit).
- • Precompute and push, so the value is always present and the miss path never runs.
- • Nothing at all, when the fetch is cheap. Single-flight is coordination, and coordination is only worth it when the duplicated work costs more than the map (Concurrency Is Always Bought With Complexity).
Single-flight: N callers, one call
without 64 callers → 64 downstream calls latency 192 ms (queued behind each other) with 64 callers → 1 downstream call latency 60 ms (everyone waits for the leader) failure one attempt, 64 disappointed callers — the blast radius of a single bad call is now N retry the followers cannot retry independently; they only ever saw the leader's outcome
Thundering herd: 10,000 waiters
no mitigation 10K requests in one 100 ms bucket vs a capacity of 90/bucket → 9,640 shed this setting 10K requests spread over one instant → peak 10K/bucket, 9,640 shed jitter sleep(base + random() * window) — decorrelates wakeups; costs a little latency batching one call serves N waiters — cuts the request count, not the wakeup count both are cheaper than the capacity you would otherwise have to buy for one instant per hour
Optimistic concurrency lab
Conflict probability grows with the number of writers touching the same item, each conflict costs a full retry, and retries consume the same cores the successful work needs. Real systems add their own effects — backoff, hot keys, transaction size — but the turnover is real and it is why an optimistic scheme that benchmarked beautifully at low load can collapse at high load rather than merely slow down.
What people believe, and what is true
Single-flight means the origin sees one request.
One per process. Twenty instances means twenty flights, and per-process metrics will happily report a 100:1 ratio while the origin sees twenty concurrent fetches.
Sharing the promise means everything happens once.
The fetch happens once; every subscriber still runs its own continuation. Any cleanup or side effect in that continuation runs once per subscriber — which is exactly the cleanup race.
If the fetch fails, the next caller will just retry.
Only if the entry was removed. A rejected promise left in the map is served to every later caller as an instant failure — the key is poisoned until something clears it.
Go deeper
Overview
A map from key to in-flight promise: the first caller fetches, everyone else awaits the same handle, and the entry is removed when it settles.
Practical
Put every result-affecting input in the key, set the entry before any await, clean up in a finally conditional on entry identity, and put a deadline inside the flight.
Advanced
The join window is a freshness policy, not an implementation detail: a caller joining a flight started before its own write will not see that write. Decide whether read-your-writes matters and bypass or version the key accordingly.
Internals
It works because a promise is a settle-once broadcast handle — safe publication with an intrusive continuation list (Futures & Promises). The map is the only mutable state, and on one event loop the lookup-and-insert pair is atomic because nothing can run between them.