IntegrationsGENERALDATABASE-SPECIFICSCALE-SPECIFIC

Rate Limiting

Deciding which caller has had enough, along which dimension — and why the counter's atomicity is the part that makes it correct.

What actually happens How to build it

The requirement, the obvious build, and why it breaks

Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.

The question

How do I stop one caller from consuming capacity that belongs to everyone, and what exactly am I counting?

The requirement

A single integration partner is sending thousands of requests a minute and every other customer's latency has doubled. We need a limit, and it must not lock out normal users.

The obvious build

Count requests per IP in a dictionary, reset it every minute, and return 429 past the threshold.

Why it breaks

The dictionary is per process. Three instances means three times the intended limit, and it resets on every deploy (Stateless Services).

How it breaks in production

The dictionary is per process. Three instances means three times the intended limit, and it resets on every deploy (Stateless Services).
IP is the wrong dimension for authenticated traffic: an entire office behind one NAT shares a limit, while an attacker with many addresses has none (Rate Limiting in Architecture covers the topology view).
The limiter runs after authentication, so every abusive request has already paid for a token lookup and a database query before being rejected (Authenticate First, or Rate-Limit First?).
X-Forwarded-For is taken at face value, so the client chooses its own identity and the limit is bypassed with one header (The Trust Boundary).
All endpoints share one limit, so a cheap health check and a report that runs for seconds count the same.
The limiter's store becomes unavailable and there is no defined behaviour, so it either fails open (no limit at all) or fails closed (a total outage) by accident rather than by decision.

Requirement→API Contract→Application Logic→Data Access→External Deps→Concurrency→Failure→Security→Observability→Deployment→Scale

What is actually happening

Rate limiting answers three separate questions and they are frequently conflated: who is being limited (the dimension), how much is allowed (the policy), and how the count is kept (the algorithm and the store) (Rate Limit Algorithms).
The dimension is the design decision with the most consequences. Common ones: IP, authenticated user, API key, tenant/organisation, endpoint, and a global limit for the service as a whole. Each protects against something different and each has a specific bypass.
Dimensions compose. A real policy is usually several limits applied together — per key, per key per endpoint, per tenant, and a global ceiling — where the request must pass all of them.
Not all requests cost the same. Cost-weighted limiting charges a request a number of units proportional to its expense, which is what makes a limit meaningful when one endpoint is thousands of times heavier than another (Quotas vs Rate Limits in API Design distinguishes short-term rate from long-term quota).
Placement matters. A limiter at the edge protects the whole service including its authentication cost; a limiter in the application knows the tenant and the endpoint cost. Most systems need both (Middleware Ordering Is a Correctness Decision).
Across instances, the counter is shared state, and its atomicity is the correctness property. A read-then-write increment loses updates under concurrency, and the limit is exceeded precisely during the burst it exists to contain (Atomic Operations).
A rejection needs to be actionable: 429, a Retry-After, and headers telling the client its limit, remaining allowance and reset time. Without them a well-behaved client cannot behave well (Status Codes From the Server's Side).

What are you counting, and per what?

Every rate limit has a dimension, and choosing the wrong one produces a limiter that is simultaneously too strict for real users and trivially bypassed by the traffic it was meant to stop. The table is the design conversation: pick the dimensions that match the threats, and know each one's evasion.

Real policies compose several rows. A public API typically limits per key, per key per endpoint, per tenant, and globally — and reports which limit was hit so the client can act on it.

Dimension	Protects against	Evasion / weakness	Where it belongs
IP address	Anonymous abuse, scraping, unauthenticated floods	NAT shares one identity; attackers rotate addresses cheaply	Edge or proxy, before authentication
Authenticated user	One account consuming shared capacity	Useless before login; requires identity to be established first	Application, after authentication
API key	A misbehaving integration	Only as good as key hygiene; one key across many callers hides them (API Keys)	Application or gateway
Tenant / organisation	One customer degrading others in a shared system	Needs tenant resolved early and trustworthily (Tenant Isolation)	Application, from the authenticated context
Endpoint	One expensive operation dominating capacity	Ignores that callers differ; a per-endpoint limit is shared by everyone	Application, combined with an identity dimension
Cost units	Cheap and expensive calls counted equally	Requires a cost model per operation, and clients find it opaque	Application, where the cost is known
Global	The sum of well-behaved callers exceeding capacity	Blunt: sheds legitimate traffic without discrimination	Edge, as a backstop ceiling (Resource Limits)
Account + IP together	Credential stuffing from many IPs, and one IP against many accounts	Neither alone is sufficient; both together is the standard for login	Authentication endpoints specifically

The counter has to be atomic, and where it lives decides how

DATABASE-SPECIFICShown for two stores because the invariant is portable and the mechanism is not. A key-value store with only conditional writes needs a compare-and-set retry loop; a store with no cross-key atomicity cannot enforce a policy spanning several keys at once, which constrains which composed policies are even expressible.

The correctness of a distributed rate limiter reduces to one property: the check and the increment must be a single indivisible operation. A read, a comparison in application code, and a write is three operations, and a burst of concurrent requests will all read the same value and all decide they are under the limit.

This is GENERAL as a principle and entirely store-specific as an implementation. The invariant does not change; what changes is which primitive provides it and what that primitive costs.

A fixed-window counter across instances

Read, decide, write

const n = await store.get(key)
if (n >= limit) return reject()
await store.set(key, n + 1)
// Three operations. Twenty concurrent requests all read 9,
// all decide 9 < 10, and all write 10.
// The limit is exceeded exactly during a burst.

One atomic operation

// Redis: INCR returns the new value atomically.
// The first caller in a window sets the expiry.
const n = await redis.incr(key)
if (n === 1) await redis.expire(key, windowSec)
if (n > limit) return reject({ retryAfter: await redis.ttl(key) })

// Postgres: the arithmetic happens inside the statement,
// and the row lock serialises concurrent writers.
// UPDATE limits SET n = n + 1
//  WHERE key = $1 AND window_start = $2
//  RETURNING n;

// Either way: the decision reads the value the store
// produced, not a value that may already be stale.

The atomic version makes the count the store's responsibility rather than the application's, so concurrency cannot interleave between the check and the update. The trade-offs differ sharply by store — Redis gives a cheap single round trip and needs a server-side script once the algorithm needs several steps; Postgres gives durability and transactional consistency and pays a row lock that becomes a contention point on a hot key.

The limiter's own failure modes

A rate limiter is infrastructure in the request path of every request, which makes its own failures unusually consequential. Each row below is a limiter that appears to be working.

When the limiter is the problem

Trigger	Symptom	Cause	Response
Deploy scales the service from one instance to four	Effective limit quadruples. Nothing alerts.	Per-process counters.	Shared store with atomic increment, or explicitly divide the limit by instance count and accept the drift (Stateless Services).
Burst of concurrent requests for one key	Limit exceeded during the burst; correct at rest.	Read-then-write increment.	Single atomic operation, or a server-side script for multi-step algorithms.
Limiter store unavailable	Either no limiting at all, or every request rejected.	No decided behaviour for store failure.	Decide per endpoint and implement it: fail-closed for login, fail-open with a local fallback for read paths (Fail Open vs Fail Closed in Security).
Client sends its own `X-Forwarded-For`	Limit trivially bypassed; per-IP metrics are fiction.	Header trusted without a trusted-proxy chain.	Only accept the header from proxies you control; parse from the correct end (The Trust Boundary).
Partner's nightly bulk sync	A legitimate integration fails every night at the same time.	One rate limit for both interactive and bulk usage.	Separate policy or a burst allowance for bulk endpoints (Rate Limit Algorithms).
Client retries every 429 immediately	Rejections increase total load rather than reducing it.	No `Retry-After`, or a client ignoring it.	Always send `Retry-After`; escalate to longer blocks for clients that ignore it (Backoff and Jitter).
Limiting applied after authentication	Abusive traffic still costs a token verification and a database read each.	Middleware ordering.	Coarse limit before authentication, identity-aware limit after (Authenticate First, or Rate-Limit First?).

How to build it

Most important first.

Choose the dimension from what you are defending. Credential stuffing is per-IP and per-account; a noisy integration is per-API-key; a multi-tenant fairness problem is per-tenant; protecting a specific expensive operation is per-endpoint (Multi-Tenancy).
Apply several limits together and reject on the first that fails, reporting which one it was.
Place cheap, coarse limits before expensive work — before authentication where possible — and precise, identity-aware limits after (Authenticate First, or Rate-Limit First?).
Derive the client identity from a trusted source. Only trust X-Forwarded-For from a proxy you control, and parse it from the correct end (The Request Lifecycle).
Weight by cost where endpoints differ materially, so one search request can charge what fifty key lookups charge.
Return 429 with Retry-After and limit headers, and document the policy. A limit clients cannot see is a limit they will hit repeatedly (The Rate-Limit Contract in API Design).
Decide fail-open versus fail-closed explicitly for a limiter-store outage and write it down. Fail-open preserves availability and removes protection; fail-closed preserves protection and creates an outage. The right answer differs between a login endpoint and a product listing (Fail Open vs Fail Closed in Security).
Keep a global ceiling as a backstop regardless of per-caller limits, so the sum of many well-behaved callers cannot exceed what the service can serve (Resource Limits).

What can go wrong

Failure modes

In-memory counters on a multi-instance deployment: the effective limit is the configured one times the instance count, and it resets on deploy.
Non-atomic increments, so a burst of concurrent requests all read the same value and all pass (Backend Races).
The limiter store as a new single point of failure with no defined degraded behaviour.
A limit low enough to break legitimate bulk usage, discovered by a customer's nightly sync failing.
A limit high enough to be decorative, discovered during an incident.
Limiting after the expensive work, so the rejection consumes the resource it was protecting.
Retry-After ignored by clients you do not control, so rejections turn into more traffic (Retries).
A per-user limit applied to an unauthenticated endpoint, where every request is the same anonymous user and the limit is effectively global.

What can race

Concurrent requests for one key incrementing the same counter: with a read-then-write the limit is exceeded exactly during a burst, which is the only time it matters (Backend Races).
The window boundary and an increment racing, so a request is counted into a window that has just reset (Rate Limit Algorithms).
Two instances both refilling a token bucket from a shared store, double-crediting tokens unless refill and consume are one atomic operation (Atomic Operations).
A limit change deployed while counters exist under the old policy, so callers are evaluated against two policies for a window.

Security

Rate limiting is a primary defence for authentication endpoints. Without a per-account and per-IP limit, password guessing is bounded only by your capacity (Credentials and Password Handling, Rate Limiting as a Security Control in Security).
Limits keyed on something the client controls are not limits. Header-derived identity, unauthenticated user ids and client-supplied tenant ids are all bypasses (The Trust Boundary).
Rejections can leak information: a different response for "rate limited" on an existing versus non-existent account is an enumeration oracle.
Consider limiting per account and per IP for authentication, because each alone has an obvious evasion — many IPs against one account, or one IP against many accounts.
A limiter that fails open under load is exactly the condition an attacker will try to create (Defence in Depth).

Misreads

"Rate limiting is the same as a quota." A rate limit bounds requests per short window; a quota bounds consumption over a billing period. Different enforcement, different response, different customer conversation (Quotas vs Rate Limits in API Design).
"Rate limiting is the same as a bulkhead." Rate limiting counts requests per unit time; a bulkhead bounds concurrent in-flight work. A dependency that becomes ten times slower breaks the second without touching the first (Bulkheads).
"Limit by IP." Fine for anonymous traffic and wrong for authenticated traffic, where NAT shares an identity and attackers do not.
"We have a limiter, so we are protected." Only along the dimension you chose, only if the counter is atomic, and only if the identity cannot be forged.
"429s are bad." A 429 is the system working. Sustained 429s for legitimate users mean the limit or the plan is wrong; occasional ones mean a client should back off (Backoff and Jitter).
"Put it in the application." Application-level limiting still spends application capacity on requests it rejects. Coarse limits belong further out (Middleware Ordering Is a Correctness Decision).

Operating it

How you see it in production

Rejections per dimension and per policy, so "which limit fired" never requires reading code.
The distribution of consumption against the limit per caller. Callers consistently near the ceiling are about to become support tickets.
Top consumers by key and by tenant, which is how a noisy integration is identified in minutes rather than hours (Hot Keys: When Aggregate Metrics Hide a Saturated Node in Performance).
Limiter store latency and error rate, plus a counter for how often the fail-open or fail-closed path was taken. That path running is an incident even when nothing else looks wrong.
Rejection rate on authentication endpoints as a security signal, not only a capacity one (Suspicious Login Detection in Security).

What changes at 10x and 100x

At small scale a per-instance limiter is approximately correct because there is one instance. It stops being correct at the second one, silently.
At high request rates the limiter becomes a hot path: a network round trip per request to a shared store adds latency to everything and makes the store a capacity concern of its own.
Common mitigations trade exactness for throughput — local counters synchronised periodically, or a local allowance drawn in batches from a shared budget. Both mean the limit is approximate, which is usually fine and must be a stated decision.
Hot keys concentrate load: one tenant's counter can become a single contended item in the store (Hot Keys: When Aggregate Metrics Hide a Saturated Node in Performance).
At very large scale, the edge or CDN is the right place for coarse limits, because rejecting at the application still consumes application capacity (Load Balancing, From the Backend's Side).

What this costs

A shared store makes the limit correct across instances and adds a dependency and a round trip to every request.
Local approximate limits are fast and allow overshoot proportional to instance count. That overshoot is acceptable for fairness and not for anything security-critical.
Strict limits protect capacity and generate support load from legitimate users near the boundary.
Cost-weighting makes limits meaningful and makes them harder for clients to reason about and for you to document.
Multiple composed dimensions give precise control and multiply the number of policies to maintain and explain.

Where this applies

Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.

GENERALDimensions, composition, placement and the need for an actionable rejection apply to every backend.
DATABASE-SPECIFICA distributed limiter is only correct if the store makes check-and-increment atomic, and what that requires is store-specific. An in-memory data store with single-threaded command execution gives atomic increment directly and needs a server-side script for multi-step algorithms such as a token bucket; a relational database needs a single UPDATE with the arithmetic inline, or SELECT ... FOR UPDATE, and pays a row lock for it; some managed stores offer conditional writes with atomic counters but no multi-key atomicity, which rules out policies spanning several keys. The invariant is identical everywhere and the implementation is not.
SCALE-SPECIFICBelow a few instances, per-instance counters are approximately the intended limit and cost nothing. The shared store becomes necessary when instance count makes the multiplication material, and approximate local counting becomes necessary again when the round trip per request costs too much.

Where the depth lives

This domain teaches the application-side mechanism and hands the rest off.

API Designrate-limiting-contract quotas-vs-rate-limits status-codes error-model

Databaseredis-data-structures

Architecturerate-limiting api-gateway

Securityrate-limiting-security fail-open-vs-fail-closed password-attacks

Performancehot-keys saturation concurrency-limits

Concurrencyatomics compare-and-swap local-lock-not-distributed

Agenticbudgets-limits-termination

Domains that do not exist yet

— Distributed Systems — enforcing a global invariant with a shared counter, and what it costs to make that counter both correct and fast.