Producers write 100k/s, consumers handle 20k/s. Now what?
“A pipeline ingests 100,000 events per second but the consumers process 20,000. The queue grows. What are your options and how do you choose?”
What this tests
- Recognising an unbounded queue as a delayed outage, not a solution
- The full menu: scale consumers, shed, sample, buffer with bounds, slow the producer, batch
- Little's law and oldest-message age as the diagnostic
- Choosing by the value of each event, not by a default
Answers by level
Read the beginner answer first and notice what is missing.
First the arithmetic: 80,000 events per second accumulate, so after one minute the backlog is 4.8 M and the oldest event is a minute old; after an hour it is 288 M and an hour stale. Whether that is acceptable depends on what the events are. A queue buys time, it does not buy capacity.
Options, in order of preference: scale consumers if the bottleneck is CPU and the downstream (usually a database) can absorb 5× the writes; batch so each consumer does 500 events per write; shed or sample low-value events (drop debug telemetry, keep payments); bound the buffer so the producer sees rejections and slows down — TCP-style flow control rather than an infinite queue; prioritise so important events skip the backlog.
The choice is per event class. Analytics can be sampled at 20%. Orders cannot be dropped, so for them you rate-limit at the edge with 429 + Retry-After and scale the consumer path properly.
Green flags · Red flags
- Does the backlog arithmetic and asks whether the events are burst or sustained
- Talks in oldest-message age / lag seconds, not queue length
- Names bounded buffers and producer-side slowdown, not only "add consumers"
- Segments by event value: sample telemetry, never drop orders
- Checks where scaling consumers moves the bottleneck
- "Just make the queue bigger; Kafka retains everything." (a bigger queue is a longer delay before the same outage)
- Scales consumers without asking what they write to
- Treats dropping as unthinkable for every event type
- Never mentions how the producer learns it is going too fast