The Messages That Were Acknowledged and Never Processed
Read what each party saw, commit to a cause, and only then find out which of them was right. The root cause, why the obvious reading was wrong, and the fixes are all held back until you have answered.
The symptom
What was reported, before anyone knew what was happening.
What each party saw, and what each concluded
The evidence, in the form it actually arrived in — several parties, several partial views, several confident conclusions.
| Who | What they could actually see | What they concluded | Verdict |
|---|---|---|---|
| Consumer | A batch of 500 records polled; offset committed immediately; handler invoked asynchronously per record. | Committing early keeps lag low and avoids reprocessing. | ✕ wrong |
| Broker | Offsets advanced to the end of the batch. | The consumer has processed up to here; these records need never be delivered again. | ✕ wrong |
| Kubernetes | A drain; SIGTERM sent; the pod exited within the 30-second grace period. | A clean shutdown. | ✕ wrong |
| On-call engineer | Lag zero, no errors, no DLQ. | The messaging layer is fine; the email provider must have dropped them. | ✕ wrong |
4 of 4 parties reasoned correctly from what they could see and still reached the wrong conclusion. Nobody in this table is careless. Each one acted on complete-looking local information, and the information was local. That gap — between what a node can observe and what is true — is the whole domain, and one of these readings will usually be yours.
Commit before you read on
Consumer lag was zero for the entire window, which normally means healthy. Commit before reading on: explain how zero lag is compatible with 14,000 unprocessed messages, and say what lag actually measures.
Write it down even if you are unsure. An unwritten guess quietly becomes “that is what I thought” the moment you read the answer.