The Products That Existed Only in Search
Read what each party saw, commit to a cause, and only then find out which of them was right. The root cause, why the obvious reading was wrong, and the fixes are all held back until you have answered.
The symptom
What was reported, before anyone knew what was happening.
What each party saw, and what each concluded
The evidence, in the form it actually arrived in — several parties, several partial views, several confident conclusions.
| Who | What they could actually see | What they concluded | Verdict |
|---|---|---|---|
| Product service | Every write committed to Postgres and an indexing event published afterwards. | The index will reflect this shortly. | ✕ wrong |
| Indexer | Events consumed and applied. Error rate 0.02%, all transient network errors, all retried. | The index is current. | ✕ wrong |
| Search service | Documents returned for queries, with the fields the documents contain. | These products exist. | ✕ wrong |
| On-call engineer | Indexer lag under one second, no dropped events in the broker, no failures in the DLQ. | The pipeline is healthy; the drift must be historic. | ✕ wrong |
4 of 4 parties reasoned correctly from what they could see and still reached the wrong conclusion. Nobody in this table is careless. Each one acted on complete-looking local information, and the information was local. That gap — between what a node can observe and what is true — is the whole domain, and one of these readings will usually be yours.
Commit before you read on
The pipeline had no dropped messages and no failed writes, yet 4,000 documents disagree with the database. Commit before reading on: name at least two mechanisms that produce drift in a pipeline where nothing ever fails.
Write it down even if you are unsure. An unwritten guess quietly becomes “that is what I thought” the moment you read the answer.