A report captured a state that never existed
A scheduled report occasionally produces a figure that matches no version of the data. It happens perhaps once a month, always in the same half-hour, and the run it read from was green.
What you would do first
Answer before revealing anything. The value of the exercise is entirely in committing to a diagnosis you can be wrong about.
- 1Correlate the anomalous report times with the publish job's write timeline. A consistent overlap window is the whole diagnosis.
- 2Check whether the publish is one atomic operation or several statements, and whether consumers can observe the intermediate states.
- 3Reproduce deliberately in a non-production environment by reading between the writes.
- 4Look for other consumers reading on their own timers, since they are exposed to the same window and may not have noticed.
What is actually going on
The trap
The fix that looks right. Read it even if you got the answer — especially then.
Move the report to run half an hour later. The overlap disappears for now, the publish window is unchanged and will drift as the job grows, and the actual defect — that consumers can observe a half-written dataset — is now hidden behind a schedule that somebody will change without knowing why it was set.