The stream join is missing matches and the state keeps growing
A streaming job joins orders to payments on order id and emits a matched record. Checkpoint size grows steadily, restarts take longer each week, and a meaningful share of matched pairs never appear in the output even though both sides are present in the log.
What you would do first
Answer before revealing anything. The value of the exercise is entirely in committing to a diagnosis you can be wrong about.
- 1Measure the distribution of the gap between the two sides arriving. That distribution, not a preference, determines what the join window must be.
- 2Compare emitted matches against a batch join over the same period from the log. The difference is the drop rate, which nothing in the stream reports.
- 3Look at what state is actually retained: unmatched rows, matched rows kept for deduplication, or both.
- 4Check whether the key distribution is even, because a hot key inflates state on one partition and hides in an aggregate metric.
What is actually going on
The trap
The fix that looks right. Read it even if you got the answer — especially then.
Increase the memory allocated to the job. The restarts stop failing, the state still grows without bound, and the dropped matches are untouched — because they were never a memory problem, and now the growth has more room to hide in.