How do you deduplicate a stream, and what does your method miss?
Whether the candidate treats duplicates as normal transport behaviour, chooses the right key, and can name the duplicates their approach will not catch.
The situation behind the question
Interviewers ask this because it happened to them.
A payment provider redelivers webhooks after an outage on their side. Revenue for the affected day is high. Nobody escalated, because the number went up.
A strong answer
Flags
Green flags
- Expects duplicates and late data as normal conditions of the transport.
- Deduplicates on the producer's identifier, and can explain why a locally generated one is useless here.
- States the window and what it costs in state.
- Names the duplicate class the method cannot catch, rather than presenting deduplication as solved.
Red flags
- Deduplicates on an ingestion id or a row hash that includes an ingestion timestamp, producing a test that can never fail.
- Assumes the broker or the provider guarantees single delivery.
- Cannot say how long the deduplication state must be kept or what happens past that.
- Treats an inflated number as less urgent than a shortfall.
Follow-ups
Where the conversation goes if the first answer holds up.
- The provider retries at their end and issues a new event id. What now?
- How wide should the deduplication window be, and what tells you?
- You are deduplicating in batch rather than in a stream. Does anything change?