Nothing in the schema changed and the metric moved. What happened?
Whether the candidate knows that meaning can change without structure changing, and whether they can name the mechanisms and the checks that would catch them.
The situation behind the question
Interviewers ask this because it happened to them.
A conversion metric stepped up on a Tuesday. There was no schema change, no deploy in the data platform, no failed task, and the source team says nothing changed on their side either.
A strong answer
Flags
Green flags
- Understands schema evolution as including meaning, not only structure.
- Names the
ELSEbranch absorbing a new enum value, which is the single most common instance. - Wants a distribution check on categorical columns, not only null and type checks.
- Looks outside the data platform for the deploy that caused it.
Red flags
- "Schema changes are easy if the column still exists." The column existing is exactly the condition under which this failure happens.
- Concludes it must be a genuine business change because no check failed.
- Cannot name a check that would have caught a new enum value.
- Blames the BI tool without checking the distribution of the underlying column.
Follow-ups
Where the conversation goes if the first answer holds up.
- How would you build a check that catches a new enum value on the day it appears?
- The change is legitimate and permanent. What do you do about the historical comparison?
- Who should have told you, and what mechanism would have made that happen?