One stage of the Spark job takes far longer than the others. Why?
Whether the candidate reasons about partitions, shuffle and skew rather than about cluster size, and whether they can explain why more workers changed nothing.
The situation behind the question
Interviewers ask this because it happened to them.
A nightly job has six stages. Five finish promptly and the sixth runs for hours. The cluster has already been doubled once with no effect, and the stage that is slow is a join followed by an aggregation.
A strong answer
Flags
Green flags
- Reasons about shuffle and skew, and knows a stage boundary is where data moves across the network.
- Asks whether it is one task or all tasks before proposing anything.
- Knows salting must be applied to the hot key specifically, and that it costs a combining stage and a worse layout for queries filtering on the original key.
- Looks at per-task metrics rather than at job-level ones.
Red flags
- "Spark makes any query fast." The engine parallelises work; it does not change the distribution of the data, and a skewed key produces a serial tail regardless of engine.
- "Just add more workers." A straggler is one task on one worker; extra workers sit idle while it finishes.
- Proposes caching or increasing memory without establishing whether the problem is skew, spill, or shuffle volume.
- Cannot explain what a shuffle actually does or why a wide transformation forces one.
Follow-ups
Where the conversation goes if the first answer holds up.
- The join is against a dimension that has grown. What changes about whether a broadcast is still viable?
- You salt the key. What have you made worse, and for which queries?
- How would you have caught this before it became a runtime problem?