What is the difference between data parallelism and model parallelism, and which one do we need?
Answer it out loud before you open anything. The value of the flags below is in comparing them to what you actually said — including whether you asked about the data before naming a model.
A team trains a ranking model on one GPU in 26 hours. They are given eight GPUs and expect training in about three hours. Their first attempt takes nine hours and produces a slightly worse model. Separately, a larger model they want to try does not fit in one GPU's memory at all.
What it is really testing
Whether the candidate can distinguish the two problems — too slow versus too big — and match each to the parallelism that addresses it, and whether they understand why eight GPUs did not give eight-fold speed-up: communication, the batch size and learning rate coupling, and the input pipeline.