Here are training and validation curves. Is the model overfitting, underfitting, or something else — and what would you do?
Answer it out loud before you open anything. The value of the flags below is in comparing them to what you actually said — including whether you asked about the data before naming a model.
A team training a document classifier is deciding whether to buy more labelled data at a significant cost per label. They have a plot of training and validation loss against training-set size, and another against epochs. The tech lead's proposal is "more data always helps".
React to this
Say what you would question, what you would trust, and what you would need to know first.
Learning curves — document classifier (illustrative)
Against training-set size (validation loss, training loss):
1k examples: 2.10 0.15
5k examples: 1.62 0.21
20k examples: 1.31 0.28
50k examples: 1.19 0.33
Against epochs at 50k (validation loss, training loss):
ep 1: 1.40 1.10 ep 5: 1.19 0.61 ep 10: 1.22 0.40 ep 20: 1.35 0.22 ep 30: 1.48 0.14
Proposal: purchase 200k more labels.What it is really testing
Whether the candidate can read the two curves as evidence about bias and variance and turn that reading into a decision about data, capacity and regularisation — including the case where more data will not help and the case where the gap is not overfitting at all.