The lake has millions of tiny files. Why does that matter?
Whether the candidate can distinguish work that scales with rows from work that scales with objects, and whether they recognise a problem that adding compute cannot solve.
The situation behind the question
Interviewers ask this because it happened to them.
A streaming ingest writes a file per micro-batch. It has been running for a year. Queries that used to finish comfortably inside the window now do not, and the cluster has already been made larger twice with no improvement.
A strong answer
Flags
Green flags
- Names the per-file overhead explicitly and explains why it is not parallelisable in the same way as scanning.
- Recognises the signature: a job that gets slower while a larger cluster changes nothing.
- Treats compaction as ordinary maintenance with a window and an atomic swap, not as a one-off cleanup.
- Connects the file count back to both the write cadence and the partition key, which are the two things that produce it.
Red flags
- "Just add more workers." The work that dominates is listing and planning over objects, and more workers do not reduce the number of objects.
- Proposes compaction without saying how consumers avoid reading a half-rewritten table.
- Treats it purely as a storage-cost problem, which is the one dimension where tiny files are almost free.
- Cannot say what would produce tiny files in the first place.
Follow-ups
Where the conversation goes if the first answer holds up.
- How would you monitor for this before it becomes a symptom?
- Compaction rewrites files that consumers are reading. How do you publish the result safely?
- The stream needs low latency, so batching before landing costs freshness. How do you decide the trade?