← PracticeAdvancedSync

A Worker Asleep on a Full Queue

Pull up the evidence one item at a time, commit to a diagnosis, and only then see the schedule that actually ran.

What was reported

Roughly one deploy in thirty, one of our four ingest workers comes up and never processes anything. It does not crash, does not log, does not appear unhealthy — the process is alive and the other three workers carry the load, so we only noticed because throughput was 25% low for a week. Restarting the pod always fixes it. We cannot reproduce it locally, and it happens more often on the faster hosts.
1std::mutex m;
2std::condition_variable cv;
3std::deque<Job> jobs;
4 
5void producer(Job j) {
6  { std::lock_guard<std::mutex> g(m); jobs.push_back(std::move(j)); }
7  cv.notify_one();
8}
9 
10void consumer() {
11  while (running) {
12    std::unique_lock<std::mutex> g(m);
13    cv.wait(g);                 // ← no predicate: waits for a signal, not for a condition
14    Job j = std::move(jobs.front());
15    jobs.pop_front();
16    g.unlock();
17    process(j);
18  }
19}

Evidence

Nothing here is labelled as relevant. Some of it is not.

What is actually happening?