Break the Scheduler

Seven things that go wrong in production, applied one at a time to a running system. Before each one lands you say what it will do to throughput. Most engineers get 'add threads' wrong, and getting it wrong here is considerably cheaper than getting it wrong at three in the morning.

SIMULATED
No number on this page was measured. Latency, throughput, CPU and queue depth all come from one pure model of a worker pool over a fixed number of cores. The model reproduces the behaviour of real systems — work does not overlap on one core, a critical section serialises whatever fraction of the work it covers, a queue whose arrival rate exceeds its service rate grows without bound — but the figures belong to no real machine, runtime or workload. Read the shape of the curve and the direction of the change; never quote the milliseconds.

The system right now

Cores
4
Workers
8
CPU / task
10 ms
I/O / task
20 ms
Critical section
0 ms
Arrivals
150/s
Throughput
150/s
Mean latency
31 ms
CPU utilization
38%
Blocked on the lock
0
Waiting for a core
0.1
Effective parallelism
2.7 / 4
healthyRetiring 150/s at 38% CPU. Headroom remains; the next constraint appears at about 267/s.

"Blocked" and "waiting" are Little's law applied to the two wait terms — arrival rate multiplied by mean wait. When the system has no steady state they are reported as unbounded rather than given a fabricated number.

Break something

Pick an action. You will be asked to predict before it is applied.

What you predicted, and what happened

Newest first. The interesting rows are the ones where you were wrong.

Nothing broken yet

Pick an action above, commit to a prediction, and the model applies it.

The three that catch people

Read these after you have been wrong at least once.

Adding threads
  • On CPU-bound work with no idle cores, extra workers add context switches and nothing else.
  • Throughput stays flat or falls; latency rises because everyone now queues behind more people.
  • The signal is effective parallelism sitting below the core count while worker count climbs.
Holding a lock
  • A critical section is a hard throughput ceiling: 2 ms held means at most 500 passes per second, on any number of cores.
  • CPU utilization can look comfortable while everything is slow — the cores are idle *because* everyone is blocked.
  • This is the one place where the fix is genuinely to make the code do less, not to give it more machine.
Retrying
  • A retry storm multiplies the load that caused the failure, at exactly the moment the system has least capacity.
  • It is the only action here that can take a stable system past the point of no steady state in one click.
  • The fix is not fewer retries in isolation — it is a budget, a backoff and a circuit that opens.