Choosing a Runtime
A decision made once, changed rarely and paid for daily — decided by workload shape, ecosystem and who is on call, not by benchmarks.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
How do I choose a runtime for a new service without arguing about benchmarks?
A new service is being started. Someone has to pick a language and runtime, and the choice will outlive most of the people in the room.
Pick the fastest one, or the one with the best benchmark numbers for the framework we like.
Framework benchmarks measure a hello-world handler. Your service spends most of a request waiting on a database, so the benchmark measures the part that does not matter (Why Is My API Slow?).
- Framework benchmarks measure a hello-world handler. Your service spends most of a request waiting on a database, so the benchmark measures the part that does not matter (Why Is My API Slow?).
- The fastest runtime with no mature client for your database or your cloud provider costs more in integration work than it saves in CPU.
- A team of four who have never operated the chosen runtime now owns its production failures, and the debugging tools are unfamiliar precisely when it matters.
- Adding a second runtime to an organisation duplicates every cross-cutting concern: logging format, metrics, tracing, deploy pipeline, base images, security patching (Structured Logging).
- The workload was misclassified: a service chosen as I/O-bound turns out to do heavy per-request computation, and the concurrency model is the wrong shape for it (Blocking the Event Loop).
What is actually happening
- The runtime decides what one waiting request costs and what one badly behaved request damages. Those two properties generate most of the operational differences (Backend Runtime Models).
- Most backend requests are dominated by waiting: a database query, a cache lookup, an external API. Local execution speed is a small share of a typical request, which is why runtime benchmarks correlate poorly with service latency.
- Ecosystem maturity — database drivers, cloud SDKs, auth libraries, observability integrations — usually decides more delivered performance than the runtime does, because a mature driver with proper connection pooling beats a fast language with a naive one.
- Operability is a runtime property: the quality of profilers, heap dumps, thread dumps, tracing integration and crash diagnostics differs enormously and is what you will use during an incident (Debugging a Backend in Production).
- Team knowledge compounds. A runtime the team knows produces better services than a theoretically superior one they do not, because most production problems are diagnosed by pattern recognition.
- Organisational cost is real: each additional runtime multiplies base images, dependency scanning, deployment templates, on-call knowledge and hiring surface.
- Some workloads genuinely constrain the choice: heavy CPU per request, hard tail-latency requirements, machine learning libraries, or an existing codebase that the new service must live next to.
The questions that actually decide it
A runtime argument becomes tractable when it stops being about speed and starts being about fit. These five questions, answered honestly for the specific service, settle most cases — and they are ordered by how often they turn out to be decisive.
Notice that only one of them is about performance, and even that one is about workload shape rather than about which runtime wins a benchmark.
- 11. What does a request spend its time doing?
Splits I/O-bound from CPU-bound, which is the axis the concurrency models actually differ on.
fails by Being guessed rather than measured, usually by assuming the service is like the last one.
- 22. How many concurrent connections, and how long-lived?
Decides whether cheap waiting matters. Thousands of idle websockets is a different problem from a hundred short requests.
fails by Confusing requests per second with concurrency; they are related by duration, not equal (Little's Law as Working Intuition from Performance).
- 33. Does the ecosystem cover our dependencies well?
Database driver and pooling quality, cloud SDKs, auth, OpenTelemetry.
fails by Being checked after the decision, when a missing or immature driver is discovered.
- 44. Can this team operate it at 3am?
Profilers, heap dumps, stack traces, and prior experience with all three.
fails by Being dismissed as a people problem. It is the most reliable predictor of incident duration there is.
- 55. What does adding it cost the organisation?
Base images, pipelines, patching, hiring, on-call knowledge, mobility between teams.
fails by Being invisible to the team making the choice and paid by everyone else.
If steps 1 and 2 do not produce a constraint, the decision belongs to steps 3, 4 and 5 — which almost always means using what the organisation already runs.
Where each model fits, without ranking them
This table is a fit map, not a leaderboard. Every row is a real production choice made well by many teams, and every row has a workload that makes it the wrong one.
The column that matters most in an incident is the last one: what happens when a request misbehaves. That is the property you will be living with.
| Runtime shape | Fits | Struggles with | One bad request causes |
|---|---|---|---|
| Node / single loop per process | Many concurrent I/O-bound requests; websockets and SSE; shared language with the frontend | CPU-heavy work in-process; using many cores without multiple processes | Correlated latency across every endpoint in that process (Blocking the Event Loop) |
| CPython, sync or threaded workers | I/O-bound services; the data, scientific and ML ecosystem; rapid delivery | CPU parallelism inside one process (GIL); very high concurrency per worker | One worker occupied; capacity falls by 1/N (Python Runtime Models) |
| CPython, async (ASGI) | High-concurrency I/O when the whole path is async | Any synchronous library on the request path | That worker's entire loop stalls |
| Go / lightweight tasks | High concurrency with blocking-style code; uses all cores by default | Workloads needing hard real-time tails; ecosystems it has not reached | One core's share; gradual degradation |
| JVM / thread per request | Mature ecosystem, strong tooling, long-running high-throughput services | Memory footprint; start-up and warm-up for short-lived processes | One thread, plus lock contention if it holds one |
| C++ / Rust, thread per core | Request-path infrastructure where tail latency is the product | Development speed; ecosystem breadth; memory-safety risk in C++ | One core's worth of connections (C++ Backend Services) |
The argument that is not about runtimes
Most runtime disagreements are proxies for something else: an unbounded query, a missing index, a chatty dependency, an absent timeout. Those follow you across any rewrite, and a rewrite is an expensive way to discover that.
The useful discipline is to require evidence that the runtime is the constraint before treating it as one. That evidence is a profile, a saturation metric and a latency breakdown — the same three artefacts you would need to fix the problem without changing anything.
"Framework X does 200k requests/second in the benchmark and ours does 8k. We should migrate." // The benchmark handler returns a constant string. // Our handler makes 3 database queries and one // external call, and 92% of the request is waiting. // The ceiling is the dependency, not the runtime.
"Our p99 is 900ms. The breakdown is:
620ms waiting on the orders query
180ms waiting on the pricing service
40ms JSON serialization
60ms everything else, including runtime overhead
So: index the query, add a timeout and a cache to
pricing, paginate the response. If after that the
runtime overhead is the largest remaining term,
we will have an actual runtime argument."A latency breakdown converts an opinion into an ordered list of fixes. If runtime overhead is 60 ms of a 900 ms request, no runtime change can produce more than a small improvement — and the same breakdown tells you exactly which three changes can (Why Is My API Slow?).
How to build it
Most important first.
- Classify the workload first: what fraction of a request is waiting versus computing, how many concurrent connections, how long-lived they are, and how strict the tail requirement is (Computing or Waiting? in Performance).
- Default to what the organisation already runs well. The burden of proof belongs to the new runtime, and it should be discharged with a workload-specific reason.
- Check the ecosystem for your specific dependencies before anything else: database driver quality and pooling behaviour, cloud SDK coverage, auth libraries, an OpenTelemetry integration that works.
- Weigh operability explicitly. Ask what a heap dump, a CPU profile and a stuck-request diagnosis look like in this runtime, and whether anyone on the team has done all three.
- Prototype the risky part rather than the easy part: the highest-concurrency path or the heaviest computation, with a realistic dependency behind it (Performance Testing a Backend).
- Write the decision down with its reasons and its assumptions, so it can be revisited when the assumptions change rather than re-argued from scratch.
- Isolate a specialised runtime to the component that needs it. One native or ML-specific service beside a boring stack is far cheaper than converting the estate (Microservices is a separate decision, and this is not a reason to make it).
What can go wrong
- The wrong model for the workload: an event-loop runtime given CPU-heavy work, or a low-concurrency worker model given tens of thousands of long-lived connections.
- Ecosystem gaps discovered late — an immature driver, a missing SDK, a client library without connection pooling — after the architecture depends on the choice.
- Operational unfamiliarity surfacing during the first incident, when the team needs a profiler they have never run.
- Runtime sprawl: five languages in an organisation of thirty engineers, so every cross-cutting improvement has to be implemented five times.
- The mitigation failing: standardising on one runtime so rigidly that a genuinely constrained workload is forced onto it and pays for the mismatch forever.
- Every runtime has in-process races; only their shape changes. A single-threaded loop races across
awaitpoints, a threaded runtime races on shared memory, and a multi-process one races across processes where no local lock helps (Backend Races). - Choosing a runtime does not remove concurrency bugs — it decides which tools you have for finding them: a race detector, a thread dump, or reasoning about yield points (A Test Strategy Chosen by What Each Layer Can Prove).
- Patch cadence and security-advisory quality differ by runtime and matter more than most benchmark differences (Dependency Security).
- Memory-safe runtimes remove an entire vulnerability class from network-facing code. That is a security argument with real weight for anything parsing untrusted input (C++ Backend Services).
- Ecosystem breadth cuts both ways: a large package ecosystem means more supply-chain surface and more transitive dependencies to keep patched.
- Each additional runtime is another dependency-scanning pipeline, another base image to patch and another set of CVEs to track (The Backend Security Checklist).
- "This runtime is faster." At what? A hello-world benchmark measures the part of a request your service spends the least time in.
- "Async is more scalable, so pick an async runtime." Async is cheaper per waiting request. For a CPU-bound service it adds machinery and removes nothing, and a bounded thread-per-request model is often simpler and just as fast.
- "We can always change it later." You can, and it is a rewrite. Treat the choice as long-lived because it is.
- "The most popular framework is the safe choice." Popularity is a good proxy for ecosystem and hiring, and says nothing about whether the concurrency model matches your workload.
- "One runtime for everything is best practice." It is a good default and a bad rule. A machine learning service, a data-plane proxy and a CRUD API have genuinely different constraints.
Operating it
- Whichever you choose, know its saturation signal on day one — loop lag, busy workers, thread-pool depth — and put it on the first dashboard (The Metrics a Backend Must Emit).
- Check that an OpenTelemetry or equivalent integration exists and works before committing, not after the first incident with no traces (Tracing From the Backend's Side).
- Measure the actual service under a realistic load profile rather than trusting framework benchmarks; the shape of your dependency latency dominates (Performance Testing a Backend).
- Track the operational cost of each runtime in the organisation — incidents, time to diagnose, patching effort — so runtime sprawl is a number rather than a feeling.
- At small scale the choice barely matters technically and matters enormously for delivery speed. Pick what the team knows.
- At 10x, the concurrency model starts deciding your instance count and your connection footprint, and the mismatch cases become expensive (Worker Processes).
- At 100x, per-request CPU cost becomes a fleet-size bill, and specialising the few components on the critical path can be worth a great deal — usually a component, not the estate (C++ Backend Services).
- Standardising on one runtime gives shared tooling, shared knowledge and cheap mobility, and forces a poor fit onto the occasional workload that genuinely needs something else.
- Choosing the best-fitting runtime per service gives local optimality and multiplies organisational cost by the number of runtimes.
- Familiar beats optimal for almost every service and is a bad reason to force a workload with a genuine constraint onto the wrong model.
- Any choice you make is one you will operate for years. Development speed and operability compound; a fixed per-request CPU advantage does not, unless the request rate is very large.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALThe criteria — workload shape, ecosystem, operability, team, organisational cost — are stable across every runtime and every era. The rankings under them are not, which is why this lesson names no winner.
- RUNTIME-SPECIFICConcrete guidance necessarily is: Node suits many concurrent I/O-bound connections and punishes CPU work in-process; CPython suits I/O-bound services and the data and ML ecosystem, and needs processes for CPU parallelism; Go and the JVM use all cores in one process and degrade proportionally under CPU load; C++ and Rust suit request-path infrastructure where the tail is the product.
- SCALE-SPECIFICBelow a few thousand requests per second the choice is dominated by delivery speed and operability. Above that, per-request CPU cost and connection footprint start to appear directly on the bill, and the calculus changes for the components on the critical path.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
- — Programming Languages & Runtime Internals — how a scheduler, a garbage collector and a JIT actually work, below the level this lesson chooses between.