Virtual Memorysystem calltransitionprivilegecostbatching

Why a System Call Costs More Than a Function Call

A function call is a jump and a stack push. A system call changes the privilege level, redirects control to an address you do not choose, and disturbs enough microarchitectural state that the cost outlives the call itself.

Follow the mechanism

Software view, hardware view

The gap between what you wrote and what the machine does is where this whole domain lives.

The question
Why does calling into the kernel cost so much more than calling a function, when both are just a jump somewhere else?
What you wrote
`read()` looks like any other function call. It is a call, it does some work, it returns. Presumably slightly slower because it does I/O, but structurally the same thing.
What the hardware does
A dedicated instruction that raises the privilege level and jumps to a kernel-chosen entry point in one step. The kernel then saves state, validates arguments, does the work, restores and returns through another dedicated instruction. Along the way, pipeline, branch predictors and caches are all disturbed in ways that cost cycles after the call has returned.
Syscall cost drives real design decisions: buffered I/O, batching interfaces, io_uring-style submission queues and user-space networking all exist because crossing the boundary is expensive relative to the work many calls perform. Knowing the cost is per crossing rather than per byte tells you which optimisation actually applies.
Source→Compiler→Instructions→Front End→Execution→Registers→Caches→Memory→I/O→Behavior

The transition, hop by hop

The dedicated syscall instruction does something no ordinary call can: it raises the privilege level and transfers control simultaneously, to an address the kernel installed in advance. That indivisibility is what makes it safe, as Why Kernel Mode Is Actually Privileged covers, and it is also the first source of cost — this is not a predicted branch to a known target, it is an architectural event.

The kernel then has work to do before it can even look at your request. Registers must be saved, the stack switched to a kernel stack, arguments validated — because a user-supplied pointer cannot be trusted and must be checked against the caller's actual mappings before it is dereferenced.

Only then does the requested work happen. Afterwards the whole thing runs in reverse: results placed where the caller expects them, state restored, privilege lowered, control returned. For a call doing substantial work this overhead is noise. For a call reading four bytes, the overhead *is* the call.

privilege raised + control transferredpointers checked against caller mappingsUser codeSyscall instructionKernel entry pointSave state, switch stackValidate argumentsDo the actual workRestore, lower privilegeBack in user code
UserLLMAgentToolDataDecisionHumanGuardrail

The cost that outlives the call

The direct cost — the instructions to transition, save and restore — is only part of the story, and often not the larger part. The indirect cost is that the kernel ran on the same hardware your code was using, and left it in a different state.

Kernel code occupies instruction cache your loop wanted. It touches data that evicts your cache lines. It trains branch predictors on its own patterns. It uses translation entries that displace yours. All of that is paid *after* the call returns, as your code re-warms structures it had already warmed, and none of it appears in a measurement that times only the call itself.

This is why a microbenchmark timing a syscall in a tight loop understates its cost in a real program: the loop keeps the kernel's state warm and never pays the re-warming that a real workload pays. It is a specific instance of the general problem in Every Way a CPU Microbenchmark Lies.

Where the cycles go on a small system call — relative, and heavily dependent on the call and the machine — 1 unit ≈ one predicted function call and returnMICROARCH-SPECIFIC
Predicted function call/return×1
Privilege transition itself×25
+ save, stack switch, validate×60
+ cache, TLB and predictor disturbance×150
Mitigation-heavy configurations×400
Ratios, not times. Absolute latencies depend on the processor, its clock, the memory it is attached to and what else is running — publishing them would be wrong everywhere except one machine. The bars are log-scaled, so each step is larger than it looks.
Predicted function call/return — The baseline: often near-free on a modern core
Privilege transition itself — Architectural event; not a predicted branch
+ save, stack switch, validate — Before the requested work begins
+ cache, TLB and predictor disturbance — Paid after the call returns; invisible to a microbenchmark
Mitigation-heavy configurations — Speculation mitigations add flushing on entry and exit — see Spectre and Meltdown: When Speculation Crossed a Boundary

The design consequence: batch, do not repeat

Because the cost is per crossing and largely independent of how much work the call does, the lever is obvious once stated: cross less often, do more per crossing. Every widely-used I/O optimisation is a version of this.

Buffered I/O exists so that a thousand small writes become one large one. Vectored calls exist so several buffers move in one crossing. Submission-queue interfaces let many operations be queued and completed with few or no crossings at all. Memory-mapped files remove the crossing from the read path entirely, converting it into page faults that the hardware handles.

The corresponding mistake is optimising the wrong side. Making the payload of an already-small syscall smaller changes almost nothing, because the payload was never the cost. Counting crossings before optimising is the whole discipline here, and the counter is trivially available.

One crossing per item
1for (i = 0; i < 10000; i++) {
2    write(fd, &record[i], sizeof(record[i]));
3}
4 
5// 10,000 privilege transitions
6// each one: save, validate, work, restore, plus re-warming afterwards
7// the work per call is trivial; the crossing is not
One crossing per batch
1buffer = pack(records, 10000);
2write(fd, buffer, total_size);
3 
4// 1 privilege transition
5// identical bytes written, identical work done in the kernel
6// the difference is 9,999 transitions that no longer happen

The same bytes reach the same file. What changed is the number of times the boundary was crossed, and since the cost is per crossing rather than per byte, that is the number that governs. This is the mechanism behind buffered I/O, vectored writes and submission-queue interfaces alike.

Key points

  • A syscall raises privilege and transfers control in one indivisible architectural event, not a predicted branch.
  • Before any work happens the kernel saves state, switches stacks and validates user-supplied pointers.
  • The indirect cost — evicted cache, disturbed predictors, displaced translations — is paid after the call returns.
  • Microbenchmarks understate syscall cost because they keep kernel state warm and skip the re-warming.
  • Cost is per crossing, not per byte, so the lever is batching — which is why buffered and vectored I/O exist.

Follow the mechanism

The path through the machine, hop by hop — and the conclusions it invites that are wrong.

  1. 1
    User code → syscall instruction: a dedicated instruction is executed with the call number and arguments in agreed registers.
  2. 2
    Instruction → kernel entry: privilege is raised and control transfers to a kernel-installed address in one step; nothing user-chosen runs privileged.
  3. 3
    Kernel → save and validate: registers are saved, the stack is switched, and user pointers are checked against the caller's mappings before any dereference.
  4. 4
    Kernel → work → return path: the operation is performed, results are placed where the caller expects, state is restored and privilege is lowered.
  5. 5
    User code → re-warm: execution resumes on hardware whose caches, predictors and translations the kernel has disturbed, and the cost of re-warming lands here.
What people conclude from this — wrongly
  • • Timing a syscall in a tight loop and concluding it is cheap — the loop keeps kernel state warm and hides the re-warming cost.
  • • Attributing syscall cost to the I/O it performs; a call that transfers nothing at all still pays the transition.
  • • Optimising bytes per call when the count of calls is the actual variable.
  • • Assuming all syscalls cost about the same; costs vary widely by call, by kernel configuration and by mitigation settings.

Consequences, controls and cost

What it causes
  • • Small frequent calls are dominated by transition overhead rather than by the work they request.
  • • Buffered, vectored and submission-queue interfaces all exist to amortise the crossing, not to move bytes faster.
  • • Memory-mapped I/O avoids the crossing entirely on the read path, converting it into faults the hardware handles.
  • • Syscall-heavy workloads show high kernel-mode time with modest throughput, a recognisable and actionable signature.
  • • Speculation mitigations increased this cost noticeably on affected hardware, changing the calculus for existing designs.
What you can do
  • • Count crossings first — `strace -c` or kernel-time share tells you immediately whether this is worth any effort.
  • • Batch: buffer small writes, use vectored calls for scattered buffers, prefer submission-queue interfaces for high-rate I/O.
  • • Consider memory-mapping for read-heavy access to files, moving the cost from crossings to faults.
  • • Avoid syscalls inside hot loops, including hidden ones — allocation, logging and time queries are common culprits.
  • • Do not micro-optimise the payload of a call whose cost is the crossing; that effort is spent on the wrong side.
How to see it
  • • Count calls by type with `strace -c` — a large count with a small per-call payload is the batching signal.
  • • Compare user-mode against kernel-mode cycles in `perf stat`; a high kernel share with modest throughput points here.
  • • Measure the syscall inside a realistic workload rather than a loop, so the re-warming cost is actually included.
  • • A/B a batched version against the unbatched one at realistic scale; the delta is the transition cost, stated directly.
What it costs
  • • Batching amortises the crossing but adds latency for the first item in a batch and buffering complexity in the application.
  • • Memory-mapped I/O removes crossings but gives up explicit error handling and makes the cost appear as unpredictable faults.
  • • Submission-queue interfaces cut crossings dramatically at the cost of a more complex asynchronous programming model.

Scope

§224 — what these claims are specific to.

What these claims are specific to
  • ISA-SPECIFICThe syscall instruction, the register convention for the call number and arguments, and the return mechanism all differ between x86-64, AArch64 and RISC-V.
  • PLATFORM-SPECIFICAbsolute cost depends heavily on the kernel, its configuration and which speculation mitigations are enabled. Mitigation-heavy configurations can multiply transition cost several times over.
  • MICROARCH-SPECIFICThe indirect cost — cache, predictor and translation disturbance — depends on structure sizes and on how much state the kernel path touches, which differs by machine and by call.

Misconceptions

Claim
“A system call is just a function call into the kernel.”
Reality
It is a privilege transition to a kernel-chosen address, followed by state saving and argument validation. A function call is a predicted branch; this is an architectural event with a long tail.
Claim
“Syscall cost is dominated by the I/O it performs.”
Reality
A call that transfers nothing still pays the full transition. That is precisely why batching identical total work into fewer calls is such a reliable win.
Claim
“My microbenchmark shows syscalls are cheap, so this does not matter.”
Reality
A tight loop keeps kernel caches, predictors and translations warm. A real workload pays to re-warm its own state after every call, and that cost never appears in the loop.
← Why Kernel Mode Is Actually Privileged