Calling Conventions
Where arguments go, where the result comes back, who is obliged to preserve what, and how the stack must be aligned at the instruction before a call. The part engineers get wrong: moving values into the argument registers is a parallel copy, and emitting the moves in source order destroys an argument.
How do a caller and a callee that were compiled separately agree on where the arguments are?
The program at a call site is a set of live values plus a fixed map from argument positions to physical locations. That map is the representation: not "the arguments", but "argument 0 is in rdi, argument 1 is in rsi, argument 7 is at [rsp+8], the result will be in rax, and these nine registers will be garbage when control comes back". It exists to answer the one question two separately compiled functions cannot ask each other at runtime — where is everything?
The backend may choose any instruction sequence it likes on either side of the boundary, but at the moment the call instruction executes, every value the convention names must be in the location the convention names, the stack pointer must satisfy the convention's alignment requirement, and any callee-saved register the function has written must have been saved somewhere recoverable. On return, every callee-saved register must hold the value it held on entry and the result must be in the return location. Nothing about this is negotiable or provable locally: the code on the other side was compiled from a different translation unit, possibly by a different compiler, and cannot be consulted.
Key points
- A calling convention fixes four things: which locations hold arguments, which holds the result, which registers the callee must preserve, and how the stack is aligned at the call.
- The structure — a few argument registers, a stack overflow area, one return register, a caller/callee-saved split — is universal. The names are target-specific and transfer to nothing.
- Moving values into the ABI registers is a parallel assignment. Emitting the moves in order destroys an argument whenever the moves form a cycle.
- Cycle breaking must rescue the destination that is about to be overwritten, not the source; the plausible alternative is wrong in the same silent way.
- Caller-saved versus callee-saved is a cost split, not a correctness one — but a callee that fails its half corrupts a caller's locals, which is nearly impossible to attribute.
- Alignment is a real requirement with a violent failure mode: a fault in vectorised code several frames away from the function that broke it.
Three conventions, one function signature
__stdcall annotations in old Windows headers exist at all.A calling convention is a small table of decisions, and the decisions are arbitrary in the way that driving on the left is arbitrary — nothing makes one set better, and disagreeing about it is catastrophic. The table below is the whole content of the idea for integer arguments, and the three columns share almost nothing.
Notice what the columns do share: there is always a small set of registers for the first few arguments, always a spill to the stack beyond that, always exactly one integer return register, and always a split of the register file into "the callee may clobber this" and "the callee must give this back". Those four structural facts transfer to every architecture. The register names transfer nowhere.
Our own backend hard-codes the System V column, in SYSV_ARG_REGISTERS, SYSV_RETURN_REGISTER, SYSV_CALLER_SAVED and SYSV_CALLEE_SAVED. That is a deliberate simplification and it is exactly the kind of hard-coding that makes a backend single-target: those four constants are the reason our emitted assembly is meaningless on Windows.
| Aspect | x86-64 System V (Linux, macOS, BSD) | Windows x64 | AArch64 AAPCS |
|---|---|---|---|
| Integer argument registerstarget | rdi, rsi, rdx, rcx, r8, r9 — six | rcx, rdx, r8, r9 — four | x0–x7 — eight |
| Further argumentstarget | Pushed on the stack, right to left | Pushed on the stack, plus a 32-byte shadow space the caller must always reserve | Pushed on the stack |
| Floating-point argumentstarget | xmm0–xmm7, counted separately from the integer registers | xmm0–xmm3, sharing the position count with the integer registers | v0–v7, counted separately |
| Integer returntarget | rax, with rdx for the second half of a 128-bit result | rax only | x0, with x1 for the second half |
| Callee-saved (preserved)target | rbx, rbp, r12–r15 | rbx, rbp, rdi, rsi, r12–r15, and parts of xmm6–xmm15 | x19–x28, x29 frame pointer |
| Stack alignment at the calltarget | rsp ≡ 0 (mod 16) *before* the call pushes the return address | rsp ≡ 0 (mod 16) before the call | sp ≡ 0 (mod 16) at all times, not just at calls |
| Red zonetarget | 128 bytes below rsp are safe from signal handlers, so leaf functions can skip adjusting rsp at all | None. Anything below rsp may be destroyed at any moment | None in the standard; some platforms define one |
| Return addresstarget | Pushed on the stack by call | Pushed on the stack by call | Placed in x30, the link register — on the stack only if the callee saves it |
Moving arguments is a parallel copy
r11 as its scratch register, which is safe in our backend because nothing else is live there at a call site. A production backend cannot assume that: it asks the register allocator for a free register, and if none exists it must spill to a stack slot instead, which turns a two-instruction cycle break into four memory accesses. The algorithm is the same; the resource acquisition is the part we are skipping.Here is the concrete mistake this lesson exists for. The caller has a in rdi and b in rsi, and it wants to call g(b, a). The required end state is rdi = b, rsi = a. Write that as two moves in the obvious order and you get mov rdi, rsi followed by mov rsi, rdi — and after the first instruction rdi no longer holds a, so the second move copies b onto itself. Both registers end up holding b. The program compiles, links, runs, and returns wrong answers.
The bug is that the ABI shuffle is a *parallel* assignment — all destinations receive their sources simultaneously — and machine instructions are sequential. Converting a parallel copy to a sequence is a real algorithm: repeatedly emit any move whose destination nothing else still needs to read, and when only cycles remain, rescue one destination into a scratch register and break the cycle there.
Our backend routes every call through sequenceParallelCopies in src/compilers/sim/ir.ts for exactly this reason, using r11 as the scratch register because it is caller-saved and unused by the argument sequence. The same helper resolves phi nodes in [[out-of-ssa]], which is the same problem wearing different clothes — phis are also a parallel assignment on an edge. And the test suite verifies it by *simulating* the emitted move sequence over an initial register state and asserting the final state: scripts/compilers-sim.test.ts checks a two-cycle swap and a three-cycle rotation and would fail on the naive ordering.
; wanted: rdi <- rsi, rsi <- rdi (simultaneously) mov rdi, rsi mov rsi, rdi ; reads the rdi we just overwrote call g
; sequenced with a scratch register mov r11, rdi ; rescue the value about to be destroyed mov rdi, rsi mov rsi, r11 call g
A parallel copy may be emitted in source order only when no move's destination is another still-pending move's source — that is, when the move graph is acyclic. Under that condition each destination is dead before it is written and sequential execution produces the parallel result. Our sequencer establishes exactly this by repeatedly selecting a move nothing else reads from.
Whenever the moves form a cycle, which the two-argument swap above is the smallest instance of. Then no ordering works and a scratch location is required. Note also that the rescue must save the *destination* of the victim move, not its source: saving the source emits tmp <- b; a <- tmp; b <- a, whose last move reads the already-overwritten a, and both registers end up holding b — the same wrong answer by a subtler route.
Caller-saved and callee-saved are one question from two sides
Somebody has to save a register whose value must survive a call, and the convention decides which side. A caller-saved (volatile) register is one the callee may destroy freely, so a caller with a live value there must save it before the call and reload it after. A callee-saved (non-volatile) register is one the callee must give back unchanged, so a callee that wants to use it must push it in the prologue and pop it in the epilogue.
Neither choice is better in general, and the split exists because the right answer depends on how the register is used. A value used once around a single call is cheaper in a caller-saved register — one save, one reload. A value used twenty times across a loop containing a call is far cheaper in a callee-saved register — one push and one pop for the whole function, and free access at all twenty uses. Convention designers split the register file roughly in half so the allocator has both options, and [[register-allocation]] chooses between them per value.
The failure mode is asymmetric and worth internalising. A caller that forgets to save a live value across a call gets a corrupted local — bad, but local. A callee that clobbers a callee-saved register and does not restore it corrupts *someone else's* local, in a frame it has never heard of, and the wrong value appears after the call returns in code that did nothing wrong. That is the single hardest class of ABI bug to attribute, and it is why hand-written assembly and JIT-generated stubs are where it almost always comes from.
- Caller-saved: cheap for short-lived values, costs a save and a reload at every call the value spans.
- Callee-saved: cheap for long-lived values, costs a push and a pop on every invocation of the function whether the register was needed on that path or not.
- A leaf function — one that calls nothing — has no callee-saved obligation to worry about and can use every volatile register freely. Compilers exploit this heavily.
- A function that uses no callee-saved registers and calls nothing may also skip the frame pointer entirely, which is
[[stack-frame-layout]].
Alignment and the red zone
sp to be 16-byte aligned at every instruction, not merely at calls, which is stricter than either.System V requires rsp to be a multiple of 16 immediately before a call executes. The call then pushes an 8-byte return address, so on entry to the callee rsp is congruent to 8 modulo 16 — which is why a typical prologue's push rbp restores the multiple of 16 and why an odd number of pushes in a hand-written function is a bug rather than a style choice.
The requirement is not decoration. It exists so that a callee may use aligned 16-byte vector loads and stores on its stack slots without checking, and so that the ABI can guarantee that a long double or a __m128 in a frame is naturally aligned. Violate it and the symptom is a general-protection fault inside a function that has nothing to do with the caller — typically in memcpy or in a vectorised loop, several frames deep, on some inputs only. Our allocator's [[spilling]] lesson names the same hazard from the other end: enlarging a frame without preserving alignment breaks callees.
The red zone is the other System V oddity. The 128 bytes *below* rsp are guaranteed not to be touched by signal handlers or interrupts, so a leaf function may use them as scratch space without adjusting rsp at all — saving two instructions in a very common case. This is exactly why kernel code is compiled with -mno-red-zone: an interrupt in kernel mode pushes onto the same stack, which would destroy the region the compiler was told was safe. Windows x64 has no red zone at all, so this optimization simply does not exist there.
1leaf:2 mov qword [rsp-8], rdi ; below rsp — legal only because of the red zone3 mov rax, qword [rsp-8]4 ret ; no sub/add of rsp at all5 6nonleaf:7 sub rsp, 8 ; must reserve properly: a call would clobber the red zone8 mov qword [rsp], rdi ; and this keeps rsp 16-byte aligned at the call below9 call helper10 add rsp, 811 retThe distinction is whether a call executes while the scratch data is live. call pushes a return address at [rsp-8], and the callee then uses everything below that — so anything the red zone was holding is gone. This is the sort of rule that a compiler applies mechanically and a human writing assembly forgets exactly once.
How it works
The steps, in the order the compiler takes them.
- The backend classifies each argument by type into a class — integer, SSE, memory — because the class decides which sequence of registers it consumes.
- Arguments are assigned to registers in order within their class; once a class runs out of registers the remaining arguments are placed on the stack in the order the convention specifies.
- The moves needed to get values into those registers are collected as a parallel copy and sequenced: emit any move whose destination is not still needed as a source, and when only cycles remain, copy one destination to a scratch register, rewrite the moves that read it, and continue.
- The stack pointer is adjusted so that it satisfies the alignment requirement at the moment of the call, counting the return address the call itself will push.
- Any live value in a caller-saved register is spilled before the call or was placed in a callee-saved register earlier by the allocator; the callee correspondingly pushes any callee-saved register it intends to write.
- After the call the result is moved from the return register to wherever the allocator wanted it, and that move is deleted by
[[peephole-optimization]]when the two coincide.
How it breaks
What the engineer observes when it goes wrong — not what goes wrong internally.
- A hand-written assembly routine or a JIT stub clobbers a callee-saved register without restoring it, and a local variable in a caller three frames up changes value across a call it does not control. Nothing crashes and nothing points at the culprit.
- Argument moves are emitted in source order, a cycle destroys one of them, and a function receives the same value for two different parameters — always, deterministically, and only for call sites where the arguments happen to be swapped.
- A function compiled against a header declaring one convention is called through a pointer typed with another, and the arguments arrive in the wrong registers. On x86-64 this reads garbage; on 32-bit x86 it also unbalances the stack and the caller returns to an address that was never a return address.
- The stack is misaligned by a hand-written prologue with an odd number of pushes, and a callee's vectorised
memcpyfaults on a 16-byte aligned store — several frames below the mistake, and only for buffer sizes large enough to hit the vector path. - Code that relies on the red zone is compiled into a context where interrupts use the same stack — kernel code, or a signal handler on a platform that does not honour it — and locals are corrupted only when an interrupt happens to arrive, so the bug is load-dependent and unreproducible.
When it helps
- Reading disassembly: knowing the convention turns an unexplained sequence of
movinstructions before acallinto "the argument shuffle" and an unexplainedpush rbxinto "the allocator wanted a value to survive a call". - Writing anything that crosses a language boundary — an FFI binding, an inline-assembly block, a signal handler, a JIT-emitted stub — where the compiler is not there to enforce the contract for you.
- Debugging a crash whose stack trace makes no sense: a corrupted return address or a caller local that changed across a call are both ABI symptoms, not memory-safety symptoms.
When it hurts
- Reasoning about performance from the argument count. Passing six integers costs nothing extra on System V; passing a seventh costs a stack store, and passing a large struct by value may cost a full copy the signature does not hint at.
- Carrying register names between platforms.
rdias "the first argument" is true on Linux and macOS and false on Windows, and the resulting confusion is entirely self-inflicted.
What it costs
Every one of these is paid by something.
- More argument registers buy fewer memory accesses at every call and cost registers that are then unavailable to the allocator inside the function body — which is why AArch64's eight is comfortable with 31 registers and would be painful with 16.
- A larger callee-saved set buys cheap long-lived values across calls and costs a push and a pop in every function that touches one, including on paths where the register was never needed.
- A red zone buys two instructions in every leaf function and costs the guarantee that anything below the stack pointer is free — which is why the kernel has to compile with it disabled and take the instructions back.
- Standardising a convention at all buys separate compilation and cross-vendor linking, and costs the ability to specialise: a compiler that could see both sides would often pass arguments somewhere cheaper, which is exactly what
[[link-time-optimization]]and internal-linkage functions allow it to do.
What else you could do
What a different compiler or language does instead, and when that is better.
- Custom conventions for internal functions: a compiler that can prove every caller of a function is in the same compilation unit may pass arguments anywhere it likes. LLVM's
fastccand Rust's default unspecified ABI both exploit this, which is whyextern "C"is needed the moment such a function becomes reachable from outside. - Passing everything on the stack, as 32-bit x86 cdecl largely does. Simpler, uniform, and slower — every argument is a memory round trip, which mattered less when there were six usable registers to begin with.
- Register windows, as SPARC used: the caller and callee see overlapping windows of a large register file, so argument passing is free and there is no save/restore split at all. It costs a much larger register file and a very expensive spill when the windows run out.
- A single argument-block pointer, as many syscall conventions and some interpreters use: pack the arguments into a struct and pass its address. Uniform, easy to generate, and it turns every argument access into a memory access.
See it for yourself
The flag, dump or tool that shows you this directly.
- See the shuffle:
clang -O2 -S -o - file.con a function that calls another with its arguments reordered. Themovinstructions before thecallare the parallel copy, and a scratch register appearing there is a cycle break. - See the ABI classification decisions LLVM made:
clang -S -emit-llvm -o - file.cshowsbyval,sretandzeroextattributes on parameters, which are the frontend telling the backend how the convention classified each one. - Read the specification rather than guessing: the System V x86-64 psABI document, Microsoft's "x64 calling convention" documentation, and ARM's AAPCS64. All three are short enough to read in an afternoon and settle every argument.
- Compare conventions on identical source: Compiler Explorer with
--target=x86_64-pc-windows-msvcbeside--target=x86_64-unknown-linux-gnushows the same function using entirely different registers. - Our own tables are
SYSV_ARG_REGISTERS,SYSV_RETURN_REGISTER,SYSV_CALLER_SAVEDandSYSV_CALLEE_SAVEDinsrc/compilers/sim/codegen.ts, and the sequencer they feed issequenceParallelCopiesinsrc/compilers/sim/ir.ts.
Plausible wrong readings
Stated the way a confident engineer states them.
- "Arguments are pushed on the stack." On every 64-bit convention in use the first several arguments are in registers and the stack is the overflow path. The pushing model is 32-bit x86 and has not been the default for twenty years.
- "The calling convention is part of the instruction set." It is not. Windows x64 and System V run identical instructions and disagree about every register role, which is precisely why an object file can link successfully and still be wrong.
- "The moves before a call are just the compiler being inefficient." They are the ABI shuffle, and the ones that look redundant are usually a cycle break. Removing them by hand produces a miscompilation.
- "If it links, the conventions match." Linking matches symbol names. Nothing in an object file records which convention a function expects, so a mismatch is discovered by the program misbehaving.
Misconceptions
The claim, and what is actually true.
Go deeper
The same idea at increasing depth. Stop wherever it stops being useful.
overview
Two functions compiled at different times must agree on where the arguments are. The agreement is a fixed table for the platform: the first few arguments go in named registers, the rest on the stack, the result comes back in one specific register, and some registers must be handed back unchanged. Nobody derives this table; everybody looks it up, and everybody on the platform has to use the same one.
practical
Two rules cover most of what you need. First, when you write assembly or generate code by hand, save every callee-saved register you touch and keep the stack 16-byte aligned at every call — those are the two ways to corrupt someone else's frame. Second, when you see a run of mov instructions before a call that look pointlessly circular, that is the argument shuffle and one of them is breaking a cycle. Do not tidy it.
advanced
The parallel-copy problem is the interesting one because it recurs. Argument shuffles, phi resolution when leaving SSA, and any register-to-register permutation the allocator produces are all instances of "implement a simultaneous assignment with sequential instructions". The standard algorithm treats the moves as a graph where an edge runs from a destination to the source it reads: nodes with out-degree zero can be emitted immediately, and what remains is a set of disjoint cycles, each needing exactly one scratch location — or, on x86, one xchg, which trades a temporary for a slower instruction. Getting the rescue direction wrong is the classic bug, and it is why our test simulates the emitted sequence over a register state rather than comparing it to an expected instruction list: the point is what the moves *do*, not what they look like.
How much this depends on
Nothing in this domain is true of every compiler. These say how much.
rcx, rdx, r8, r9, preserves rdi and rsi, and requires 32 bytes of shadow space; AArch64 AAPCS uses x0–x7, returns in x0, and holds the return address in the link register x30 rather than on the stack.r11 as its cycle-breaking scratch register, and never reserves shadow space or checks alignment. It demonstrates the parallel-copy problem honestly and everything about struct classification not at all.If you were asked this in an interview
- A caller has
ainrdiandbinrsiand wants to callg(b, a). Write the moves, and say what goes wrong if you write them in the obvious order. - What is the difference between a caller-saved and a callee-saved register, and when would an allocator prefer each?
- Why does an object file compiled for Windows x64 link against a System V library and then misbehave, rather than failing to link?
Connections
- Programming Languages & Runtime Internals — Foreign function interfaces and how a managed runtime crosses into native codeA runtime calling into C must build a frame that satisfies the platform convention while also keeping its own invariants — stack maps, safepoints, exception state. The compiler side is this lesson; what the runtime does on either side of that boundary is owned there.