intermediateABI
What is a calling convention, and what happens if two pieces of code disagree about one?
Whether the candidate knows there is a contract there at all, and can predict the failure mode. The discriminator is that ABI mismatches do not produce clean errors, and a candidate who expects one has never debugged a real one.
What a strong answer covers
- It is the contract between a caller and a callee that neither can see the other to negotiate: which registers carry the first arguments, where the return value goes, how the stack is aligned at the call, who cleans up, which registers the callee may clobber and which it must preserve, and how aggregates and variadic arguments are passed. On x86-64 System V the first six integer arguments go in specific registers and the return comes back in
rax; on Windows x64 it is four registers and a shadow space; on AArch64 AAPCS it is a different set again. None of that is chosen by the compiler — it is the platform ABI. - A mismatch does not produce an error. There is no check. The caller writes an argument where the callee is not looking and the callee reads whatever was there — so you get garbage arguments, a corrupted stack, a return into the wrong address, or a value that is silently wrong three functions later. Because the corruption depends on register contents, it changes with optimization level and with unrelated edits, which is what makes it feel non-deterministic.
- The ABI covers more than the call: struct layout, padding, enum size, name mangling, exception unwinding tables, and how virtual tables are laid out. Two C++ objects compiled with different standard-library versions can disagree about a struct's size and produce exactly the same kind of silent corruption.
- The defences are all about making the contract explicit: a single toolchain and flags across the build,
extern "C"at boundaries to fix mangling and convention, versioned symbols, and treating the ABI as part of the public API of any shared library.
✓ Green flags
- Lists several elements of the contract, not just "arguments in registers".
- Says explicitly that nothing checks it and the failure is silent corruption.
- Knows that the ABI is a platform property, and can name two platforms that differ.
- Includes struct layout and name mangling as ABI, not just the call sequence.
- Mentions caller-saved versus callee-saved and what each side must do.
✗ Red flags
- "The linker would catch it." The linker matches symbol names. Two functions with the same name and different conventions link cleanly.
- "It is the same on every platform, arguments go on the stack." Register passing is the norm on every 64-bit platform in common use.
- "Just recompile everything and it is fine." Usually true and not a model — it does not tell you what to do when you cannot recompile a vendor binary.
- "C++ name mangling exists to make linking harder." It encodes the signature so overloads can coexist, and it is what makes an ABI change a link error instead of silent corruption — occasionally.
Follow-up
You call into a vendor .so and get plausible but wrong numbers back. How do you determine whether it is an ABI problem, and what would you look at first?
Implementation challenge
What to ask them to write or trace on a whiteboard.
Write the caller-side sequence for f(a, b, c) under System V x86-64: which registers, what stack alignment, and what must be saved around the call.