call, ret and the System V convention

A subroutine is a piece of code that can be called from several places and then return to the place that called it. It is also called a function, procedure, or routine.

Returning is the interesting part. The same subroutine may have many callers, so it needs the return address for this particular call. x86 keeps that address on the stack.

What call and ret do

• call label pushes the address of the instruction immediately after the call, then puts the address named by label in rip.
• ret pops a qword into rip, so execution resumes at that address.

This program calls sum and continues at mov r12, rax when sum returns:

Trace the stack with a symbolic starting value. Immediately before call sum, let rsp = S.

moment	rsp	stack and rip effect
before call	S	the next instruction is mov r12, rax
on entry to sum	S - 8	[S - 8] holds the address of mov r12, rax; rip points at sum
after ret	S	rip holds the address loaded from [S - 8]

There is no tag saying that the qword at [S - 8] is a return address. ret simply loads the qword at [rsp] into rip and adds 8 to rsp. If sum pushed a value and failed to pop it, its ret would read that value instead of the return address and try to execute code at the wrong address. A function must undo its stack allocations and pushes in the right order so that rsp points at its return address when it executes ret.

Integer and pointer values in System V

The hardware defines call and ret, but it does not assign registers to parameters or return values. Linux x86-64 code uses the System V AMD64 ABI, an application binary interface that lets separately written functions agree on those details.

This table covers the integer and pointer subset used in this lesson:

value	location
integer/pointer argument 1	rdi
integer/pointer argument 2	rsi
integer/pointer argument 3	rdx
integer/pointer argument 4	rcx
integer/pointer argument 5	r8
integer/pointer argument 6	r9
further integer/pointer arguments	the stack
one integer/pointer return value	rax

The convention also assigns responsibility for preserving general-purpose registers:

kind	registers	responsibility
caller-saved	rax, rcx, rdx, rsi, rdi, r8–r11	the caller saves any value it needs after a call
callee-saved	rbx, rbp, r12–r15	a function that changes one restores the value it inherited

rsp is special stack state. At the point just before ret, a function must have restored rsp to its entry value, where the return address is waiting. The ret then removes that address and restores the caller's pre-call rsp. Do not use rsp as an ordinary register for a long-lived value.

The save rules apply at every call boundary. If a caller needs its current rdi or rcx after a call, the caller must save that value before the call and restore it afterward. If a function uses rbx, rbp, or r12–r15, that function must preserve the value supplied by its own caller.

Here scale uses rbx as scratch storage, so it saves and restores rbx itself:

Removing the push rbx and pop rbx would make scale return the right result while breaking its register-preservation promise: the caller would find 3 in rbx instead of 999.

Stack alignment at a call

System V adds one more rule: immediately before a normal call, rsp must be divisible by 16. The call then pushes an eight-byte return address, so on function entry rsp is 8 modulo 16.

For example, if the caller has aligned rsp = S, the boundary looks like this:

moment	stack pointer modulo 16
immediately before call	0
on entry to the callee	8
after the callee's ret	0

The playground starts _start with a 16-byte-aligned rsp. A direct call from _start therefore meets the rule as long as earlier instructions have not changed rsp. Inside a function, pushes and local allocations must be counted before every nested call. Code that ignores this rule may seem to work until a called function uses an instruction or stack object that requires alignment.

A stack frame with a local variable

When a function moves rsp, a fixed frame pointer makes its stack slots easier to name. By convention that pointer is rbp. This function keeps its argument in a local qword while it calls another function:

On entry to three_times, rsp is 8 modulo 16. push rbp both preserves the caller's frame pointer and makes rsp divisible by 16. Reserving 16 more bytes keeps it divisible by 16 for the nested call twice. Only [rbp - 8] is used here; the other eight reserved bytes provide the space needed to keep the total allocation aligned.

rbp stays fixed while rsp moves, so [rbp - 8] keeps naming the same local qword. At the end, leave is exactly mov rsp, rbp followed by pop rbp: it discards the local area, restores the caller's rbp, and leaves rsp pointing at the return address. Then ret removes that address.

A seventh integer argument

The first six integer or pointer arguments use registers. A seventh one is passed on the stack. The caller must place it there while still satisfying the alignment rule.

Starting with aligned rsp in _start, padding comes before the argument:

Why does [rbp + 16] hold g? The caller places g on the stack, call places the return address below it, and push rbp places the saved frame pointer below that. After mov rbp, rsp, the layout is:

location	contents	placed there by
[rbp + 24]	alignment padding	sub rsp, 8
[rbp + 16]	seventh argument g	push 7
[rbp + 8]	return address	call seventh
[rbp]	caller's rbp	push rbp
below rbp	local storage, if reserved	the callee

Arguments already on the stack have positive offsets from rbp; locals reserved after the frame is established have negative offsets. The caller removes its stack argument and padding after the call, restoring its original rsp.

Recursion and caller-saved arguments

A recursive function follows the same rules as any other caller and callee. Every call adds its own return address, and each active invocation has its own saved values.

rdi is caller-saved. Each non-base invocation needs its own n after the recursive call, so it pushes rdi. Function entry has rsp at 8 modulo 16; the push makes it divisible by 16 for call fact. The matching pop restores both n and the entry value of rsp before ret.

If the push and pop were removed, fact(5) would return 1. The recursion reaches the base case with rdi = 1, and that invocation leaves rdi equal to 1. Every invocation unwinding from it would therefore multiply by 1 instead of by its own saved value.

Each non-base recursive step adds 16 bytes while the next invocation is active: eight for the saved rdi and eight for the next return address. Enough recursive levels exhaust the available stack and fault.

Your turn

Write maximum(a, b) using the System V convention. Treat both arguments as signed qwords and return the larger one in rax. The three calls cover left-less, left-greater, and equal inputs. The first pair, -5 and 3, also distinguishes signed comparison from unsigned comparison because the qword bit pattern for -5 is above 3 when read as unsigned. Keep the three results in r12–r14.

Show solution

Now write recursive fib(n) for n >= 0, with fib(0) = 0 and fib(1) = 1. Return the result in rax and use rbx to keep fib(n - 1) across the second recursive call. Because rbx is callee-saved, fib must restore the value it inherited before every return.

The test calls the base cases and a recursive case. A sentinel is a recognizable check value. Before fib(10), the test puts one in rbx; afterward it copies the preserved value to r15. Keep the three Fibonacci results in r12–r14 and the copied sentinel in r15. Every call must be made with aligned rsp.

Show solution