Assembly Language and Calling Conventions
Assembly is a human-readable form of machine code: one line per instruction, with labels for addresses and directives (.text, .globl, .ascii) for the assembler.
Concept
Assembly is a human-readable form of machine code: one line per instruction, with labels for addresses and directives (.text, .globl, .ascii) for the assembler. The assembler (as) translates it to an object file; the linker (ld) produces the executable.
x86-64 Registers (System V ABI, used by Linux and macOS) rdi, rsi, rdx, rcx, r8, r9 Integer/pointer arguments 1–6 Return value (and system call number) Frame pointer (optional) rbx, rbp, r12–r15
Callee-saved: a function must restore them rax, rcx, rdx, rsi, rdi, r8–r11
Caller-saved: may be destroyed by a call Floating-point arguments; xmm0 returns floats
Arguments beyond the sixth go on the stack. The stack must be 16-byte aligned at each call. Windows x64 uses a different convention (rcx, rdx, r8, r9 and 32 bytes of shadow space).
The Call Stack call f pushes the return address and jumps to f; ret pops it back into the PC.
A function's stack frame holds saved registers, local variables that don't fit in registers, and outgoing stack arguments.
The stack grows toward lower addresses on x86-64, ARM64 and RISC-V.
ARM64 and RISC-V Conventions ARM64: arguments x0–x7, return x0, link register x30 holds the return address (bl/ret). RISC-V: arguments a0–a7, return a0, return address in ra (jal/ret).
Talking to the Kernel Directly A system call on x86-64 Linux: number in rax, arguments in rdi, rsi, rdx, r10, r8, r9, then the syscall instruction. write is number 1, exit is 60. The C library normally does this for you (Module 11).
Why It Matters
Reading compiler output, debugging crashes without source, reverse engineering, writing boot code and interrupt handlers, and understanding security exploits (return addresses on the stack) all require assembly and ABI knowledge.
Syntax
call func ; push return address, jump
ret ; pop return address into rip
push rbx ; rsp -= 8; [rsp] = rbx
lea rax, [rdi+rdx] ; address arithmetic without memory access (often used for a + b)Example
7.1 C functions and the assembly GCC generates 7.2 A program with no C library: raw system calls in assembly
__attribute__((noinline)) long scale(long value, long factor, long offset)
{
return value * factor + offset;
}
long sum_array(const long *a, long n)
{
long s = 0;
for (long i = 0; i < n; i++) s += a[i];
return s;
}
__attribute__((noinline)) long many(long a, long b, long c, long d, long e, long f, long g, long h)
{
return a + b + c + d + e + f + g + h; /* args 7 and 8 arrive on the stack */
}
long caller(void)
{
return scale(10, 3, 2) + many(1, 2, 3, 4, 5, 6, 7, 8);
}$ gcc -O1 -S -masm=intel -fcf-protection=none -fno-asynchronous-unwind-tables calls.c -o calls.s && grep -vE '^\s*\.(file|text|globl|type|size|ident|section|p2align|intel_syntax)' calls.s
scale:
imul rdi, rsi
lea rax, [rdi+rdx]
ret
sum_array:
test rsi, rsi
jle .L5
mov rax, rdi
lea rcx, [rdi+rsi*8]
mov edx, 0
.L4:
add rdx, QWORD PTR [rax]
add rax, 8
cmp rax, rcx
jne .L4
.L2:
mov rax, rdx
ret
.L5:
mov edx, 0
jmp .L2
many:
add rdi, rsi
add rdi, rdx
add rdi, rcx
add rdi, r8
lea rax, [rdi+r9]
add rax, QWORD PTR 8[rsp]
add rax, QWORD PTR 16[rsp]
ret
caller:
push rbx
mov edx, 2
mov esi, 3
mov edi, 10
call scale
mov rbx, rax
push 8
push 7
mov r9d, 6
mov r8d, 5
mov ecx, 4
mov edx, 3
mov esi, 2
mov edi, 1
call many
add rsp, 16
add rax, rbx
pop rbx
ret# hello.s - x86-64 Linux, no C library: talk to the kernel directly with the syscall instruction
.intel_syntax noprefix
.globl _start
.section .rodata
msg: .ascii "hello from raw assembly\n"
len = . - msg
.text
_start:
mov rax, 1 # system call number 1 = write
mov rdi, 1 # arg 1: file descriptor 1 = stdout
lea rsi, [rip + msg] # arg 2: address of the bytes
mov rdx, len # arg 3: number of bytes
syscall # enter the kernel
mov rax, 60 # system call number 60 = exit
mov rdi, 7 # arg 1: exit status 7
syscall$ as hello.s -o hello.o && ld hello.o -o hello
$ ./hello; echo "exit status: $?"
hello from raw assembly
exit status: 7
$ ls -l hello | awk '{print $5 " bytes"}'
8880 bytes
$ file hello
hello: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, not strippedExpected Result
scale(value, factor, offset) receives its arguments in rdi, rsi, rdx; the result value*factor + offset is formed with imul and a lea used as a fast adder, and returned in rax.
sum_array became a pointer loop: rcx holds the end address a + n*8 (base + index × 8 addressing).
many takes arguments 7 and 8 from the stack at [rsp+8] and [rsp+16] (above the return address at [rsp]).
caller pushes 8 and 7 before call many, fills six registers with 1–6, saves rbx (callee-saved) because it keeps scale's result across the second call, and removes the stack arguments afterwards (add rsp, 16).
In 7.2 the program printed its message and exited with status 7 using two syscall instructions and no C library at all. The statically linked executable is about 9 KB instead of the ~16 KB of a minimal dynamically linked C program.
Common Mistakes
Using a callee-saved register without saving and restoring it.
Misaligning the stack before call (crashes in SSE code such as printf with floats).
Assuming Linux and Windows x64 use the same calling convention.
Confusing lea (computes an address, no memory access) with mov (loads from memory).
Best Practices
Prefer inline functions and intrinsics to hand-written assembly; write assembly only for boot code, context switches or measured hot spots.
When you must, follow the ABI exactly and document which registers are clobbered.
Read -O1/-O2 output rather than -O0, which is full of redundant stack traffic.
Interview Relevance
High for systems roles: "how are arguments passed on x86-64?", "caller-saved vs callee-saved?", "what happens during call and ret?", "what is a stack frame?", "how does a program make a system call without libc?".
Practice Questions
Compile a recursive factorial with -O1 and identify the stack frame setup.
Write an assembly function long add3(long, long, long) and call it from C.
Modify hello.s to print the message twice using a loop.
What does the stack look like right after call many in caller?
Quick Quiz
Q1. (Multiple choice) In the System V x86-64 ABI, the third integer argument is in: A) rcx B) rdx C) r8 D) the stack
Answer: B (verified: offset arrived in rdx).
Q2. (Predict the output) After running hello from 7.2, what does echo $? print?
Answer: 7 (verified).
Q3. (True/False) rbx may be freely overwritten by any function without restoring it.
Answer: False. It is callee-saved (verified: caller pushed and popped it).
Q4. (Find the bug) An assembly function uses r12 as a loop counter without push r12/pop r12, and the C caller later misbehaves.
Answer: r12 is callee-saved; the caller expected its value to survive the call.
Coding Challenge
Write a C function that calls the write system call through the generic syscall() wrapper (<unistd.h>, SYS_write) and returns the number of bytes written; print that number.
#define _GNU_SOURCE
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <sys/syscall.h>
int main(void)
{
const char *msg = "written by syscall(SYS_write, ...)\n";
long n = syscall(SYS_write, 1, msg, strlen(msg));
printf("SYS_write is system call number %d, it returned %ld\n", SYS_write, n);
return 0;
}