Functions, Call Frames and Calling Conventions Inside a VM

Calling a function means: remember where you are, give the function its own workspace for parameters and local variables, run it, take its result and continue where you left off.

Learning Objectives

Represent function values, declarations, function objects and code references; implement call and return instructions.

Distinguish the call stack from the operand stack; lay out VM call frames with parameters and local slots; define return-value conventions.

Implement recursion, arity checks, native functions, closures and upvalues (introduced here, completed in Module 12), escaping locals, tail calls, unwinding, stack traces, debug symbols, stable calling conventions, FFI and stack-overflow handling.

Plain-Language Explanation

Calling a function means: remember where you are, give the function its own workspace for parameters and local variables, run it, take its result and continue where you left off. The VM keeps a stack of these workspaces — call frames — so functions can call functions, including themselves.

Precise Technical Explanation

Functions are values: they can be stored in variables, passed and returned. In MiniLang the compiler produces a function prototype (name, arity, bytecode chunk, upvalue count) and the VM wraps it in a closure at run time.

Function declarations and definitions fun square(n) { return n * n; } compiles the body into its own chunk, stored as a constant in the enclosing chunk; executing the declaration creates a closure and binds it to the name.

Function objects and code references A function object references its code (the prototype) plus run-time context (captured variables). Many closures can share one prototype.

CALL a: the callee sits below its a arguments on the operand stack. The VM checks the callee is callable and the arity matches, then pushes a frame.

RETURN pops the result, discards the frame's slots (arguments, locals and the callee), pushes the result for the caller and resumes the caller's instruction pointer.

Call stack versus operand stack The call stack holds frames (function, instruction pointer, base pointer); the operand stack holds values. MiniLang keeps locals inside the operand stack (frame windows), like clox and the JVM's frame-local arrays.

VM call frames { closure, ip, base } — base is the index of the frame's first slot in the value stack.

Parameters and local slots Teaching convention: slot 0 is the first parameter, then the remaining parameters, then locals in declaration order; the callee itself sits just below slot 0.

Return value conventions Exactly one value is returned (nil if the function ends without return); the caller finds it where the callee was.

Each call gets a new frame, so recursive calls have independent locals.

Calling with the wrong number of arguments is a runtime error (Expected 1 argument(s) but got 2.).

Native/built-in functions Implemented in the host language (MiniLang's str); the VM calls them directly without a bytecode frame.

Calling native functions from the VM Collect the arguments from the stack, call the host function, replace callee and arguments with the result. Natives must validate arguments and must not leak host exceptions.

Closures and captured variables Inner functions that refer to outer locals capture them (Module 12).

Upvalues / closed-over variables Lua's term (also clox's): a reference to a captured variable — pointing into the stack while the variable's frame is alive (open), copied to the heap when it ends (closed).

Lexical environments versus stack slots Tree-walkers keep variables in environment objects; bytecode VMs keep them in stack slots and move only *captured* ones to the heap.

Escaping local variables A local escapes when a closure that captures it outlives its frame; the VM must keep it alive (closing the upvalue).

A call in tail position (return f(x);) can reuse the current frame, so tail recursion runs in constant stack space. Scheme requires it; ES2015 specified proper tail calls but only Safari's JavaScriptCore ships them; CPython, the JVM and MiniLang do not.

Exceptions and unwinding across frames Errors unwind frames until a handler is found (Module 13); MiniLang has no handlers, so errors unwind all frames and report a trace.

The list of active frames with function names and current lines — produced from frames' instruction pointers and line tables.

Debug symbols and source locations Line tables (offset → line) and local-variable tables (slot → name, live range) let debuggers show positions and variables.

Calling convention stability Within one VM the convention is internal; across language boundaries (native code, FFI, other compiled modules) it must be fixed and documented — an ABI.

Foreign-function interface (FFI) introduction Calling functions in native libraries from managed code (Python ctypes, Java JNI/FFM API,.NET P/Invoke): the runtime marshals arguments to the native ABI and back. Native code bypasses the VM's safety guarantees.

VM stack overflow handling Limit frames (MiniLang: 64) and stack values (256) and report a clean runtime error instead of crashing the host.

Example bytecode Local-slot convention: the parameter n is slot 0 of square's frame. Calling square(6) pushes the closure and 6; the frame's base points at 6; GET_LOCAL 0 twice, MULTIPLY → 36, RETURN replaces the closure and argument with 36. The real listing and trace are below (the VM's opcode is named MULTIPLY).

FUNCTION square (1 parameter)
  GET_LOCAL 0
  GET_LOCAL 0
  MUL
  RETURN

Why It Matters in Real Systems

Frame layout determines call overhead, debugging (stack traces), recursion limits and how closures, generators, exceptions and FFI work.

Key Terms

Per-call record: function, ip, base Number of parameters Function implemented by the host Reference to a captured variable Call as the last action of a function

Binary calling convention between compiled components

Diagram, Execution Trace or State Transition

Text equivalent: the caller pushes the callee and its argument; the new frame's base points at the argument, so GET_LOCAL 0 reads n; temporaries are pushed above it; RETURN truncates the stack to just below the callee and pushes the result.

flowchart LR
  subgraph Stack["Value stack"]
    S0["slot: closure square"] --> S1["slot 0 of square: n = 6"] --> S2["temporaries: 6, 6 → 36"]
  end
  F0["frame 0: script, base 0"] --> F1["frame 1: square, base = index of n"]

Syntax or Instruction Format

CALL a (a = argument count), RETURN, CLOSURE k (isLocal, index)…, GET_LOCAL s, GET_UPVALUE u.

Complete Example

Output (bytecode for the script and for square):

Output (VM trace — note the frame switch inside square):

Recursion, arity errors and stack overflow with stack traces (both back ends):

Production runtimes: Python frames, the recursion limit (CPython has no tail-call optimisation), and a native call through the FFI (CPython 3.13.1, Windows C runtime):

fun square(n) { return n * n; }
print(square(6));

== <script> ==
constants: 0 → <fn square>, 1 → 6
names:     0 → square
0000    1 CLOSURE          0    ; <fn square>
0002    | DEFINE_GLOBAL    0    ; 'square'
0004    2 GET_GLOBAL       0    ; 'square'
0006    | CONSTANT         1    ; 6
0008    | CALL             1    ; 1 argument
0010    | PRINT
0011    | HALT

== square (1 parameter) ==
0000    1 GET_LOCAL        0    ; slot 0
0002    | GET_LOCAL        0    ; slot 0
0004    | MULTIPLY
0005    | RETURN

fn          offset  instruction                      stack after
<script>    0000    CLOSURE 0  ; <fn square>         [<fn square>]
<script>    0002    DEFINE_GLOBAL 0  ; 'square'      []
<script>    0004    GET_GLOBAL 0  ; 'square'         [<fn square>]
<script>    0006    CONSTANT 1  ; 6                  [<fn square>, 6]
<script>    0008    CALL 1  ; 1 argument             [<fn square>, 6]
square      0000    GET_LOCAL 0  ; slot 0            [<fn square>, 6, 6]
square      0002    GET_LOCAL 0  ; slot 0            [<fn square>, 6, 6, 6]
square      0004    MULTIPLY                         [<fn square>, 6, 36]
square      0005    RETURN                           [36]
<script>    0010    PRINT                            []
<script>    0011    HALT                             []
output: 36

fun fact(n) { if (n <= 1) return 1; return n * fact(n - 1); }
print(fact(10));
fun countdown(n) { return countdown(n - 1); }   // no base case
countdown(3);

3628800
Runtime error [line 3:27]: Stack overflow (call depth limit 64).
  in countdown() [line 3]
  in countdown() [line 3]
  in countdown() [line 3]
  in countdown() [line 3]
  ... 57 more frames ...
  in countdown() [line 3]
  in countdown() [line 3]
  in script [line 4]

import sys, ctypes


def inner():
    frame = sys._getframe()
    chain = []
    while frame:
        chain.append(f"{frame.f_code.co_name}:{frame.f_lineno}")
        frame = frame.f_back
    return chain


def outer():
    return inner()


print(" ← ".join(outer()))


def countdown(n):
    return countdown(n - 1)      # a tail call, but CPython still pushes a frame


sys.setrecursionlimit(200)
try:
    countdown(10)
except RecursionError as e:
    print("RecursionError:", e)

libc = ctypes.cdll.msvcrt                  # FFI: call strlen from the C runtime
libc.strlen.argtypes = [ctypes.c_char_p]
libc.strlen.restype = ctypes.c_size_t
print("strlen via FFI:", libc.strlen(b"interpreter"))

inner:8 ← outer:14 ← <module>:17
RecursionError: maximum recursion depth exceeded
strlen via FFI: 11

Line-by-Line Explanation

square's chunk is constant 0 of the script; CLOSURE 0 creates the function value and DEFINE_GLOBAL names it. GET_GLOBAL square; CONSTANT 6; CALL 1 sets up the call.

In the trace, rows with fn = square run in the new frame: two GET_LOCAL 0 read n, MULTIPLY leaves 36, RETURN drops the frame and leaves 36 for PRINT.

countdown has no base case: after 64 frames the VM reports a stack overflow with the active frames; both back ends report the same error.

Python: f_back links frames into the call stack; tail recursion still grows the stack; ctypes marshals a Python bytes to a C char* and the result back.

Expected Output

36; 3628800 and the overflow error; Python's frame chain names (line numbers depend on the file).

Step-by-Step Runtime Behavior

Before CALL 1: stack [<fn square>, 6]. After the call: frame base = 1. GET_LOCAL 0 → [<fn square>, 6, 6]; again → […, 6, 6, 6]; MULTIPLY → [<fn square>, 6, 36]; RETURN → [36].

Common Misconceptions

"The call stack and the operand stack are the same thing." Related, but frames and values are separate structures (MiniLang stores locals in the value stack and frames in a frame array).

"Recursion depth is unlimited if memory allows." VMs impose limits to stay safe.

"Every language optimises tail calls." Few mainstream implementations do.

Common Bugs and Failure Modes

Off-by-one frame bases (reading the callee as a parameter), forgetting to drop the callee on return, missing arity checks, natives that throw host exceptions, unbounded recursion crashing the host process.

Corrected or Safer Patterns

Define and test the slot convention; check arity; wrap natives; enforce depth limits; produce stack traces from frames.

Debugging and Inspection Techniques

Stack traces, frame inspection (sys._getframe, debuggers), VM traces showing the current function, and line tables.

Performance Considerations

Call overhead (frame setup, argument copying, arity checks) matters for small functions — the motivation for inlining in JITs (Module 16).

Security Considerations

Natives and FFI are trust boundaries: validate arguments, never expose raw pointers to untrusted code, and limit recursion to prevent host stack exhaustion.

Portability and Version Caveats

sys._getframe is CPython-specific; recursion limits and frame representations differ by implementation (CPython 3.11 made frames lazily materialised).

Real-World Use Cases

Every function call in every language; debuggers; profilers; FFI bindings to native libraries.

Best Practices

Document the frame layout; keep native calls thin and validated; report stack overflow as a language-level error.

Exercises

E1 · Beginner. In fun add(a, b) { let c = a + b; return c; }, which slots hold a, b and c?

Answer: a slot 0, b slot 1, c slot 2.

E2 · Intermediate. Why must RETURN truncate the stack to base − 1 rather than base?

Answer: The callee (the closure) sits at base − 1; it must be removed too, so the result replaces it.

E3 · Advanced challenge. How would you add tail-call optimisation to the MiniLang VM?

Answer: Add a TAIL_CALL a instruction emitted for return f(args);: instead of pushing a new frame, close the current frame's upvalues, copy the callee and arguments down to the current frame's slots (base − 1 …), and restart the frame with the new closure and ip = 0 — reusing the frame keeps the frame count constant.

Quiz

? In MiniLang's teaching convention, what is in slot 0 of a function's frame?
+ The first parameter :: The callee sits just below slot 0.
- The function itself :: That is below the frame (clox uses slot 0 for it; this course does not).
- The return address :: Kept in the frame record, not a slot.
- The global table :: Globals are separate.

? What happens when MiniLang calls a function more than 64 levels deep?
+ A runtime "Stack overflow" error with a stack trace. :: The VM enforces a call-depth limit.
- The host process crashes. :: Limits prevent that.
- The VM switches to tail calls. :: MiniLang has no TCO.
- The oldest frames are discarded. :: Frames are never discarded mid-call.

? Which are trust boundaries in a VM? (select all that apply)
+ Native functions :: Host code outside the VM's checks.
+ FFI calls into native libraries :: Bypass memory safety.
- Reading a local slot :: Verified and bounds-checked.
+ Loading bytecode from outside :: Must be verified.

Summary and Key Takeaways

Calls push frames that window the value stack; returns pop them and leave one result. Arity checks, depth limits, native-call wrappers, stack traces and debug tables turn this mechanism into a safe, debuggable foundation for closures, exceptions and FFI.