The Full Pipeline

A compiler is rarely one single step — it's a pipeline of distinct stages, each handing a transformed version of the program to the next. Understanding the pipeline as a whole is the map; the rest of this module is exploring each stage on that map in detail.

The compilation pipeline is the ordered sequence of stages source code passes through before it becomes something a CPU can actually execute: preprocessing, lexical analysis, parsing, semantic analysis, generating an intermediate representation, optimizing it, generating machine code, assembling it, linking it with other code, and finally loading it into memory to run.

Splitting compilation into distinct stages, each with one clear job, makes an enormously complex translation problem manageable — each stage can be understood, tested, debugged and even reused (the same optimizer and code generator can serve many different source languages) largely independently of the others.

Every time you run gcc, clang, rustc, javac, or any other compiler on source code, all of these stages happen — sometimes visibly (you can ask most compilers to stop after any one stage and inspect its output), usually invisibly, all within a single command.

Think of an assembly line building a car: one station attaches the frame, the next installs the engine, the next the electronics, the next the paint. Each station transforms the car a little further toward finished, and problems are far easier to diagnose when you know which station introduced them. A compiler's pipeline works the same way, with source code as the car and machine code as the finished product.

Every one of these stages is covered as its own section in this module. Notice that a lot happens between typing source code and the CPU actually running anything — none of it is magic, and none of it is optional.

Not every stage is present in every implementation, and the boundaries aren't always sharp — some compilers merge stages for efficiency, and some languages skip certain stages entirely (an interpreter has no separate assembler or linker step, for instance, a distinction developed fully in the future Interpreters, Bytecode & Runtimes module).

Most real compilers are organized into a front end (language-specific: preprocessing through semantic analysis and IR generation) and a back end (target-specific: optimization through code generation) — this split is exactly what lets a single back end, like LLVM's, serve many different front-end languages (C, C++, Rust, Swift and others all compile down to the same LLVM IR).

Each stage typically both consumes and produces a well-defined data structure — characters in, tokens out (lexer); tokens in, an AST out (parser); an AST in, an annotated/checked AST or IR out (semantic analysis) — which is exactly what makes independent testing and reasoning about each stage possible.

You can usually ask a real compiler to stop early and show you an intermediate stage's output — for example, gcc -E stops after preprocessing, and gcc -S stops after code generation, emitting assembly instead of a binary. This is an extremely useful debugging and learning technique used throughout this module's later sections.

Treating "compiling" as one indivisible black-box operation, which makes genuinely different classes of errors (a syntax error from the parser vs. a type error from semantic analysis vs. a linker error about an undefined symbol) all look like the same kind of mysterious failure.

Assuming every language's compiler follows an identical pipeline — the shape is similar across many compiled languages, but the specific stages, their names, and which are merged or skipped vary meaningfully by language and implementation.

Confusing where an error actually originates — for example, blaming "the compiler" for what's actually a linker error (a missing symbol) happening well after code generation has already succeeded.

Using a compiler's stage-stopping flags (like gcc's -E, -S, and -c) to inspect exactly what a specific stage produced, when debugging an unexpected compilation result.

Reading a compiler error message's wording carefully to identify which stage it came from — "unexpected token" points to the parser, "undeclared identifier" points to semantic analysis, "undefined reference" points to the linker — since that immediately narrows down what kind of problem you're actually looking at.

Using an online tool like Compiler Explorer (godbolt.org) to see the actual assembly a compiler generates for a snippet of source code, making the code-generation stage concrete rather than theoretical.

Front end / middle end / back end Production compilers are often described in three parts: a front end (lexing, parsing, semantic analysis — everything specific to the source language), a middle end (the IR and most optimizations — largely language- and target-independent), and a back end (instruction selection, register allocation, and final code generation — everything specific to the target architecture). This three-way split is precisely why infrastructure like LLVM can support dozens of source languages and dozens of target architectures without needing a distinct compiler for every combination.

Single-pass vs. multi-pass compilation Some compilers (especially older or simpler ones) are single-pass, processing the source roughly once, stage by stage, in a single sweep — simple and fast, but limiting for optimization, since later stages can't easily see the whole program before earlier stages finish. Most modern optimizing compilers are multi-pass, revisiting the IR repeatedly (each optimization is itself typically its own pass) to progressively improve the generated code, at the cost of longer compile times.

Compile time vs. link time vs. load time vs. run time Different decisions and different classes of error are resolved at genuinely different times in a program's life: compile time (syntax and most type errors), link time (whether every referenced symbol is actually found somewhere), load time (relocating and mapping the executable into a specific process's memory), and run time (everything the program's actual logic does once executing) — a distinction that recurs constantly throughout this module and is essential vocabulary for diagnosing where a given problem is actually coming from.

Name the main stages of a typical compiler pipeline, in order.

Preprocessing, lexical analysis (tokenizing), parsing (building an AST), semantic analysis (type/scope checking), intermediate representation generation, optimization, code generation, assembly, linking, and loading — though exact stage names and boundaries vary by compiler.

What's the difference between a compiler's front end and back end?

The front end handles everything specific to the source language (lexing through semantic analysis and IR generation). The back end handles everything specific to the target architecture (optimization, instruction selection, register allocation, final code generation) — this split is what lets one back end serve many front-end languages.

At which stage would an undeclared-variable error typically be caught?

Semantic analysis — the parser only checks that the code is grammatically well-formed (correct syntax), while checking whether a specific name has actually been declared before use is a semantic check, not a syntactic one.

At which stage would a missing function definition (declared but never implemented anywhere) typically be caught?

At link time — the compiler itself is often satisfied as long as a matching declaration exists (so it knows the function's signature), but the linker is what actually needs to find that function's real machine code somewhere among the object files and libraries, and fails with an "undefined reference" error if it can't.

Source Code
  -> Preprocessor        (macro expansion, #include)
  -> Lexer               (characters -> tokens)
  -> Parser              (tokens -> AST)
  -> Semantic Analysis   (type checking, scope checking)
  -> Intermediate Rep.   (a lower-level, language-independent form)
  -> Optimization        (make the IR faster/smaller)
  -> Code Generation     (IR -> target assembly)
  -> Assembler           (assembly -> object file, machine code)
  -> Linker              (object files + libraries -> executable)
  -> Loader              (OS loads the executable into memory)
  -> CPU                 (fetches and executes the machine instructions)