Advanced Build Engineering
A hermetic build uses only explicitly declared inputs and controlled tools/environment.
What this module covers
Incremental build correctness Dependency discovery and generated sources
Parallel scheduling and resource constraints Build graph partitioning
Content-addressed storage and build caches Remote caching and remote execution Hermetic builds and sandboxing
Reproducible builds and source-date normalization Toolchain pinning and environment isolation Build performance profiling
Monorepo builds and affected-target analysis Code generation, schema compilers, and asset pipelines
Build configurations, feature flags, and matrix builds Cross-compiling firmware and embedded applications
Build provenance and artifact attestations
8.1 Hermetic builds
A hermetic build uses only explicitly declared inputs and controlled tools/environment. It avoids silently depending on arbitrary files from the host machine or network.
Improved remote execution compatibility.
Common sources of non-hermetic behavior include reading undeclared environment variables, using absolute machine-specific paths, downloading unpinned files during compilation, reading the current time, and relying on files outside the declared workspace.
8.2 Reproducible builds
A reproducible build aims for bit-for-bit identical outputs when inputs and the build environment are equivalent. It may require:
Pinned compiler and linker versions.
Stable archive ordering.
Stable locale and path settings.
Avoiding embedded machine paths and timestamps.
Deterministic code generation.
A documented build environment.
Repeatability (“the process usually works again”) is weaker than reproducibility (“the output bytes match under the specified conditions”).
What actually changes a binary Four real experiments with GCC 11.2 on Ubuntu 22.04 (WSL2). Identical inputs give identical bytes:
Terminal session: Embedding the build time breaks that; SOURCE_DATE_EPOCH (honoured by GCC for __DATE__/__TIME__) restores it:
Building the same source in two directories differs because debug information records the build path; -ffile-prefix-map normalises it:
Archives store member timestamps unless the archiver runs in deterministic mode (the default D on this binutils build; U turns it off):
$ cc -O2 hello.c -o hello-1 && cc -O2 hello.c -o hello-2
$ sha256sum hello-1 hello-2
86aac487be9ab771f42c0c2c8569fb392a9b7f5d5b8e98dff79bb837e1269b62 hello-1
86aac487be9ab771f42c0c2c8569fb392a9b7f5d5b8e98dff79bb837e1269b62 hello-2$ cc -O2 version.c -o v1 && sleep 1 && cc -O2 version.c -o v2
$ ./v1; ./v2
built Oct 19 2026 14:52:19
built Oct 19 2026 14:52:20
$ sha256sum v1 v2 | cut -c1-16
2545a062cf852fa9
4f4a0890a8b50974
$ export SOURCE_DATE_EPOCH=1767225600
$ cc -O2 version.c -o v3 && sleep 1 && cc -O2 version.c -o v4
$ ./v3; ./v4
built Jan 1 2026 00:00:00
built Jan 1 2026 00:00:00
$ sha256sum v3 v4 | cut -c1-16
c155a5b11ae291a2
c155a5b11ae291a2
$ unset SOURCE_DATE_EPOCH$ mkdir -p /tmp/bsvc/alice /tmp/bsvc/bob && cp hello.c /tmp/bsvc/alice/ && cp hello.c /tmp/bsvc/bob/
$ (cd /tmp/bsvc/alice && cc -g -O2 hello.c -o hello) && (cd /tmp/bsvc/bob && cc -g -O2 hello.c -o hello)
$ sha256sum /tmp/bsvc/alice/hello /tmp/bsvc/bob/hello | cut -c1-16
232d44dbd3fcda3c
1073b9d5504d06e9
$ strings /tmp/bsvc/alice/hello | grep -m1 /tmp/bsvc
/tmp/bsvc/alice
$ (cd /tmp/bsvc/alice && cc -g -O2 -ffile-prefix-map=/tmp/bsvc/alice=. hello.c -o hello) && (cd /tmp/bsvc/bob && cc -g -O2 -ffile-prefix-map=/tmp/bsvc/bob=. hello.c -o hello)
$ sha256sum /tmp/bsvc/alice/hello /tmp/bsvc/bob/hello | cut -c1-16
a96e373368529fdd
a96e373368529fdd$ cc -c hello.c -o hello.o
$ ar rcs lib1.a hello.o && sleep 1 && touch hello.o && ar rcs lib2.a hello.o && cmp lib1.a lib2.a && echo identical
identical
$ ar rcsU lib3.a hello.o && sleep 1 && touch hello.o && ar rcsU lib4.a hello.o && cmp lib3.a lib4.a
lib3.a lib4.a differ: char 34, line 2
[exit status 1]
$ ar tv lib1.a; ar tv lib3.a
rw-r--r-- 0/0 1504 Jan 1 00:00 1970 hello.o
rw-r--r-- 1000/1000 1504 Oct 19 14:52 2026 hello.o8.3 Content-addressed caching
A cache key may conceptually include:
If the key and cache model are correct, a matching output can be reused. Missing an input from the key can cause an incorrect cache hit; adding irrelevant inputs can reduce cache efficiency.
A cache key in practice Output (CPython 3.13.1): When the header is not part of the key, changing it produces the same key — the cache would hand back an object compiled against the old header. That is the cache equivalent of the stale-object bug in Module 2.
hash(
source contents,
declared dependencies,
compiler identity,
flags,
relevant environment,
build action
)import hashlib
import json
def action_key(sources, flags, compiler, env):
h = hashlib.sha256()
for path in sorted(sources):
h.update(path.encode() + b"\0" + hashlib.sha256(sources[path]).digest())
h.update(json.dumps({"flags": flags, "compiler": compiler, "env": env}, sort_keys=True).encode())
return h.hexdigest()[:16]
sources = {"main.c": b'#include "math.h"\nint main(void){return SCALE;}\n', "math.h": b"#define SCALE 10\n"}
base = action_key(sources, ["-O2"], "gcc 11.2.0", {"LANG": "C"})
print("baseline ", base)
print("same inputs again ", action_key(dict(sources), ["-O2"], "gcc 11.2.0", {"LANG": "C"}))
print("header changed ", action_key({**sources, "math.h": b"#define SCALE 100\n"}, ["-O2"], "gcc 11.2.0", {"LANG": "C"}))
print("flag changed ", action_key(sources, ["-O3"], "gcc 11.2.0", {"LANG": "C"}))
print("compiler upgraded ", action_key(sources, ["-O2"], "gcc 12.3.0", {"LANG": "C"}))
for header in (b"#define SCALE 10\n", b"#define SCALE 100\n"):
# the compiler reads this header, but the build rule never declared it as an input
declared = {"main.c": sources["main.c"]}
print(f"undeclared, SCALE={header.split()[-1
…baseline 0e8e628afeb98d46
same inputs again 0e8e628afeb98d46
header changed 0d1eb18555485637
flag changed 2752e35977affd6f
compiler upgraded 10702d4e87f6c5d9
undeclared, SCALE=10 f24caf5c8c3c75ce
undeclared, SCALE=100 f24caf5c8c3c75ce8.4 Remote cache and remote execution
A remote cache stores action outputs for reuse across machines. Remote execution dispatches eligible build actions to remote workers. Important design questions include:
Are all inputs declared?
Is the toolchain available and pinned remotely?
Are outputs uploaded only from trusted jobs?
Can untrusted pull requests access sensitive cache entries?
How are cache namespaces and permissions separated?
What is the policy for cache invalidation?
8.5 Build performance
Measure before optimizing. Track:
Clean build duration.
Incremental build duration.
Cache hit rate.
CPU and memory usage.
Time spent in configuration, dependency resolution, compilation, linking, and tests.
Number of rebuilt targets after a small change.
Typical optimizations:
Fix missing or overly broad dependencies.
Improve parallelism without introducing races.
Cache stable tool and dependency downloads.
Split targets at useful boundaries.
Avoid unnecessary global rebuild triggers.
Use precompiled headers or compiler caches when appropriate.
Avoid excessively granular targets when orchestration overhead dominates.
Avoid giant targets when a small change rebuilds everything.
8.6 Monorepo and affected-target analysis
A monorepo may contain many projects with shared dependencies. An affected-target system identifies which tests/builds are relevant to a change. Correctness depends on modeling dependencies across directories, generated files, shared configuration, and toolchain definitions.
8.7 Generated code and assets
Protocol buffer or schema compiler output.
Register headers generated from MCU descriptions.
API clients generated from API schemas.
Web asset bundling and minification.
Version headers generated from Git metadata.
Treat generators as tools with versioned inputs. Decide whether generated files are committed, generated during build, or published as separate artifacts. Avoid having developers and CI silently use different generator versions.
A version header generated from Git The rule below regenerates build/version.h on every build but replaces it only when the text changes, so an unchanged version never triggers recompilation:
Terminal session (GNU Make 4.4.1, Git 2.34.1) in a repository tagged v1.4.0:
git describe gives the nearest tag, the number of commits since it, the abbreviated commit ID and -dirty for uncommitted changes — useful build identification without embedding a timestamp.
$ cat Makefile
VERSION := $(shell git describe --tags --always --dirty)
build/version.h: FORCE | build
@echo '#define FIRMWARE_VERSION "$(VERSION)"' > [email protected]
@cmp -s [email protected] $@ || mv [email protected] $@
@rm -f [email protected]
build:
mkdir -p build
.PHONY: FORCE
FORCE:
$ make -s && cat build/version.h
#define FIRMWARE_VERSION "v1.4.0"
$ echo '/* tweak */' >> hello.c && git commit -qam 'refactor: tidy' && make -s && cat build/version.h
#define FIRMWARE_VERSION "v1.4.0-1-g49a747f"
$ echo '/* local edit */' >> hello.c && make -s && cat build/version.h
#define FIRMWARE_VERSION "v1.4.0-1-g49a747f-dirty"
$ ls --full-time build/version.h | awk '{print $7}' > /tmp/bsvc/t1; sleep 1; make -s; ls --full-time build/version.h | awk '{print $7}' > /tmp/bsvc/t2; cmp -s /tmp/bsvc/t1 /tmp/bsvc/t2 && echo 'unchanged header keeps its timestamp'
unchanged header keeps its timestamp8.8 Build variants
A project may have debug/release, product variants, board revisions, feature configurations, architecture targets, and customer-specific options. Define supported combinations deliberately and use a build matrix where appropriate. Prevent invalid combinations at configuration time rather than discovering them late.
8.9 Embedded build details
Embedded builds may require:
Cross compiler and binutils.
Startup code and vector table.
Device-specific headers and libraries.
Bootloader/application offset.
Compiler ABI and FPU options.
Flash image conversion.
Map-file inspection and memory budget checks.
Example linker option (toolchain-specific):
_Not run — no ARM cross toolchain on the build machine. Module 17 shows the map-file and objcopy steps with the host toolchain._
Convert ELF to Intel HEX with a compatible tool:
_Not run — no ARM cross toolchain on the build machine._ This is illustrative. Startup files, linker flags, output formats, and flashing workflows are MCU/toolchain-specific. Verify memory usage, section placement, and boot requirements for the actual target.
arm-none-eabi-gcc startup.o main.o \
-T linker_script.ld \
-Wl,-Map=firmware.map \
-o firmware.elfarm-none-eabi-objcopy -O ihex firmware.elf firmware.hexQuiz
? Two builds of the same source, one second apart, produced different SHA-256 digests. What fixed it in the real run?
- Building with `-O0` :: Optimisation was not the cause.
+ Setting `SOURCE_DATE_EPOCH`, which GCC uses for `__DATE__`/`__TIME__` :: The embedded time became fixed.
- Running `make clean` :: Still embeds the current time.
- Using `ar rcsU` :: That makes archives *less* deterministic.
? A cache key omits a header that the action reads. What can happen?
+ A changed header still produces the same key, so a stale output is reused. :: Shown by the `undeclared header` lines.
- The build becomes slower but correct. :: It becomes fast and wrong.
- The cache refuses to store results. :: Nothing detects the omission.
- Nothing, because headers are not compiled. :: They are compiled into every includer.
? Which statements about remote caches are true? (select all that apply)
+ Outputs should be uploaded only from trusted jobs. :: Otherwise an untrusted build can poison the cache.
+ Every input must be declared for hits to be correct. :: Undeclared inputs cause wrong hits.
+ Toolchains must be pinned and available on remote workers. :: Different tools produce different outputs
…