Troubleshooting and Performance

What this module covers

Clean build succeeds but incremental build fails Missing headers, libraries, symbols, and generated files Stale cache and configuration mismatch Dependency version conflict

Nondeterministic and flaky build failures Parallel race conditions CI-only failures and environment drift

Git conflict and rejected push diagnosis Slow builds and cache misses Git repository bloat

Logging, tracing, and minimal reproduction Root-cause analysis and preventive fixes

16.1 Clean build works, incremental build fails

Likely causes:

Generated output not modeled.

Build cache key missing an input.

Race condition in parallel build.

Reproduce the incremental failure.

Capture the exact changed input.

Compare the dependency graph before and after.

Run a clean build only as a diagnostic, not as the final fix.

Add the missing dependency or correct the cache identity.

Test both clean and incremental paths.

The stale-header session in Module 2 (2.4) is exactly this failure: the clean build was correct, the incremental build silently reused main.o, and the fix was the missing prerequisite — not running make clean more often.

16.2 Undefined reference or unresolved symbol

Was the implementation compiled?

Was its object/library linked?

Is the library order correct for the linker?

Are symbol names mangled differently because of language/ABI mismatch?

Are compile-time feature flags inconsistent?

Is the symbol excluded by conditional compilation?

16.3 Header not found

Check include paths, target-scoped include configuration, case sensitivity, generated-header ordering, sysroot/toolchain, and whether the dependency was installed or fetched correctly.

16.4 CMake cache mismatch

If changing generator/compiler/toolchain, configure a fresh build directory. Inspect CMake's configure output and cache values. Avoid sharing one generated build directory across incompatible machines or toolchains.

16.5 Git rejected push

Typical causes include remote commits you do not have, branch protection, missing permission, or a non-fast-forward update.

Terminal session (Git 2.34.1, Ubuntu 22.04 on WSL2; the same practice repository throughout the Git modules, with identity Example Developer and commit dates fixed to 1 January 2026 so the IDs are reproducible): a teammate pushes to the same branch first:

Integrate remote work according to team policy, then push. Do not use force push as a reflex. If rewriting a branch is explicitly allowed, prefer --force-with-lease over an unconditional force push and understand its implications.

git fetch origin
git log --oneline --graph --decorate --all

$ git push -u origin feature/can-timeout
To /tmp/bsvc/remote/project.git
 * [new branch]      feature/can-timeout -> feature/can-timeout
Branch 'feature/can-timeout' set up to track remote branch 'feature/can-timeout' from 'origin'.
$ (cd ../teammate && git fetch -q && git switch -q feature/can-timeout && mkdir -p docs && echo 'Retries are configurable.' > docs/can.md && git add docs && git commit -qm 'docs: describe CAN retries' && git push -q)
$ echo 'Timeout is 500 ms.' >> README.md && git commit -qam 'docs: mention timeout'
$ git push
To /tmp/bsvc/remote/project.git
 ! [rejected]        feature/can-timeout -> feature/can-timeout (fetch first)
error: failed to push some refs to '/tmp/bsvc/remote/project.git'
hint: Updates were rejected because the remote contains work that you do
hint: not have locally. This is usually caused by another repository pushing
hint: to the same ref. You may want to first integrate the remote changes
hint: (e.g., 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
[exit status 1]
$ git fetch origin
From /tmp/bsvc/remote/project
   bc036e5..b5d230b  feature/can-timeout -> origin/feature/can-t
…

16.6 Recovering lost work

If a commit was moved out of the current branch, the reflog may still reference it. Create a recovery branch before experimenting. Recovery is not guaranteed if objects have been pruned or were never stored.

git reflog
git branch recovery-point <commit-id>

$ git log --oneline -1
16d6c73 notes: step 1
$ git reflog -4
16d6c73 HEAD@{0}: reset: moving to HEAD~1
b3a5a82 HEAD@{1}: reset: moving to ORIG_HEAD
16d6c73 HEAD@{2}: reset: moving to HEAD~1
b3a5a82 HEAD@{3}: reset: moving to ORIG_HEAD
$ git branch recovery-point HEAD@{1}
$ git log --oneline -1 recovery-point && cat notes.md
b3a5a82 notes: step 2
step 1
$ git show recovery-point:notes.md
step 2

16.7 Slow build

Which targets rebuild?

Is configuration repeated unnecessarily?

Are dependencies downloaded every time?

Is the critical path dominated by one large target?

Are caches invalidated by volatile inputs?

Is parallelism limited by missing graph edges or resource pressure?

Do not add more parallel jobs blindly; excessive concurrency can increase memory pressure or trigger races.

16.8 Git repository bloat

Investigate large files and history. Removing a large file from the latest commit does not remove it from older history. History rewriting may be needed but affects all collaborators and remote references. Coordinate a migration plan, back up the repository, communicate new clone instructions, and consider LFS or external artifact storage.

16.9 Root-cause template

Problem:
Expected behavior:
Observed behavior:
Environment/tool versions:
Exact command:
Minimal reproduction:
Logs and relevant diff:
Root cause:
Corrective change:
Regression test:
Preventive action:

Quiz

? “Undefined reference to `sensor_scale`” appears only when the library is listed first. What is the cause?
+ Link order: the archive was scanned before the reference existed. :: Module 3 reproduced this.
- The header is missing. :: That fails at compile time, not link time.
- The function is `static`. :: Then it would fail in every order.
- CMake cache mismatch. :: This was a direct `cc` command.

? A push is rejected. What should you avoid as a reflex?
+ `git push --force` :: It can overwrite others' work; prefer integrating, or `--force-with-lease` only when rewriting is agreed.
- `git fetch` :: Fetching is the safe first step.
- `git log --graph` :: Inspecting is safe.
- Asking the team :: Always fine.

? Which belong in a root-cause write-up? (select all that apply)
+ The exact command and tool versions :: Needed to reproduce.
+ A minimal reproduction :: Isolates the cause.
+ A regression test or preventive control :: Stops recurrence.
- Only the final fix :: Without cause and prevention the problem returns.