Numerical Foundations & Errors

Every numerical answer is approximate. Learn exact vs approximate solutions, floating-point representation and machine epsilon, round-off and truncation error, absolute and relative error, error propagation, subtractive cancellation, conditioning vs stability, and convergence order.

Simulation software, controllers and spreadsheets all compute with finite precision. Understanding where errors come from — and how they grow — separates trustworthy engineering computation from nonsense with many decimal places.

An analogy

Numerical error is like measuring with a ruler that has a limited number of marks (round-off) and approximating a curved road with straight segments (truncation).

Types of error

Modelling error (wrong physics), data error (measurement), round-off (finite digits), truncation (approximating an infinite process — series, derivatives, integrals — by a finite one).

Floating point

IEEE 754 doubles have 53-bit mantissas (≈ 16 digits). 0.1 + 0.2 = 0.30000000000000004; 10¹⁶ + 1 − 10¹⁶ = 0. Never test floats with ==; use tolerances.

Cancellation

Subtracting nearly equal numbers loses significant digits: (1 − cos x)/x² at x = 10⁻⁸ evaluates to 0 instead of 0.5. Rewrite algebraically: 2 sin²(x/2)/x².

Conditioning, stability, convergence

Conditioning is a property of the problem (sensitivity); stability is a property of the algorithm (does it amplify errors?). Convergence order p tells how fast errors shrink as h → 0 or iterations increase.

Formulas

• Errors: E_abs = |x − x̃|, E_rel = E_abs/|x|
• Floating point: x̃ = x(1 + δ), |δ| ≤ ε_mach/2
• Truncation order: E ≈ C hᵖ → halving h divides error by 2ᵖ
• Conditioning: relative output error ≈ κ × relative input error

Worked example: Relative error

Given: π ≈ 22/7

Find: Absolute and relative errors.

• 22/7 = 3.142857
• E_abs = 0.001264
• E_rel = 4.0 × 10⁻⁴

Answer: 0.04 % relative error

Worked example: Cancellation

Given: (1 − cos x)/x² at x = 10⁻⁸

Find: Computed vs true.

• cos(10⁻⁸) rounds to 1 exactly → 0/x² = 0
• Rewrite 2 sin²(x/2)/x² → 0.5

Answer: 0 (wrong) vs 0.5 (correct)

Common mistakes

• Wrong: Comparing floats with ==. — Right: Use |a − b| < tol·max(|a|, |b|).
• Wrong: Smaller step is always better. — Right: Round-off grows as h shrinks — there's an optimum.

Applications

• Simulation credibility: Error estimates for CFD/FEA results.
• Embedded code: Fixed/floating-point precision choices.
• Data processing: Avoiding cancellation in formulas.

Key takeaways

• Round-off vs truncation vs model error.
• ε_mach ≈ 2.2 × 10⁻¹⁶ for doubles.
• Avoid subtractive cancellation.
• Conditioning (problem) vs stability (algorithm).

Going further

• Kahan summation: Compensated summation for accuracy.
• Interval arithmetic: Guaranteed error bounds.
• Richardson extrapolation: Cancel leading truncation terms.