Correlation & Regression
Relationships between variables. Learn covariance and the correlation coefficient r, simple linear regression by least squares, slope and intercept, residuals and R², prediction vs extrapolation, multiple and non-linear regression, and why correlation is not causation.
Calibrating a sensor, estimating fuel use from speed, or predicting strength from curing time all require fitting a relationship to data. Regression quantifies it; correlation measures its strength.
An analogy
Regression is like stretching a rubber band through a cloud of points so that it sits as close as possible to all of them at once.
Scatter and correlation
r near ±1: strong linear relationship; near 0: no linear relationship (could still be non-linear). r is unitless and symmetric in x and y.
Least squares fit
Minimises the sum of squared residuals. Always plot residuals: patterns (curves, funnels) mean the model is wrong or variance isn't constant.
Correlation is not causation
Two variables can correlate because a third factor drives both (ice-cream sales and drowning both rise in summer), by coincidence, or with reversed cause. Only controlled experiments establish causation.
Beyond simple regression
Multiple regression (several predictors), polynomial and non-linear fits, transformations (log-log for power laws), regularisation, and prediction intervals. Never extrapolate far outside the data range.
Formulas
- Slope & intercept:
b₁ = Sxy/Sxx, b₀ = ȳ − b₁x̄ - Correlation:
r = Sxy/√(Sxx Syy) - R²:
R² = 1 − SS_res/SS_tot (= r² for simple regression) - Sums:
Sxy = Σ(x − x̄)(y − ȳ), Sxx = Σ(x − x̄)²
Worked example: Sensor calibration
Given: x = 1…5, y = 2.1, 3.9, 6.2, 7.8, 10.1
Find: Line, r, R².
- x̄ = 3, ȳ = 6.02; Sxy = 19.9, Sxx = 10
- b₁ = 1.99, b₀ = 0.05
- r = 0.9987, R² = 0.9973
Answer: ŷ = 0.05 + 1.99x; R² = 0.997
Worked example: Spurious correlation
Given: Cooling-fan failures correlate with ice-cream sales (r = 0.8)
Find: Interpretation.
- Both rise with summer temperature (confounder)
Answer: No causal link — control for temperature
Common mistakes
- Wrong: Inferring causation from correlation. — Right: Use experiments or causal reasoning.
- Wrong: Extrapolating beyond the data. — Right: Models are valid only in the fitted range.
Applications
- Calibration: Sensor output vs reference.
- Empirical models: Fuel use, wear rates, strength.
- Process improvement: Identifying influential factors.
Key takeaways
- r measures linear association.
- b₁ = Sxy/Sxx, b₀ = ȳ − b₁x̄.
- Check residual plots and R².
- Correlation ≠ causation; don't extrapolate.
Going further
- Multiple regression: Matrix form β = (XᵀX)⁻¹Xᵀy.
- Prediction intervals: Uncertainty of individual predictions.
- Causal inference: DAGs and controlled experiments.