Data Analysis and Statistics

accumarray(subs, vals, [], @fun) applies a function to groups of values sharing an index — MATLAB's compact "group by".

Concept

mean, median, mode, std, var, min, max, prctile/quantile (toolbox in MATLAB), iqr (toolbox)

NaN markers; mean(x, 'omitnan') (MATLAB), isnan, fillmissing, rmmissing (MATLAB) Median/IQR rules, isoutlier (MATLAB) Correlation, regression

corrcoef, cov, polyfit, \ for multiple regression accumarray, unique with index outputs, splitapply/groupsummary (MATLAB)

movmean, movmedian, smoothdata (MATLAB) histcounts (MATLAB), histc, hist z-score (x - mean(x)) / std(x), normalize (MATLAB)

rng(seed), rand, randn, randi, randperm accumarray(subs, vals, [], @fun) applies a function to groups of values sharing an index — MATLAB's compact "group by".

Why It Matters

Test benches produce thousands of samples; statistics separate real effects from noise, find faulty sensors, and quantify relationships (for example between temperature and current draw).

Syntax

R = corrcoef(x, y);          r = R(1, 2);
groupMeans = accumarray(groupIdx, values, [], @mean);
xs = movmean(x, 5);

Example

27.1 Cleaning, outliers, correlation, regression, grouping and smoothing

rng(42);
n = 200;
temp = 25 + 10 * rand(n, 1);                                      % ambient temperature
current = 5 + 0.4 * (temp - 25) + 0.5 * randn(n, 1);              % current rises with temperature
current([17 88]) = [30 -8];                                       % two faulty samples

fprintf('mean %.3f, median %.3f, std %.3f, min %.2f, max %.2f\n', mean(current), median(current), std(current), min(current), max(current))
q = sort(current);
iqr_v = q(round(0.75 * n)) - q(round(0.25 * n));
outliers = find(abs(current - median(current)) > 3 * iqr_v);
fprintf('outliers (3 x IQR from the median): rows %s\n', mat2str(outliers'))
clean = current; clean(outliers) = [];
tclean = temp; tclean(outliers) = [];
R = corrcoef(tclean, clean);
fprintf('correlation temperature vs current: %.4f\n', R(1, 2))
c = polyfit(tclean, clean, 1);
fprintf('regression: current = %.4f * temp %+.4f\n', c)

% grouping: mean current per temperature band with accumarray
band = floor((tclean - 25) / 2.5) + 1;                            % 4 bands of 2.5 degC
m = accumarray(band, clean, [], @mean);
cnt = accumarray(band, 1);
for k = 1:numel(m)
    fprintf('band %d (%.1f-%.1f C): %3d samples, mean current %.3f A
…

Expected Result

The two injected faulty samples (rows 17 and 88) were detected as outliers. After removing them, current and temperature are strongly correlated and the regression recovered a slope close to the true 0.4 A/°C. accumarray produced per-band means in one call. (The data come from a seeded random generator, so the exact numbers differ in MATLAB, but the conclusions are the same.)

mean 6.942, median 6.779, std 2.333, min -8.00, max 30.00
outliers (3 x IQR from the median): rows [17 88]
correlation temperature vs current: 0.9199
regression: current = 0.4035 * temp -5.1453
band 1 (25.0-27.5 C):  57 samples, mean current 5.497 A
band 2 (27.5-30.0 C):  45 samples, mean current 6.371 A
band 3 (30.0-32.5 C):  48 samples, mean current 7.460 A
band 4 (32.5-35.0 C):  48 samples, mean current 8.508 A
z-scores: mean -4.28e-16, std 1.0000
movmean([1 2 3 4 5], 3) = [1.5 2 3 4 4.5]
histogram counts per 2 A bin: [0 0 59 91 48 0 0]

Common Mistakes

Computing statistics on data containing NaN or obvious sensor faults.

Removing outliers without a documented rule.

Correlation ≠ causation.

Using std with the wrong normalization for the purpose.

Best Practices

Always plot the raw data first.

Use robust statistics (median, IQR) when outliers are possible.

Seed random generators for reproducible analyses.

Keep a record of every cleaning step.

Interview Relevance

Medium: "mean vs median?", "how do you detect outliers?", "what does the correlation coefficient tell you?", "how do you group data in MATLAB?".

Practice Questions

Replace NaN values in a signal by linear interpolation of their neighbours.

Compute the 5th and 95th percentiles of a dataset by sorting.

Group test results by bench number and compute pass rates with accumarray.

Fit a multiple linear regression y = a·x₁ + b·x₂ + c with the backslash operator.

Quick Quiz

Q1. (Multiple choice) Which statistic is robust to outliers? A) Mean B) Median C) Standard deviation D) Maximum Answer: B.

Q2. (Predict the output) movmean([1 2 3 4 5], 3) Answer: [1.5 2 3 4 4.5] — the window shrinks at the ends (verified).

Q3. (True/False) A correlation coefficient of 0.95 proves that temperature causes current changes.

Answer: False. It shows a strong linear relationship, not causation.

Q4. (Find the bug) mean([12.6 NaN 12.4]) returns NaN.

Answer: NaN propagates; remove NaNs first (x(~isnan(x))) or use the 'omitnan' flag in MATLAB.

Coding Challenge

For pass/fail results of five benches bench = [1 2 1 3 2 1 3 3] and passed = [1 0 1 1 1 0 1 1], compute the pass rate per bench with accumarray.

bench = [1 2 1 3 2 1 3 3]';
passed = [1 0 1 1 1 0 1 1]';
rate = accumarray(bench, passed, [], @mean);
for b = 1:numel(rate)
    fprintf('bench %d: %.0f%% passed\n', b, rate(b) * 100)
end