Model Evaluation, Validation & Metrics

Knowing whether a model is actually good. Learn the confusion matrix, accuracy, precision, recall, F1, specificity, ROC and AUC, precision–recall curves, threshold selection and cost-sensitive decisions, regression metrics (MAE, RMSE, R², MAPE), calibration, cross-validation (k-fold, stratified, time-series), data leakage, statistical significance of improvements, and evaluating LLMs and ranking systems — with NumPy code computing a full confusion matrix, F1 and ROC-AUC on an imbalanced problem.

A model is only as trustworthy as its evaluation. Choosing the wrong metric — like accuracy on imbalanced data — can make a useless model look excellent.

Good evaluation mirrors real use: the right data split, the right metric for the business cost of errors, and honest estimates of uncertainty.

An analogy

Evaluating a fraud model by accuracy is like judging a smoke detector by how often it stays quiet: one that never beeps is 'right' 99.9% of the time — and useless in a fire. Recall asks: when there IS a fire, does it beep?

Classification metrics

Choose by the cost of errors:

Thresholds, ROC and PR curves

Most models output scores; the threshold turns them into decisions. The ROC curve plots TPR vs FPR across thresholds (AUC = ranking quality, insensitive to imbalance). The precision–recall curve is more informative when positives are rare. Choose the threshold from business costs (expected cost minimisation) or capacity (e.g. top 100 alerts per day). Calibration checks whether predicted probabilities match observed frequencies (reliability diagrams, Brier score).

Validation strategies

Hold-out split for large data; k-fold cross-validation for smaller data (stratified to keep class balance); time-series split (train on past, test on future) for temporal data; group k-fold when multiple rows belong to the same user/patient. Nested CV separates tuning from evaluation. Leakage sources: future information, duplicates across splits, target encoding without folds, preprocessing fitted on all data.

Beyond single numbers

Report confidence intervals (bootstrap) and test significance when comparing models (McNemar, paired tests). Slice metrics by segment (region, device, demographic) to find hidden failures and fairness issues. Ranking/recommendation: precision@k, recall@k, NDCG, MAP. LLM evaluation: exact match, BLEU/ROUGE (limited), human preference, LLM-as-judge with rubrics, task-specific test suites, and online A/B tests measuring real user outcomes.

Formulas

Worked example: Precision & recall

Given: TP 40, FP 10, FN 20

Find: P, R, F1.

Answer: P 0.80, R 0.67, F1 0.73

Worked example: RMSE

Given: Errors 2, −3, 1, 4

Find: RMSE and MAE.

Answer: RMSE 2.74, MAE 2.5

Worked example: Expected cost

Given: FP cost ₹50 (review), FN cost ₹5000 (fraud loss); FP 9, FN 2

Find: Cost.

Answer: ₹10 450

Confusion matrix, F1 and AUC

import numpy as np
# Confusion matrix, precision/recall/F1 and ROC-AUC for a fraud-style imbalanced problem
y_true = np.array([0]*90 + [1]*10)
rng = np.random.default_rng(7)
scores = np.where(y_true == 1, rng.normal(0.7, 0.15, 100), rng.normal(0.35, 0.15, 100))
pred = (scores > 0.5).astype(int)
TP = int(((pred == 1) & (y_true == 1)).sum()); FP = int(((pred == 1) & (y_true == 0)).sum())
FN = int(((pred == 0) & (y_true == 1)).sum()); TN = int(((pred == 0) & (y_true == 0)).sum())
prec, rec = TP / (TP + FP), TP / (TP + FN)
print(f"TP {TP} FP {FP} FN {FN} TN {TN}")
print(f"accuracy {(TP+TN)/100:.2f}, precision {prec:.2f}, recall {rec:.2f}, F1 {2*prec*rec/(prec+rec):.2f}")
# ROC-AUC = probability a random positive scores above a random negative
pos, neg = scores[y_true == 1], scores[y_true == 0]
auc = np.mean(pos[:, None] > neg[None, :])
print(f"ROC-AUC {auc:.3f}; a model predicting all 0 would have accuracy 0.90 but recall 0")
# TP 8 FP 9 FN 2 TN 81
# accuracy 0.89, precision 0.47, recall 0.80, F1 0.59
# ROC-AUC 0.943; a model predicting all 0 would have accuracy 0.90 but recall 0

This model catches 8 of 10 frauds (recall 0.80) but half its alerts are false (precision 0.47). Its accuracy (0.89) is actually below the useless 'never fraud' model (0.90) — proof that accuracy misleads on imbalanced data. AUC 0.94 shows the scores rank well; moving the threshold trades precision against recall.

Common mistakes

Applications

Key takeaways

Going further