Unconstrained Optimisation

Find the best design without constraints. Learn objective functions and decision variables, local vs global optima, gradient and Hessian conditions, convexity, gradient descent and learning rates, Newton's method for optimisation, and line searches — with a Python gradient-descent example.

Minimising weight, cost, error or energy is at the heart of design, control tuning, curve fitting and machine learning. Unconstrained methods are the building blocks for all optimisation algorithms.

An analogy

Gradient descent is like walking downhill in fog: you feel the slope under your feet and step in the steepest downward direction until the ground is flat.

Formulating the problem

Choose decision variables (dimensions, gains), an objective (cost, weight, error) and check units. Maximising f is minimising −f.

Optimality conditions

Stationary points have zero gradient; the Hessian classifies them (positive definite → minimum, indefinite → saddle). Convex problems have a single global minimum — highly desirable.

Gradient descent

Simple and scalable (machine learning). Too large α diverges or oscillates; too small is slow. Ill-conditioned problems (elongated valleys) converge slowly — scaling and momentum help.

Newton and quasi-Newton

Newton's method uses curvature for quadratic convergence but needs the Hessian. Quasi-Newton (BFGS) approximates it; trust-region and line-search strategies make methods robust. Derivative-free: Nelder–Mead, genetic algorithms.

Formulas

• First-order condition: ∇f(x*) = 0
• Second-order condition: H(x*) positive definite ⇒ local minimum
• Gradient descent: xₖ₊₁ = xₖ − α∇f(xₖ)
• Newton's method: xₖ₊₁ = xₖ − H⁻¹∇f(xₖ)
• Convexity: f(λx + (1 − λ)y) ≤ λf(x) + (1 − λ)f(y) ⇒ local = global

Worked example: Stationary point

Given: f = (x − 3)² + 2(y + 1)²

Find: Minimum.

• ∇f = (2(x − 3), 4(y + 1)) = 0 → (3, −1)
• H = diag(2, 4) positive definite

Answer: Global minimum f = 0 at (3, −1)

Worked example: Maximum power transfer

Given: P(R) = V²R/(R + Rₛ)²

Find: Optimal R.

• dP/dR = V²(Rₛ − R)/(R + Rₛ)³ = 0

Answer: R = Rₛ

Gradient descent and a small LP (Python)

import numpy as np

# Minimise f(x, y) = (x - 3)^2 + 2 (y + 1)^2 by gradient descent
def grad(p):
    x, y = p
    return np.array([2 * (x - 3), 4 * (y + 1)])

p, lr = np.array([0.0, 0.0]), 0.1
for k in range(1, 201):
    step = lr * grad(p)
    p = p - step
    if np.linalg.norm(step) < 1e-8:
        break
print(f"gradient descent: x = {p[0]:.6f}, y = {p[1]:.6f} after {k} iterations")

# Linear programme (brute-force vertices): max 40x + 30y
# s.t. x + y <= 12, 2x + y <= 16, x, y >= 0
vertices = [(0, 0), (8, 0), (4, 8), (0, 12)]
best = max(vertices, key=lambda v: 40 * v[0] + 30 * v[1])
print("LP optimum at", best, "profit =", 40 * best[0] + 30 * best[1])
# gradient descent: x = 3.000000, y = -1.000000 after 82 iterations
# LP optimum at (4, 8) profit = 400

Gradient descent needs many small steps (82 here); Newton's method would jump to the minimum of this quadratic in one step. The LP part previews the next lessons: optima of linear programmes sit at vertices.

Common mistakes

• Wrong: Assuming a local minimum is global. — Right: Only guaranteed for convex problems; use multiple starts.
• Wrong: Unscaled variables. — Right: Normalise variables to similar magnitudes.

Applications

• Design optimisation: Minimum weight or cost designs.
• Parameter fitting: Least-squares model calibration.
• Machine learning: Training neural networks.

Key takeaways

• ∇f = 0; Hessian classifies.
• Convex ⇒ local = global.
• Gradient descent: choose α carefully.
• Newton/BFGS for fast convergence.

Going further

• BFGS: Quasi-Newton workhorse.
• Stochastic gradient descent: Large-scale ML training.
• Global optimisation: Simulated annealing, genetic algorithms.