Automatic Differentiation & Optimization

DifferentiationInterface.jl gives one API over all AD backends (gradient(f, AutoForwardDiff(), x)).

32.1 Automatic differentiation

DifferentiationInterface.jl gives one API over all AD backends (gradient(f, AutoForwardDiff(), x)). ChainRules.jl defines custom differentiation rules.

using ForwardDiff                         # forward mode (few inputs)
f(x) = sum(sin.(x) .* x.^2)
ForwardDiff.gradient(f, [1.0, 2.0])
ForwardDiff.derivative(x -> x^3, 2.0)     # 12.0
ForwardDiff.hessian(f, [1.0, 2.0])
ForwardDiff.jacobian(x -> [x[1]^2, x[1]*x[2]], [1.0, 2.0])

using Zygote                              # reverse mode (many inputs, ML)
gradient(x -> 3x^2 + 2x, 5.0)             # (32.0,)

using Enzyme                              # LLVM-level AD (fast, supports mutation)
autodiff(Reverse, x -> x^2, Active, Active(3.0))

32.2 Numerical optimization

using Optim
rosenbrock(x) = (1 - x[1])^2 + 100 * (x[2] - x[1]^2)^2
res = optimize(rosenbrock, [0.0, 0.0], BFGS(); autodiff=:forward)
Optim.minimizer(res), Optim.minimum(res)

using Optimization, OptimizationOptimJL   # unified interface (SciML)

32.3 Mathematical programming with JuMP

Solvers: HiGHS, GLPK, Ipopt (nonlinear), Gurobi, CPLEX, SCIP, Juniper, Pajarito. JuMP supports LP, MILP, QP, SOCP, SDP, and nonlinear problems.

using JuMP, HiGHS
model = Model(HiGHS.Optimizer)
@variable(model, x >= 0)
@variable(model, 0 <= y <= 3)
@objective(model, Max, 12x + 20y)
@constraint(model, c1, 6x + 8y <= 100)
@constraint(model, c2, 7x + 12y <= 120)
optimize!(model)
value(x), value(y), objective_value(model), dual(c1)

32.4 Root finding & other numerics

Roots.jl (find_zero(f, (0, 2))), NLsolve.jl / NonlinearSolve.jl, QuadGK.jl / Integrals.jl (quadgk(sin, 0, π)), Interpolations.jl, FFTW.jl (fft), SpecialFunctions.jl, Polynomials.jl, Symbolics.jl (computer algebra), IntervalArithmetic.jl, Unitful.jl (physical units: 1u"m" / 2u"s"), Measurements.jl (error propagation).