Optimization is the mathematical study of finding the best possible solution among all feasible options โ the maximum profit, minimum cost, shortest route, or most accurate prediction. From simple calculus-based methods for smooth functions to the gradient descent algorithms training today's largest artificial intelligence systems, optimization is one of the most broadly and practically applied branches of mathematics.
A farmer has a fixed length of fencing and wants to enclose the largest possible rectangular field. A delivery company wants the shortest route visiting all its stops. A company wants to set a price maximising total profit. All of these are optimization problems โ situations where you're not just looking for any valid answer, but specifically the best one, according to some precise, clearly defined measure of "best."
For smooth, well-behaved functions, calculus offers a direct and elegant strategy: a maximum or minimum can only occur where the function's derivative is exactly zero (a "flat" point on the graph โ see the Derivative entry) โ or possibly at a domain boundary. Checking these specific candidate points reveals where the true best (or worst) values genuinely occur.
For a function of one variable, critical points (where f'(x)=0) are checked using the second derivative: if f''(x)>0, it's a local minimum; if f''(x)<0, a local maximum. For functions of several variables, the equivalent condition involves the gradient (see the Derivative entry) being the zero vector, with the nature of the critical point determined by the Hessian matrix of second partial derivatives (see the Matrix entry).
Many real optimization problems involve explicit constraints โ like the farmer's fixed fence length. The method of Lagrange multipliers (Joseph-Louis Lagrange, 1788) handles this elegantly: to maximise f(x,y) subject to a constraint g(x,y)=c, introduce a new variable ฮป (the "multiplier") and solve the system โf=ฮปโg together with the constraint itself. This transforms a constrained optimization problem into a system of ordinary equations, and the multiplier ฮป itself carries a useful economic interpretation โ roughly, "how much would the optimal value improve if the constraint were relaxed slightly."
When both the objective function and all constraints are linear, the problem is a linear program. George Dantzig's simplex method (1947) solves such problems remarkably efficiently by systematically moving between the "corners" of the geometric region defined by the constraints, always improving the objective at each step until reaching the true optimum. Linear programming is extensively used throughout logistics, manufacturing, resource allocation, and operations research more broadly.
When calculus-based exact solutions aren't practically available (as is typical for the extremely complex, high-dimensional functions in modern machine learning), gradient descent offers an iterative numerical alternative: repeatedly take small steps in the direction that most steeply decreases the objective function (the negative gradient direction), gradually approaching a local minimum. This deceptively simple algorithm, when combined with a technique called backpropagation for efficiently computing gradients, is the fundamental workhorse training essentially every modern deep learning neural network.
A function is convex if the line segment connecting any two points on its graph always lies on or above the graph itself (informally: it curves consistently "upward," like a bowl shape, with no separate local dips). For convex optimization problems, any local minimum is automatically guaranteed to be the true global minimum โ a hugely valuable property that dramatically simplifies both the theoretical analysis and the practical solving of such problems, and which motivates substantial research effort into reformulating otherwise difficult non-convex problems into convex ones wherever practically possible.
Many optimization problems have an associated "dual" problem, related to the original ("primal") problem in a deep and useful way: the dual problem's optimal value provides a rigorous bound on the primal problem's optimal value (weak duality), and for convex problems satisfying certain regularity conditions, the two optimal values are exactly equal (strong duality). This duality framework โ closely related in spirit to the Lagrange multiplier method described above โ provides both powerful practical solution algorithms and, often just as importantly, deep theoretical insight into the fundamental structure of a given optimization problem.
Ordinary optimization finds the best specific number or point. The calculus of variations instead finds the best entire function or curve โ for instance, the specific curve connecting two points that a ball would follow to roll between them in minimum time (the historic brachistochrone problem, posed by Johann Bernoulli in 1696 and famously solved by several leading mathematicians of the era, including Newton, working anonymously overnight). This more general framework directly underlies the Principle of Least Action in physics (see the Noether's Theorem entry), where physical systems can be shown to always evolve along whichever specific path makes a certain quantity called the "action" mathematically stationary (typically minimal).
Most real-world machine learning problems are genuinely non-convex, meaning gradient descent-style methods can potentially become permanently stuck at a merely local minimum rather than reliably finding the true global minimum. Considerable modern research effort focuses on understanding why gradient descent nonetheless tends to work remarkably well in practice for training large neural networks despite this theoretical concern โ including surprising and still not fully understood findings that, in the extremely high-dimensional parameter spaces typical of modern deep learning models, poor local minima appear to be comparatively rare in practice relative to what naive lower-dimensional geometric intuition might otherwise suggest.
Many important optimization problems involve choosing from a finite (but often astronomically large) set of discrete possibilities rather than continuously varying real numbers โ such as the Travelling Salesman Problem (see the Graph Theory entry). Many such combinatorial optimization problems are provably NP-hard (see the Big-O Notation entry for the relevant P vs NP context), meaning no known algorithm can reliably solve every instance efficiently as problem size grows โ motivating extensive practical use of heuristic and approximation algorithms that reliably find good, though not always strictly provably optimal, solutions within a reasonable amount of computation time.