Broyden–Fletcher–Goldfarb–Shanno algorithm
In numerical optimization, the Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm is an iterative method for solving unconstrained nonlinear optimization problems.
The BFGS method approximates Newton's method, a class of hill-climbing optimization techniques that seeks a stationary point of a (preferably twice continuously differentiable) function. For such problems, a necessary condition for optimality is that the gradient be zero. Newton's method and the BFGS methods are not guaranteed to converge unless the function has a quadratic Taylor expansion near an optimum. These methods use both the first and second derivatives of the function. However, BFGS has proven to have good performance even for non-smooth optimizations.[1]
In quasi-Newton methods, the Hessian matrix of second derivatives doesn't need to be evaluated directly. Instead, the Hessian matrix is approximated using rank-one updates specified by gradient evaluations (or approximate gradient evaluations). Quasi-Newton methods are generalizations of the secant method to find the root of the first derivative for multidimensional problems. In multi-dimensional problems, the secant equation does not specify a unique solution, and quasi-Newton methods differ in how they constrain the solution. The BFGS method is one of the most popular members of this class.[2] Also in common use is L-BFGS, which is a limited-memory version of BFGS that is particularly suited to problems with very large numbers of variables (e.g., >1000). The BFGS-B[3] variant handles simple box constraints.
Contents
Rationale
The search direction pk at stage k is given by the solution of the analogue of the Newton equation
where is an approximation to the Hessian matrix which is updated iteratively at each stage, and is the gradient of the function evaluated at xk. A line search in the direction pk is then used to find the next point xk+1. Instead of requiring the full Hessian matrix at the point xk+1 to be computed as Bk+1, the approximate Hessian at stage k is updated by the addition of two matrices.
Both Uk and Vk are symmetric rank-one matrices but have different (matrix) bases. The symmetric rank one assumption here means that we may write
So equivalently, Uk and Vk construct a rank-two update matrix which is robust against the scale problem often suffered in the gradient descent searching (e.g., in Broyden's method).
The quasi-Newton condition imposed on this update is
Algorithm
From an initial guess and an approximate Hessian matrix the following steps are repeated as converges to the solution.
- Obtain a direction by solving:
- Perform a line search to find an acceptable stepsize in the direction found in the first step, then update
- Set
denotes the objective function to be minimized. Convergence can be checked by observing the norm of the gradient, . Practically, can be initialized with , so that the first step will be equivalent to a gradient descent, but further steps are more and more refined by , the approximation to the Hessian.
The first step of the algorithm is carried out using the inverse of the matrix , which is usually obtained efficiently by applying the Sherman–Morrison formula to the fifth line of the algorithm, giving
This can be computed efficiently without temporary matrices, recognizing that is symmetric, and that and are scalar, using an expansion such as
In statistical estimation problems (such as maximum likelihood or Bayesian inference), credible intervals or confidence intervals for the solution can be estimated from the inverse of the final Hessian matrix. However, these quantities are technically defined by the true Hessian matrix, and the BFGS approximation may not converge to the true Hessian matrix.
Implementations
The GSL implements BFGS as gsl_multimin_fdfminimizer_vector_bfgs2. Ceres Solver implements both BFGS and L-BFGS. In SciPy, the scipy.optimize.fmin_bfgs function implements BFGS. It is also possible to run BFGS using any of the L-BFGS algorithms by setting the parameter L to a very large number.
Octave uses BFGS with a double-dogleg approximation to the cubic line search.
In R, the BFGS algorithm (and the L-BFGS-B version that allows box constraints) is implemented as an option of the base function optim().
In the MATLAB Optimization Toolbox, the fminunc function uses BFGS with cubic line search when the problem size is set to "medium scale."
A high-precision arithmetic version of BFGS (pBFGS), implemented in C++ and integrated with the high-precision arithmetic package ARPREC is robust against numerical instability (e.g. round-off errors).
Another C++ implementation of BFGS, along with L-BFGS, L-BFGS-B, CG, and Newton's method) using Eigen (C++ library) are available on github under the MIT License here.
BFGS and L-BFGS are also implemented in C as part of the open-source Gnu Regression, Econometrics and Time-series Library (gretl).
See also
- Quasi-Newton methods
- Davidon–Fletcher–Powell formula
- L-BFGS
- Gradient descent
- Nelder–Mead method
- Levenberg–Marquardt algorithm
- Pattern search (optimization)
- BHHH algorithm
Notes
- ↑ Lua error in package.lua at line 80: module 'strict' not found.
- ↑ Nocedal & Wright (2006), page 24
- ↑ Lua error in package.lua at line 80: module 'strict' not found.
Bibliography
- Lua error in package.lua at line 80: module 'strict' not found.
- Lua error in package.lua at line 80: module 'strict' not found.
- Lua error in package.lua at line 80: module 'strict' not found.
- Lua error in package.lua at line 80: module 'strict' not found.
- Lua error in package.lua at line 80: module 'strict' not found.
- Lua error in package.lua at line 80: module 'strict' not found.
- Lua error in package.lua at line 80: module 'strict' not found.
- Lua error in package.lua at line 80: module 'strict' not found.
- Lua error in package.lua at line 80: module 'strict' not found.
- Lua error in package.lua at line 80: module 'strict' not found.
External links
- Source code of high-precision BFGS A C++ source code of BFGS with high-precision arithmetic