Optimization tutorial

This archive includes a set of functions introducing into optimization and line search techniques. It was designed for educational purposes.

Most of the functions run as script on toy problems. It is possible to visualize the line search and experiment with different update rules for the inverse Hessian in order to understand the optimization process.

This package includes

* conjugate gradient
* BFGS algorithm
* LBFGS algorithm
* Levenberg Marquart algorithm
* backtraicking Armijo line search
* line search enforcing strong Wolfe conditions
* line search bases on a 1D quadratic approximation of the objective function
* a function for naive numerical differentation

Have fun!

References:

* Nocedal & Wright: Numerical optimizaion