Preface
1 Introduction
Mathematical Formulation
Example�� A Transportation Problem
Continuous versus Discrete Optimization
Constrained and Unconstrained Optimization
Global and Local Optimization
Stochastic and Deterministic Optimization
Optimization Algorithms
Convexity
Notes and References

2 Fundamentals of Unconstrained Optimization
2��1 What Is a Solution��
Recognizing a Local Minimum
Nonsmooth Problems
2��2 Overview of Algorithms
Two Strategies�� Line Search and Trust Region
Search Directions for Line Search Methods
Models for Trust��Region Methods
Scaling
Rates of Convergence
R��Rates of Convergence
Notes and References
Exercises

3 Line Search Methods
3��1 Step Length
The Wolfe Conditions
The Goldstein Conditions
Sufficient Decrease and Backtracking
3��2 Convergence of Line Search Methods
3��3 Rate of Convergence
Convergence Rate of Steepest Descent
Quasi��Newton Methods
Newton's Method
Coordinate Descent Methods
3��4 Step��Length Selection Algorithms
Interpolation
The Initial Step Length
A Line Search Algorithm for the Wolfe Conditions
Notes and References
Exerases

4 Trust��Region Methods
Outline of the Algorithm
4��1 The Cauchy Point and Related Algorithms
The Cauchy Point
Improving on the Cauchy Point
The DoglegMethod
Two��Dimensional Subspace Minimization
Steihaug's Approach
4��2 Using Nearly Exact Solutions to the Subproblem
Characterizing Exact Solutions
Calculating Nearly Exact Solutions
The Hard Case
Proof of Theorem 4��3
4��3 Global Convergence
Reduction Obtained by the Cauchy Point
Convergence to Stationary Points
Convergence of Algorithms Based on Nearly Exact Solutions
4��4 Other Enhancements
Scaling
Non��Euclidean Trust Regions
Notes and References
Exercises

5 Conjugate Gradient Methods
5��1 The Linear Conjugate Gradient Method
Conjugate Direction Methods
Basic Properties of the Conjugate Gradient Method
A Practical Form of the Conjugate Gradient Method
Rate of Convergence
Preconditioning
Practical Preconditioners
5��2 Nonlinear Conjugate Gradient Methods
The Fletcher��Reeves Method
The Polak��Ribiere Method
Quadratic Termination and Restarts
Numerical Performance
Behavior of the Fletcher��Reeves Method
Global Convergence
Notes and References
Exerases

6 Practical Newton Methods
6��1 Inexact Newton Steps
6��2 Line Search Newton Methods
Line Search Newton��CG Method
Modified Newton's Method
6��3 Hessian Modifications
Eigenvalue Modification
Adding a Multiple of the Identity
Modified Cholesky Factorization
Gershgorin Modification
Modified Symmetric Indefinite Factorization
6��4 Trust��Region Newton Methods
Newton��Dogleg and Subspace��Minimization Methods
Accurate Solution of the Trust��Region Problem
Trust��Region Newton��CG Method
Preconditioning the Newton��CG Method
Local Convergence of Trust��Region Newton Methods
Notes and References
Exerases

7 Calculating Derivatives
7��1 Finite��Difference Derivative Approximations
Approximating the Gradient
Approximating a Sparse Jacobian
Approximatingthe Hessian
Approximating a Sparse Hessian
7��2 Automatic Differentiation
An Example
The Forward Mode
The Reverse Mode
Vector Functions and Partial Separability
Calculating Jacobians of Vector Functions
Calculating Hessians�� Forward Mode
Calculating Hessians�� Reverse Mode
Current Limitations
Notes and References
Exercises

8 Quasi��Newton Methods
8��1 The BFGS Method
Properties ofthe BFGS Method
Implementation
8��2 The SR1 Method
Properties of SRl Updating
8��3 The Broyden Class
Properties ofthe Broyden Class
8��4 Convergence Analysis
Global Convergence ofthe BFGS Method
Superlinear Convergence of BFGS
Convergence Analysis of the SR1 Method
Notes and References
Exercises

9 Large��Scale Quasi��Newton and Partially Separable Optimization
9��1 Limited��Memory BFGS
Relationship with Conjugate Gradient Methods
9��2 General Limited��Memory Updating
Compact Representation of BFGS Updating
SR1 Matrices
Unrolling the Update
9��3 Sparse Quasi��Newton Updates
9��4 Partially Separable Functions
A Simple Example
Internal Variables
9��5 Invariant Subspaces and Partial Separability
Sparsity vs��Partial Separability
Group Partial Separability
9��6 Algorithms for Partially Separable Functions
Exploiting Partial Separabilityin Newton's Method
Quasi��Newton Methods for Partially Separable Functions
Notes and References
Exercises
����
10 Nonlinear Least��Squares Problems
11 Nonlinear Equations
12 Theory of Constrained Optimization
13 Linear Programming�� The Simplex Method
14 Linear Programming��Interior��Point Methods
15 Fundamentals of Algorithms for Nonlinear Constrained Optimization
16 Quadratic Programnung
17 Penalty�� Barrier�� and Augmented Lagrangian Methods
18 Sequential Quadratic Programming
A Background Material
References
Index