1 Exact and Approximate Dynamic Programming Principles
1.1 AlphaZero, Off-Line Training, and On-Line Play
1.2 Deterministic Dynamic Programming
1.2.1 Finite Horizon Problem Formulation
1.2.2 The Dynamic Programming Algorithm
1.2.3 Approximation in Value Space
1.3 Stochastic Dynamic Programming
1.3.1 Finite Horizon Problems
1.3.2 Approximation in Value Space for Stochastic DP
1.3.3 Infinite Horizon Problems-An Overview
1.3.4 Infinite Horizon-Approximation in Value Space
1.3.5 Infinite Horizon-Policy Iteration, Rollout, andNewton's Method
1.4 Examples, Variations, and Simplifications
1.4.1 A Few Words About Modeling
1.4.2 Problems with a Termination State
1.4.3 State Augmentation, Time Delays, Forecasts, and Uncontrollable State Components
1.4.4 Partial State Information and Belief States
1.4.5 Multiagent Problems and Multiagent Rollout
1.4.6 Problems with Unknown Parameters-AdaptiveControl
1.4.7 Adaptive Control by Rollout and On-LineReplanning
1.5 Reinforcement Learning and Optimal Control-SomeTerminology
1.6 Notes and Sources
2 General Principles of Approximation in Value Space
2.1 Approximation in Value and Policy Space
2.1.1 Approximation in Value Space-One-Step and Multistep Lookahead
2.1.2 Approximation in Policy Space
2.1.3 Combined Approximation in Value and Policy Space
2.2 Approaches for Value Space Approximation
2.2.1 Off-Line and On-Line Implementations
2.2.2 Model-Based and Model-Free Implementations
2.2.3 Methods for Cost-to-Go Approximation
2.2.4 Methods for Expediting the Lookahead Minimization
2.3 Deterministic Rollout and the Policy Improvement Principle
2.3.1 On-Line Rollout for Deterministic Discrete Optimization
2.3.2 Using Multiple Base Heuristics-Parallel Rollout
2.3.3 The Simplified Rollout Algorithm
2.3.4 The Fortified Rollout Algorithm
2.3.5 Rollout with Multistep Lookahead
2.3.6 Rollout with an Expert
2.3.7 Rollout with Small Stage Costs and Long Horizon-Continuous-Time Rollout
2.4 Stochastic Rollout and Monte Carlo Tree Search
2.4.1 Simulation-Based Implementation of the Rollout Algorithm
2.4.2 Monte Carlo Tree Search
2.4.3 Randomized Policy Improvement by Monte Carlo Tree Search
2.4.4 The Effect of Errors in Rollout-Variance Reduction
2.4.5 Rollout Parallelization
2.5 Rollout for Infinite-Spaces Problems-Optimization Heuristics
2.5.1 Rollout for Infinite-Spaces Deterministic Problems
2.5.2 Rollout Based on Stochastic Programming
2.6 Notes and Sources
3 Specialized Rollout Algorithms
3.1 Model Predictive Control
3.1.1 Target Tubes and Constrained Controllability
3.1.2 Model Predictive Control with Terminal Cost
3.1.3 Variants of Model Predictive Control
3.1.4 Target Tubes and State-Constrained Rollout
3.2 Multiagent Rollout
3.2.1 Asynchronous and Autonomous Multiagent Rollout
3.2.2 Multiagent Coupling Through Constraints
3.2.3 Multiagent Model Predictive Control
3.2.4 Separable and Multiarmed Bandit Problems
3.3 Constrained Rollout-Deterministic Optimal Control
3.3.1 Sequential Consistency, Sequential Improvement, and the Cost Improvement Property
3.3.2 The Fortified Rollout Algorithm and Other Variations
3.4 Constrained Rollout-Discrete Optimization
3.4.1 General Discrete Optimization Problems
3.4.2 Multidimensional Assignment
3.5 Rollout for Surrogate Dynamic Programming and Bayesian Optimization
3.6 Rollout for Minimax Control
3.7 Notes and Sources
4 Learning Values and Policies
4.1 Parametric Approximation Architectures
4.1.1 Cost Function Approximation
4.1.2 Feature-Based Architectures
4.1.3 Training of Linear and Nonlinear Architectures
4.2 Neural Networks
4.2.1 Training of Neural Networks
4.2