The standard approach to control and optimization of stochastic systems is based on dynamic programming. Recently, an alternative approach to stochastic learning and optimization has been developed . The approach was based on sensitivity; it was first developed for discrete event dynamic systems and is being extended to continuous-time and continuous state systems. The basic idea is as follows: fundamentally, one can only compare the performance of two policies at a time; and therefore, when developing optimization theories and methodologies, one has to first study the difference of the performance of any two polices. It turns out that many results in optimization can be obtained by a direct comparison of the performance of any two policies based on this performance difference formula. We found that this “direct comparison” method is essential for optimization.

This approach has some advantages over the dynamic programming approach: It is simple and intuitive clear because it is based on a direct comparison of any two policies. The approach applies in the same way to different performance criteria, including finite and infinite-horizon problems. Furthermore, the approach brings some new insights that leads to new methods and results in control and optimization; for example, it may be applied to non-smooth functions and lead to a solution without using viscocity solution.

We may also develop event-based optimization and gradient-based learning, etc.