Speaker Prof. Ian R. Petersen
Research School of Engineering,
Australian National University, Australia
Date｜Time Feb 25 (Wednesday), 2026｜14:00
Zoom https://snu-ac-kr.zoom.us/my/jingyu.lee
Abstract
This seminar will describe some recent work in which control theory methods are used to analyze some gradient based optimization methods. Gradient based optimization methods are widely used in many areas of optimization the area such as in machine learning and artificial intelligence approaches. The heavy ball optimization method is analyzed using the discrete time circle criterion for a class of non-convex cost functions and it is shown that convergence could only be guaranteed for cost functions with a condition number which less than a fixed maximum value. Then the coefficients of the original heavy ball algorithm are modified so that the circle criterion could be used to guarantee convergence for all cost functions in the class being considered, at the expense of a slightly slower convergence rate. Also, an additional term is added to the heavy ball algorithm to obtain an improved guaranteed convergence rate. This leads to an optimization algorithm with a similiar form to the triple momentum method (TMM). However, with our choice of parameters, a faster convergence rate is obtained than the TMM method. In addition, a control theory result on optimal gain margin is used to prove that the original heavy ball method has the fastest convergence rate of any linear gradient base optimization method for strictly convex quadratic cost functions. In addition, the case of quadratic cost functions with moving optimal points is addressed. Such problems arise in the area of online convex optimization algorithms. By introducing integral action into the optimization algorithm, an algorithm can be obtained with zero steady state error. For the case of optimization algorithms having the same form as the heavy ball method, this leads to an algorithm with a convergence rate which is slower than the standard gradient algorithm. However, by applying a control theory method for optimal gain margin, a more complicated algorithm can be derived with the optimal convergence rate.
Biography
This seminar will describe some recent work in which control theory methods are used to analyze some gradient based optimization methods. Gradient based optimization methods are widely used in many areas of optimization the area such as in machine learning and artificial intelligence approaches. The heavy ball optimization method is analyzed using the discrete time circle criterion for a class of non-convex cost functions and it is shown that convergence could only be guaranteed for cost functions with a condition number which less than a fixed maximum value. Then the coefficients of the original heavy ball algorithm are modified so that the circle criterion could be used to guarantee convergence for all cost functions in the class being considered, at the expense of a slightly slower convergence rate. Also, an additional term is added to the heavy ball algorithm to obtain an improved guaranteed convergence rate. This leads to an optimization algorithm with a similiar form to the triple momentum method (TMM). However, with our choice of parameters, a faster convergence rate is obtained than the TMM method. In addition, a control theory result on optimal gain margin is used to prove that the original heavy ball method has the fastest convergence rate of any linear gradient base optimization method for strictly convex quadratic cost functions. In addition, the case of quadratic cost functions with moving optimal points is addressed. Such problems arise in the area of online convex optimization algorithms. By introducing integral action into the optimization algorithm, an algorithm can be obtained with zero steady state error. For the case of optimization algorithms having the same form as the heavy ball method, this leads to an algorithm with a convergence rate which is slower than the standard gradient algorithm. However, by applying a control theory method for optimal gain margin, a more complicated algorithm can be derived with the optimal convergence rate.
