Performance of first-order methods for smooth convex minimization: A novel approach

Yoel Drori, Marc Teboulle*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

Abstract

We introduce a novel approach for analyzing the worst-case performance of first-order black-box optimization methods. We focus on smooth unconstrained convex minimization over the Euclidean space. Our approach relies on the observation that by definition, the worst-case behavior of a black-box optimization method is by itself an optimization problem, which we call the performance estimation problem (PEP). We formulate and analyze the PEP for two classes of first-order algorithms. We first apply this approach on the classical gradient method and derive a new and tight analytical bound on its performance. We then consider a broader class of first-order black-box methods, which among others, include the so-called heavy-ball method and the fast gradient schemes. We show that for this broader class, it is possible to derive new bounds on the performance of these methods by solving an adequately relaxed convex semidefinite PEP. Finally, we show an efficient procedure for finding optimal step sizes which results in a first-order black-box method that achieves best worst-case performance.

Original languageEnglish
Pages (from-to)451-482
Number of pages32
JournalMathematical Programming
Volume145
Issue number1-2
DOIs
StatePublished - Jun 2014

Keywords

  • Complexity
  • Duality
  • Fast gradient schemes
  • Heavy Ball method
  • Performance of first-order algorithms
  • Rate of convergence
  • Semidefinite relaxations
  • Smooth convex minimization

Fingerprint

Dive into the research topics of 'Performance of first-order methods for smooth convex minimization: A novel approach'. Together they form a unique fingerprint.

Cite this