Gradient descent finds the cubic-regularized nonconvex Newton step

Yair Carmon, John Duchi

Research output: Contribution to journalArticlepeer-review

Abstract

We consider the minimization of a nonconvex quadratic form regularized by a cubic term, which may exhibit saddle points and a suboptimal local minimum. Nonetheless, we prove that, under mild assumptions, gradient descent approximates the global minimum to within ε accuracy in O(ε1 log(1/ε)) steps for large ε and O(log(1/ε)) steps for small ε (compared to a condition number we define), with at most logarithmic dependence on the problem dimension. When we use gradient descent to approximate the cubic-regularized Newton step, our result implies a rate of convergence to second-order stationary points of general smooth nonconvex functions.

Original languageEnglish
Pages (from-to)2146-2178
Number of pages33
JournalSIAM Journal on Optimization
Volume29
Issue number3
DOIs
StatePublished - 2019
Externally publishedYes

Keywords

  • Cubic regularization
  • Global optimization
  • Gradient descent
  • Newton's method
  • Nonasymptotic rate of convergence
  • Nonconvex quadratics
  • Power method
  • Trust region methods

Fingerprint

Dive into the research topics of 'Gradient descent finds the cubic-regularized nonconvex Newton step'. Together they form a unique fingerprint.

Cite this