TY - JOUR

T1 - Lower bounds for non-convex stochastic optimization

AU - Arjevani, Yossi

AU - Carmon, Yair

AU - Duchi, John C.

AU - Foster, Dylan J.

AU - Srebro, Nathan

AU - Woodworth, Blake

N1 - Publisher Copyright:
© 2022, Springer-Verlag GmbH Germany, part of Springer Nature and Mathematical Optimization Society.

PY - 2023/5

Y1 - 2023/5

N2 - We lower bound the complexity of finding ϵ-stationary points (with gradient norm at most ϵ) using stochastic first-order methods. In a well-studied model where algorithms access smooth, potentially non-convex functions through queries to an unbiased stochastic gradient oracle with bounded variance, we prove that (in the worst case) any algorithm requires at least ϵ- 4 queries to find an ϵ-stationary point. The lower bound is tight, and establishes that stochastic gradient descent is minimax optimal in this model. In a more restrictive model where the noisy gradient estimates satisfy a mean-squared smoothness property, we prove a lower bound of ϵ- 3 queries, establishing the optimality of recently proposed variance reduction techniques.

AB - We lower bound the complexity of finding ϵ-stationary points (with gradient norm at most ϵ) using stochastic first-order methods. In a well-studied model where algorithms access smooth, potentially non-convex functions through queries to an unbiased stochastic gradient oracle with bounded variance, we prove that (in the worst case) any algorithm requires at least ϵ- 4 queries to find an ϵ-stationary point. The lower bound is tight, and establishes that stochastic gradient descent is minimax optimal in this model. In a more restrictive model where the noisy gradient estimates satisfy a mean-squared smoothness property, we prove a lower bound of ϵ- 3 queries, establishing the optimality of recently proposed variance reduction techniques.

UR - http://www.scopus.com/inward/record.url?scp=85131593123&partnerID=8YFLogxK

U2 - 10.1007/s10107-022-01822-7

DO - 10.1007/s10107-022-01822-7

M3 - ???researchoutput.researchoutputtypes.contributiontojournal.article???

AN - SCOPUS:85131593123

SN - 0025-5610

VL - 199

SP - 165

EP - 214

JO - Mathematical Programming

JF - Mathematical Programming

IS - 1-2

ER -