Second-Order Information in Non-Convex Stochastic Optimization: Power and Limitations

Yossi Arjevani; Yair Carmon; John C. Duchi; Dylan J. Foster; Ayush Sekhari; Karthik Sridharan

Second-Order Information in Non-Convex Stochastic Optimization: Power and Limitations

Yossi Arjevani, Yair Carmon, John C. Duchi, Dylan J. Foster, Ayush Sekhari, Karthik Sridharan

School of Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

We design an algorithm which finds an ϵ-approximate stationary point (with ∥∇F(x)∥≤ϵ) using O(ϵ−3) stochastic gradient and Hessian-vector products, matching guarantees that were previously available only under a stronger assumption of access to multiple queries with the same random seed. We prove a lower bound which establishes that this rate is optimal and—surprisingly—that it cannot be improved using stochastic pth order methods for any p≥2, even when the first p derivatives of the objective are Lipschitz. Together, these results characterize the complexity of non-convex stochastic optimization with second-order methods and beyond. Expanding our scope to the oracle complexity of finding (ϵ,γ)-approximate second-order stationary points, we establish nearly matching upper and lower bounds for stochastic second-order methods. Our lower bounds here are novel even in the noiseless case.

Original language	English
Title of host publication	Proceedings of Thirty Third Conference on Learning Theory
Editors	Jacob Abernethy, Shivani Agarwal
Publisher	PMLR
Pages	242-299
Number of pages	58
Volume	125
State	Published - 1 Sep 2020
Event	33rd Annual Conference on Learning Theory, COLT 2020 - virtual Duration: 9 Jul 2020 → 12 Jul 2020 Conference number: 33

Publication series

Name	Proceedings of Machine Learning Research
Publisher	PMLR
ISSN (Electronic)	2640-3498

Conference

Conference	33rd Annual Conference on Learning Theory, COLT 2020
Abbreviated title	COLT 2020
Period	9/07/20 → 12/07/20

Access to Document

Cite this

Arjevani, Y., Carmon, Y., Duchi, J. C., Foster, D. J., Sekhari, A., & Sridharan, K. (2020). Second-Order Information in Non-Convex Stochastic Optimization: Power and Limitations. In J. Abernethy, & S. Agarwal (Eds.), Proceedings of Thirty Third Conference on Learning Theory (Vol. 125, pp. 242-299). (Proceedings of Machine Learning Research). PMLR. http://proceedings.mlr.press/v125/arjevani20a/arjevani20a.pdf

@inproceedings{75505fd5929c49c0b8d35d042289d9e8,

title = "Second-Order Information in Non-Convex Stochastic Optimization: Power and Limitations",

abstract = "We design an algorithm which finds an ϵ-approximate stationary point (with ∥∇F(x)∥≤ϵ) using O(ϵ−3) stochastic gradient and Hessian-vector products, matching guarantees that were previously available only under a stronger assumption of access to multiple queries with the same random seed. We prove a lower bound which establishes that this rate is optimal and—surprisingly—that it cannot be improved using stochastic pth order methods for any p≥2, even when the first p derivatives of the objective are Lipschitz. Together, these results characterize the complexity of non-convex stochastic optimization with second-order methods and beyond. Expanding our scope to the oracle complexity of finding (ϵ,γ)-approximate second-order stationary points, we establish nearly matching upper and lower bounds for stochastic second-order methods. Our lower bounds here are novel even in the noiseless case.",

author = "Yossi Arjevani and Yair Carmon and Duchi, {John C.} and Foster, {Dylan J.} and Ayush Sekhari and Karthik Sridharan",

year = "2020",

month = sep,

day = "1",

language = "אנגלית",

volume = "125",

series = "Proceedings of Machine Learning Research",

publisher = "PMLR",

pages = "242--299",

editor = "Jacob Abernethy and Shivani Agarwal",

booktitle = "Proceedings of Thirty Third Conference on Learning Theory",

note = "33rd Annual Conference on Learning Theory, COLT 2020 ; Conference date: 09-07-2020 Through 12-07-2020",

}

Arjevani, Y, Carmon, Y, Duchi, JC, Foster, DJ, Sekhari, A & Sridharan, K 2020, Second-Order Information in Non-Convex Stochastic Optimization: Power and Limitations. in J Abernethy & S Agarwal (eds), Proceedings of Thirty Third Conference on Learning Theory. vol. 125, Proceedings of Machine Learning Research, PMLR, pp. 242-299, 33rd Annual Conference on Learning Theory, COLT 2020, 9/07/20. <http://proceedings.mlr.press/v125/arjevani20a/arjevani20a.pdf>

Second-Order Information in Non-Convex Stochastic Optimization: Power and Limitations. / Arjevani, Yossi; Carmon, Yair; Duchi, John C. et al.
Proceedings of Thirty Third Conference on Learning Theory. ed. / Jacob Abernethy; Shivani Agarwal. Vol. 125 PMLR, 2020. p. 242-299 (Proceedings of Machine Learning Research).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Second-Order Information in Non-Convex Stochastic Optimization

T2 - 33rd Annual Conference on Learning Theory, COLT 2020

AU - Arjevani, Yossi

AU - Carmon, Yair

AU - Duchi, John C.

AU - Foster, Dylan J.

AU - Sekhari, Ayush

AU - Sridharan, Karthik

N1 - Conference code: 33

PY - 2020/9/1

Y1 - 2020/9/1

N2 - We design an algorithm which finds an ϵ-approximate stationary point (with ∥∇F(x)∥≤ϵ) using O(ϵ−3) stochastic gradient and Hessian-vector products, matching guarantees that were previously available only under a stronger assumption of access to multiple queries with the same random seed. We prove a lower bound which establishes that this rate is optimal and—surprisingly—that it cannot be improved using stochastic pth order methods for any p≥2, even when the first p derivatives of the objective are Lipschitz. Together, these results characterize the complexity of non-convex stochastic optimization with second-order methods and beyond. Expanding our scope to the oracle complexity of finding (ϵ,γ)-approximate second-order stationary points, we establish nearly matching upper and lower bounds for stochastic second-order methods. Our lower bounds here are novel even in the noiseless case.

AB - We design an algorithm which finds an ϵ-approximate stationary point (with ∥∇F(x)∥≤ϵ) using O(ϵ−3) stochastic gradient and Hessian-vector products, matching guarantees that were previously available only under a stronger assumption of access to multiple queries with the same random seed. We prove a lower bound which establishes that this rate is optimal and—surprisingly—that it cannot be improved using stochastic pth order methods for any p≥2, even when the first p derivatives of the objective are Lipschitz. Together, these results characterize the complexity of non-convex stochastic optimization with second-order methods and beyond. Expanding our scope to the oracle complexity of finding (ϵ,γ)-approximate second-order stationary points, we establish nearly matching upper and lower bounds for stochastic second-order methods. Our lower bounds here are novel even in the noiseless case.

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

VL - 125

T3 - Proceedings of Machine Learning Research

SP - 242

EP - 299

BT - Proceedings of Thirty Third Conference on Learning Theory

A2 - Abernethy, Jacob

A2 - Agarwal, Shivani

PB - PMLR

Y2 - 9 July 2020 through 12 July 2020

ER -

Second-Order Information in Non-Convex Stochastic Optimization: Power and Limitations

Abstract

Publication series

Conference

Access to Document

Fingerprint

Cite this