ReSQueing Parallel and Private Stochastic Convex Optimization

Yair Carmon, Arun Jambulapati, Yujia Jin, Yin Tat Lee, Daogao Liu, Aaron Sidford, Kevin Tian

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

We introduce a new tool for stochastic convex optimization (SCO): a Reweighted Stochastic Query (ReSQue) estimator for the gradient of a function convolved with a (Gaussian) probability density. Combining ReSQue with recent advances in ball oracle acceleration [CJJ+20], [ACJ+21], we develop algorithms achieving state-of-the-art complexities for SCO in parallel and private settings. For a SCO objective constrained to the unit ball in Rd, we obtain the following results (up to polylogarithmic factors).1)We give a parallel algorithm obtaining optimization error ϵ_ opt with d1/3 ϵ_ opt -2/3 gradient oracle query depth and d1/3 ϵ_ opt -2/3+ϵ_ opt -2 gradient queries in total, assuming access to a bounded-variance stochastic gradient estimator. For ϵ_ opt ∈[d-1, d-1/4], our algorithm matches the state-of-the-art oracle depth of [BJL+19] while maintaining the optimal total work of stochastic gradient descent.2)Given n samples of Lipschitz loss functions, prior works [BFTT19], [BFGT20], [AFKT21], [KLL21] established that if n > rsim d ϵ_dp-2,(ϵ_dp, Δ)-differential privacy is attained at no asymptotic cost to the SCO utility. However, these prior works all required a superlinear number of gradient queries. We close this gap for sufficiently large n > rsim d2 ϵ_d p-3, by using ReSQue to design an algorithm with near-linear gradient query complexity in this regime.

Original languageEnglish
Title of host publicationProceedings - 2023 IEEE 64th Annual Symposium on Foundations of Computer Science, FOCS 2023
PublisherIEEE Computer Society
Pages2031-2058
Number of pages28
ISBN (Electronic)9798350318944
DOIs
StatePublished - 2023
Event64th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2023 - Santa Cruz, United States
Duration: 6 Nov 20239 Nov 2023

Publication series

NameProceedings - Annual IEEE Symposium on Foundations of Computer Science, FOCS
ISSN (Print)0272-5428

Conference

Conference64th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2023
Country/TerritoryUnited States
CitySanta Cruz
Period6/11/239/11/23

Funding

FundersFunder number
National Science FoundationCCF-1955039
Microsoft Research
Blavatnik Family Foundation
Israel Science Foundation2486/21

    Keywords

    • differential privacy
    • parallel computation
    • stochastic optimization

    Fingerprint

    Dive into the research topics of 'ReSQueing Parallel and Private Stochastic Convex Optimization'. Together they form a unique fingerprint.

    Cite this