TY - GEN

T1 - Private Stochastic Convex Optimization

T2 - 38th International Conference on Machine Learning, ICML 2021

AU - Asi, Hilal

AU - Feldman, Vitaly

AU - Koren, Tomer

AU - Talwar, Kunal

N1 - Publisher Copyright:
Copyright © 2021 by the author(s)

PY - 2021

Y1 - 2021

N2 - Stochastic convex optimization over an ℓ1-bounded domain is ubiquitous in machine learning applications such as LASSO but remains poorly understood when learning with differential privacy. We show that, up to logarithmic factors the optimal excess population loss of any (ε, δ)-differentially private optimizer is plog(d)/n + √d/εn. The upper bound is based on a new algorithm that combines the iterative localization approach of Feldman et al. (2020a) with a new analysis of private regularized mirror descent. It applies to ℓp bounded domains for p ∈ [1, 2] and queries at most n3/2 gradients improving over the best previously known algorithm for the ℓ2 case which needs n2 gradients. Further, we show that when the loss functions satisfy additional smoothness assumptions, the excess loss is upper bounded (up to logarithmic factors) by plog(d)/n + (log(d)/εn)2/3. This bound is achieved by a new variance-reduced version of the Frank-Wolfe algorithm that requires just a single pass over the data. We also show that the lower bound in this case is the minimum of the two rates mentioned above.

AB - Stochastic convex optimization over an ℓ1-bounded domain is ubiquitous in machine learning applications such as LASSO but remains poorly understood when learning with differential privacy. We show that, up to logarithmic factors the optimal excess population loss of any (ε, δ)-differentially private optimizer is plog(d)/n + √d/εn. The upper bound is based on a new algorithm that combines the iterative localization approach of Feldman et al. (2020a) with a new analysis of private regularized mirror descent. It applies to ℓp bounded domains for p ∈ [1, 2] and queries at most n3/2 gradients improving over the best previously known algorithm for the ℓ2 case which needs n2 gradients. Further, we show that when the loss functions satisfy additional smoothness assumptions, the excess loss is upper bounded (up to logarithmic factors) by plog(d)/n + (log(d)/εn)2/3. This bound is achieved by a new variance-reduced version of the Frank-Wolfe algorithm that requires just a single pass over the data. We also show that the lower bound in this case is the minimum of the two rates mentioned above.

UR - http://www.scopus.com/inward/record.url?scp=85161341468&partnerID=8YFLogxK

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85161341468

T3 - Proceedings of Machine Learning Research

SP - 393

EP - 403

BT - Proceedings of the 38th International Conference on Machine Learning, ICML 2021

PB - ML Research Press

Y2 - 18 July 2021 through 24 July 2021

ER -