TY - GEN
T1 - Sharper bounds for regularized data fitting
AU - Avron, Haim
AU - Clarkson, Kenneth L.
AU - Woodruff, David P.
N1 - Publisher Copyright:
© Haim Avron, Kenneth L. Clarkson, and David P. Woodruff.
PY - 2017/8/1
Y1 - 2017/8/1
N2 - We study matrix sketching methods for regularized variants of linear regression, low rank approximation, and canonical correlation analysis. Our main focus is on sketching techniques which preserve the objective function value for regularized problems, which is an area that has remained largely unexplored. We study regularization both in a fairly broad setting, and in the specific context of the popular and widely used technique of ridge regularization; for the latter, as applied to each of these problems, we show algorithmic resource bounds in which the statistical dimension appears in places where in previous bounds the rank would appear. The statistical dimension is always smaller than the rank, and decreases as the amount of regularization increases. In particular, for the ridge low-rank approximation problem minY,XkY X -Ak2 F + kY k2 F + kXk2 F , where Y 2 Rn×k and X 2 Rk×d, we give an approximation algorithm needing O(nnz(A)) + O((n + d)-1k min{k, -1 sd(Y ϵ)}) + poly(sd(Y ϵ)-1) time, where s(Y ϵ)k is the statistical dimension of Y ϵ, Y ϵ is an optimal Y ,is an error parameter, and nnz(A) is the number of nonzero entries of A. This is faster than prior work, even when= 0. We also study regularization in a much more general setting. For example, we obtain sketching-based algorithms for the low-rank approximation problem minX,Y kY X -Ak2 F +f(Y,X) where f is a regularizing function satisfying some very general conditions (chiefly, invariance under orthogonal transformations).
AB - We study matrix sketching methods for regularized variants of linear regression, low rank approximation, and canonical correlation analysis. Our main focus is on sketching techniques which preserve the objective function value for regularized problems, which is an area that has remained largely unexplored. We study regularization both in a fairly broad setting, and in the specific context of the popular and widely used technique of ridge regularization; for the latter, as applied to each of these problems, we show algorithmic resource bounds in which the statistical dimension appears in places where in previous bounds the rank would appear. The statistical dimension is always smaller than the rank, and decreases as the amount of regularization increases. In particular, for the ridge low-rank approximation problem minY,XkY X -Ak2 F + kY k2 F + kXk2 F , where Y 2 Rn×k and X 2 Rk×d, we give an approximation algorithm needing O(nnz(A)) + O((n + d)-1k min{k, -1 sd(Y ϵ)}) + poly(sd(Y ϵ)-1) time, where s(Y ϵ)k is the statistical dimension of Y ϵ, Y ϵ is an optimal Y ,is an error parameter, and nnz(A) is the number of nonzero entries of A. This is faster than prior work, even when= 0. We also study regularization in a much more general setting. For example, we obtain sketching-based algorithms for the low-rank approximation problem minX,Y kY X -Ak2 F +f(Y,X) where f is a regularizing function satisfying some very general conditions (chiefly, invariance under orthogonal transformations).
KW - Canonical Correlation Analysis
KW - Low-rank approximation
KW - Matrices
KW - Regression
KW - Regularization
UR - http://www.scopus.com/inward/record.url?scp=85028721872&partnerID=8YFLogxK
U2 - 10.4230/LIPIcs.APPROX/RANDOM.2017.27
DO - 10.4230/LIPIcs.APPROX/RANDOM.2017.27
M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???
AN - SCOPUS:85028721872
T3 - Leibniz International Proceedings in Informatics, LIPIcs
BT - Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques - 20th International Workshop, APPROX 2017 and 21st International Workshop, RANDOM 2017
A2 - Rolim, Jose D. P.
A2 - Jansen, Klaus
A2 - Williamson, David P.
A2 - Vempala, Santosh S.
PB - Schloss Dagstuhl- Leibniz-Zentrum fur Informatik GmbH, Dagstuhl Publishing
T2 - 20th International Workshop on Approximation Algorithms for Combinatorial Optimization Problems, APPROX 2017 and the 21st International Workshop on Randomization and Computation, RANDOM 2017
Y2 - 16 August 2017 through 18 August 2017
ER -