TY - GEN

T1 - Learning rates for Q-learning

AU - Even-Dar, Eyal

AU - Mansour, Yishay

N1 - Publisher Copyright:
© Springer-Verlag Berlin Heidelberg 2001.

PY - 2001

Y1 - 2001

N2 - In this paper we derive convergence rates for Q-learning. We show an interesting relationship between the convergence rate and the learning rate used in the Q-learning. For a polynomial learning rate, one which is 1/tω at time t where ω ∈ (1/2, 1), we show that that the convergence rate is polynomial in 1/(1 − γ), where γ is the discount factor. In contrast we show that for a linear learning rate, one which is 1/t at time t, the convergence rate has an exponential dependence on 1/(1 − γ). In addition we show a simple example that proves that this exponential behavior is inherent for a linear learning rate.

AB - In this paper we derive convergence rates for Q-learning. We show an interesting relationship between the convergence rate and the learning rate used in the Q-learning. For a polynomial learning rate, one which is 1/tω at time t where ω ∈ (1/2, 1), we show that that the convergence rate is polynomial in 1/(1 − γ), where γ is the discount factor. In contrast we show that for a linear learning rate, one which is 1/t at time t, the convergence rate has an exponential dependence on 1/(1 − γ). In addition we show a simple example that proves that this exponential behavior is inherent for a linear learning rate.

UR - http://www.scopus.com/inward/record.url?scp=84943257812&partnerID=8YFLogxK

U2 - 10.1007/3-540-44581-1_39

DO - 10.1007/3-540-44581-1_39

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:84943257812

SN - 9783540423430

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 589

EP - 604

BT - Computational Learning Theory - 14th Annual Conference on Computational Learning Theory, COLT 2001 and 5th European Conference on Computational Learning Theory, EuroCOLT 2001, Proceedings

A2 - Helmbold, David

A2 - Williamson, Bob

PB - Springer Verlag

T2 - 14th Annual Conference on Computational Learning Theory, COLT 2001 and 5th European Conference on Computational Learning Theory, EuroCOLT 2001

Y2 - 16 July 2001 through 19 July 2001

ER -