Efficient Rate Optimal Regret for Adversarial Contextual MDPs Using Online Function Approximation

Orin Levy; Alon Cohen; Asaf Cassel; Yishay Mansour

Efficient Rate Optimal Regret for Adversarial Contextual MDPs Using Online Function Approximation

Orin Levy^*, Alon Cohen^*, Asaf Cassel^*, Yishay Mansour^*

^*Corresponding author for this work

Alphabet Inc.

Research output: Contribution to journal › Conference article › peer-review

Abstract

We present the OMG-CMDP! algorithm for regret minimization in adversarial Contextual MDPs. The algorithm operates under the minimal assumptions of realizable function class and access to online least squares and log loss regression oracles. Our algorithm is efficient (assuming efficient online regression oracles), simple and robust to approximation errors. It enjoys an O^e(H^2.5pT|S||A|(R_TH(O) + H log(δ⁻¹))) regret guarantee, with T being the number of episodes, S the state space, A the action space, H the horizon and R_TH(O) = R_TH(O_sq^F ) + R_TH(O_log^P ) is the sum of the square and log-loss regression oracles' regret, used to approximate the context-dependent rewards and dynamics, respectively. To the best of our knowledge, our algorithm is the first efficient rate optimal regret minimization algorithm for adversarial CMDPs that operates under the minimal standard assumption of online function approximation.

Original language	English
Pages (from-to)	19287-19314
Number of pages	28
Journal	Proceedings of Machine Learning Research
Volume	202
State	Published - 2023
Event	40th International Conference on Machine Learning, ICML 2023 - Honolulu, United States Duration: 23 Jul 2023 → 29 Jul 2023

Funding

Funders	Funder number
Yandex Initiative for Machine Learning
Horizon 2020 Framework Programme
Blavatnik Family Foundation
European Research Council
Israel Science Foundation	2250/22
Israel Science Foundation
Tel Aviv University
Horizon 2020	2549/19, 882396, 993/17
Horizon 2020

Cite this

@article{010070e071df42aa8cf2e2f79de06e66,

title = "Efficient Rate Optimal Regret for Adversarial Contextual MDPs Using Online Function Approximation",

abstract = "We present the OMG-CMDP! algorithm for regret minimization in adversarial Contextual MDPs. The algorithm operates under the minimal assumptions of realizable function class and access to online least squares and log loss regression oracles. Our algorithm is efficient (assuming efficient online regression oracles), simple and robust to approximation errors. It enjoys an Oe(H2.5pT|S||A|(RTH(O) + H log(δ−1))) regret guarantee, with T being the number of episodes, S the state space, A the action space, H the horizon and RTH(O) = RTH(OsqF ) + RTH(OlogP ) is the sum of the square and log-loss regression oracles' regret, used to approximate the context-dependent rewards and dynamics, respectively. To the best of our knowledge, our algorithm is the first efficient rate optimal regret minimization algorithm for adversarial CMDPs that operates under the minimal standard assumption of online function approximation.",

author = "Orin Levy and Alon Cohen and Asaf Cassel and Yishay Mansour",

note = "Publisher Copyright: {\textcopyright} 2023 Proceedings of Machine Learning Research. All rights reserved.; 40th International Conference on Machine Learning, ICML 2023 ; Conference date: 23-07-2023 Through 29-07-2023",

year = "2023",

language = "אנגלית",

volume = "202",

pages = "19287--19314",

journal = "Proceedings of Machine Learning Research",

issn = "2640-3498",

}

TY - JOUR

T1 - Efficient Rate Optimal Regret for Adversarial Contextual MDPs Using Online Function Approximation

AU - Levy, Orin

AU - Cohen, Alon

AU - Cassel, Asaf

AU - Mansour, Yishay

PY - 2023

Y1 - 2023

N2 - We present the OMG-CMDP! algorithm for regret minimization in adversarial Contextual MDPs. The algorithm operates under the minimal assumptions of realizable function class and access to online least squares and log loss regression oracles. Our algorithm is efficient (assuming efficient online regression oracles), simple and robust to approximation errors. It enjoys an Oe(H2.5pT|S||A|(RTH(O) + H log(δ−1))) regret guarantee, with T being the number of episodes, S the state space, A the action space, H the horizon and RTH(O) = RTH(OsqF ) + RTH(OlogP ) is the sum of the square and log-loss regression oracles' regret, used to approximate the context-dependent rewards and dynamics, respectively. To the best of our knowledge, our algorithm is the first efficient rate optimal regret minimization algorithm for adversarial CMDPs that operates under the minimal standard assumption of online function approximation.

AB - We present the OMG-CMDP! algorithm for regret minimization in adversarial Contextual MDPs. The algorithm operates under the minimal assumptions of realizable function class and access to online least squares and log loss regression oracles. Our algorithm is efficient (assuming efficient online regression oracles), simple and robust to approximation errors. It enjoys an Oe(H2.5pT|S||A|(RTH(O) + H log(δ−1))) regret guarantee, with T being the number of episodes, S the state space, A the action space, H the horizon and RTH(O) = RTH(OsqF ) + RTH(OlogP ) is the sum of the square and log-loss regression oracles' regret, used to approximate the context-dependent rewards and dynamics, respectively. To the best of our knowledge, our algorithm is the first efficient rate optimal regret minimization algorithm for adversarial CMDPs that operates under the minimal standard assumption of online function approximation.

UR - http://www.scopus.com/inward/record.url?scp=85174407759&partnerID=8YFLogxK

M3 - ???researchoutput.researchoutputtypes.contributiontojournal.conferencearticle???

AN - SCOPUS:85174407759

SN - 2640-3498

VL - 202

SP - 19287

EP - 19314

JO - Proceedings of Machine Learning Research

JF - Proceedings of Machine Learning Research

T2 - 40th International Conference on Machine Learning, ICML 2023

Y2 - 23 July 2023 through 29 July 2023

ER -

Efficient Rate Optimal Regret for Adversarial Contextual MDPs Using Online Function Approximation

Abstract

Funding

Other files and links

Fingerprint

Cite this