Regret Minimization and Convergence to Equilibria in General-sum Markov Games

Liad Erez*, Tal Lancewicki*, Uri Sherman*, Tomer Koren, Yishay Mansour

*Corresponding author for this work

Research output: Contribution to journalConference articlepeer-review

Abstract

An abundance of recent impossibility results establish that regret minimization in Markov games with adversarial opponents is both statistically and computationally intractable. Nevertheless, none of these results preclude the possibility of regret minimization under the assumption that all parties adopt the same learning procedure. In this work, we present the first (to our knowledge) algorithm for learning in general-sum Markov games that provides sublinear regret guarantees when executed by all agents. The bounds we obtain are for swap regret, and thus, along the way, imply convergence to a correlated equilibrium. Our algorithm is decentralized, computationally efficient, and does not require any communication between agents. Our key observation is that online learning via policy optimization in Markov games essentially reduces to a form of weighted regret minimization, with unknown weights determined by the path length of the agents' policy sequence. Consequently, controlling the path length leads to weighted regret objectives for which sufficiently adaptive algorithms provide sublinear regret guarantees.

Original languageEnglish
Pages (from-to)9343-9373
Number of pages31
JournalProceedings of Machine Learning Research
Volume202
StatePublished - 2023
Event40th International Conference on Machine Learning, ICML 2023 - Honolulu, United States
Duration: 23 Jul 202329 Jul 2023

Funding

FundersFunder number
Adelis Research Fund for Artificial Intelligence
Yandex Initiative for Machine Learning
Horizon 2020 Framework Programme
Blavatnik Family Foundation
European Research Council
Israel Science Foundation993/17
Tel Aviv University
Horizon 2020882396

    Fingerprint

    Dive into the research topics of 'Regret Minimization and Convergence to Equilibria in General-sum Markov Games'. Together they form a unique fingerprint.

    Cite this