Distributed reinforcement learning based on factored action-spaces

This paper addresses the decision-making mechanism, for controlling Markov Decision Processes (MDPs) with factored action-spaces. The paper proposes a simple decision-making approach, which decomposes a multidimensional action-space into its dimensions and distributes the dimensions amongst multiple autonomous agents. A distinct Q-function is defined for each agent, and a Reinforcement Learning (RL) algorithm, which uses the decision-making approach, is proposed. The algorithm maintains estimations that converge to the agents' Q-functions. It is shown that by distributing the decision-making, and due to the factored action-space, new opportunities for representation abstractions emerge. The paper's contribution is illustrated, using a navigation task that involves three robots.