TY - JOUR

T1 - R-max – A General Polynomial Time Algorithm for Near-Optimal Reinforcement Learning Ronen

AU - Brafman I., Ronen

AU - Tennenholtz, Moshe

PY - 2002

Y1 - 2002

N2 - R-max is a very simple model-based reinforcement learning algorithm which can attain near-optimal average reward in polynomial time. In R-max, the agent always maintains a complete, but possibly inaccurate model of its environment and acts based on the optimal policy derived from this model. The model is initialized in an optimistic fashion: all actions in all states return the maximal possible reward (hence the name). During execution, it is updated based on the agent’s observations. R-max improves upon several previous algo- rithms: (1) It is simpler and more general than Kearns and Singh’s E3 algorithm, covering zero-sum stochastic games. (2) It has a built-in mechanism for resolving the exploration vs. exploitation dilemma. (3) It formally justifies the “optimism under uncertainty” bias used in many RL algorithms. (4) It is simpler, more general, and more efficient than Brafman and Tennenholtz’s LSG algorithm for learning in single controller stochastic games. (5) It generalizes the algorithm by Monderer and Tennenholtz for learning in repeated games. (6) It is the only algorithm for learning in repeated games, to date, which is provably efficient, considerably improving and simplifying previous algorithms by Banos and by Megiddo.

AB - R-max is a very simple model-based reinforcement learning algorithm which can attain near-optimal average reward in polynomial time. In R-max, the agent always maintains a complete, but possibly inaccurate model of its environment and acts based on the optimal policy derived from this model. The model is initialized in an optimistic fashion: all actions in all states return the maximal possible reward (hence the name). During execution, it is updated based on the agent’s observations. R-max improves upon several previous algo- rithms: (1) It is simpler and more general than Kearns and Singh’s E3 algorithm, covering zero-sum stochastic games. (2) It has a built-in mechanism for resolving the exploration vs. exploitation dilemma. (3) It formally justifies the “optimism under uncertainty” bias used in many RL algorithms. (4) It is simpler, more general, and more efficient than Brafman and Tennenholtz’s LSG algorithm for learning in single controller stochastic games. (5) It generalizes the algorithm by Monderer and Tennenholtz for learning in repeated games. (6) It is the only algorithm for learning in repeated games, to date, which is provably efficient, considerably improving and simplifying previous algorithms by Banos and by Megiddo.

KW - Learning in Games

KW - Markov Decision Processes

KW - Provably Efficient Learning

KW - Reinforcement Learning

KW - Stochastic Games

UR - http://www.jmlr.org/papers/volume3/brafman02a/brafman02a.pdf

UR - https://www.mendeley.com/catalogue/93da27d5-a8b3-3d59-b91b-895fcea63848/

M3 - Article

SN - 1475-4916

VL - 60

SP - 192

EP - 193

JO - Homeopathy

JF - Homeopathy

IS - 3

ER -