Entropy-regularized deep reinforcement learning from a linear programming perspective.

Vila Alarcón, Nicolás

Entropy-regularized deep reinforcement learning from a linear programming perspective.

License

Citation

Vila Alarcón, N. Entropy-regularized deep reinforcement learning from a linear programming perspective.. 2024. handle: http://hdl.handle.net/10230/68216

This citation was generated automatically.

Abstract

In recent years, Reinforcement Learning (RL) has gained significant attention due to its applications in real-world problems, particularly leveraging Deep Neural Networks (DNNs). While state of the art models like Proximal Policy Optimization (PPO), Twin Delayed DDPG (TD3), or Soft Actor-Critic (SAC) employ the squared Bellman error (SBE) in some way or other to address the Bellman Optimality Equations, the limitations of SBE are well-documented. To address these issues, alternative algorithms rooted in the Linear Program (LP) reformulation of Markov Decision Processes (MDPs), such as REPS and QREPS, have shown to be promising. Yet, there exists a notable gap in research regarding the adaptation and application of these LP-based approaches within a large scale DNN learning setting. We propose an adaptation of LP-based algorithms with the use of DNNs and perform an empirical study to understand the impact of different design choices in their practical implementation that may directly or indirectly affect their performance. We train over 30,000 agents in several discrete classical control environments of different complexity and provide insights and practical recommendations. Furthermore, we evaluate their performance compared to other well-known RL algorithms.