Reinforcement Leaning for Infinite-Dimensional Systems

Interest in reinforcement learning (RL) for massive-scale systems consisting of large populations of intelligent agents interacting with heterogeneous environments has witnessed a significant surge in recent years across diverse scientific domains. However, the large-scale nature of these systems often results in high computational costs or compromised performance for most state-of-the-art RL techniques. To address these challenges, we propose a novel RL architecture along with the derivation of effective algorithms to learn optimal policies for arbitrarily large systems of agents. In our formulation, we model such a system as a parameterized control system defined on an infinite-dimensional function space. We then develop a moment kernel transform to map the parameterized system and the value function into a reproducing kernel Hilbert space. This transformation generates a sequence of finite-dimensional moment representations for the RL problem, which are organized into a filtrated structure. Leveraging this RL filtration, we develop a hierarchical algorithm for learning optimal policies for the infinite-dimensional parameterized system. We further enhance the efficiency of the algorithm by exploiting early stopping at each hierarchy, which demonstrates the fast convergence property of the algorithm through the construction of a convergent spectral sequence. The performance and efficiency of the proposed algorithm are validated using practical examples.