Collaborative Research: Decentralized Multi-Agent Reinforcement Learning (MARL) Algorithms for Solving Constrained Markov Games

This project seeks to develop new methods that allow multiple autonomous decision-makers, such as robots, energy resources, or actuators, to learn how to operate safely and efficiently in dynamic and uncertain environments without relying on a central coordinator. In many emerging technologies, individual components must make decisions based only on limited local information, while collectively satisfying safety, resource, or operational constraints. Ensuring that such systems learn and adapt without causing unsafe behavior or resource violations is a fundamental engineering challenge. The project addresses the foundational question of how decentralized decision-makers can learn to coordinate in complex, competitive, and constrained environments while still providing rigorous performance and safety guarantees. Current approaches often assume full information sharing, or ignore constraints during learning, limiting their applicability to real-world systems. The intellectual merit of the project lies primarily in establishing new mathematical foundations for safe learning in multi-agent systems and developing provably convergent multi-agent reinforcement learning algorithms that explicitly incorporate constraints into learning dynamics. The broader impacts of the project include advancing national priorities in safe and trustworthy AI and autonomous systems, integrating research outcomes into undergraduate and graduate curricula, mentoring students, and organizing academic events focused on safe multi-agent learning and outreach activities that broaden participation in STEM fields. This project aims to enable multiple autonomous decision-makers to safely and efficiently operate in dynamic and competitive environments without relying on a central coordinator. Such settings arise in teams of robots navigating shared spaces and distributed energy resources in power grids. The project will develop convergent multi-agent reinforcement learning algorithms for constrained Markov games using best-response and Lagrangian-based formulations that incorporates constraints explicitly into learning dynamics. In contrast to existing approaches that require repeatedly solving centralized constrained Markov decision processes or nested Lagrangian games at every iteration, the project designs single-loop algorithms that directly update policies and dual variables, improving scalability and reducing coordination requirements while preserving convergence and constraint satisfaction guarantees. The research further introduces structural characterizations of multi-agent interactions and allows for adaptive, potentially non-stationary strategies when stationary equilibria may not exist, establishing a unified framework that connects equilibrium concepts with modern reinforcement learning. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria. NSF Award ID: 2533865 | Program: 01002627DB NSF RESEARCH & RELATED ACTIVIT | Principal Investigator: Santiago Paternain | Institution: Rensselaer Polytechnic Institute, TROY, NY | Award Amount: $299,935 View on NSF Award Search: https://www.nsf.gov/awardsearch/show-award/?AWD_ID=2533865 View on Research.gov: https://www.research.gov/awardapi-service/v1/awards/2533865.html