CAREER: Foundations of Robust and Sample-Efficient Data-Driven Decision Making and Control

As engineering systems become more complex, it becomes progressively harder to obtain accurate physical models that describe their behavior. This had led to a shift from classical model-based control to data-driven and learning-based techniques, including reinforcement learning (RL). Despite the widespread use of RL algorithms across several domains, their design hinges crucially on the idealistic assumption of perfect feedback data for learning decision-making policies. However, in practice, data streams can be noisy with significant uncertainty ("heavy tailed probability distributions"), can contain out-of-distribution samples, and can be corrupted by sophisticated adversaries. Failing to account for such non-ideal feedback can lead to dire consequences in safety-critical autonomous systems. As it stands, there is limited theoretical understanding of the interplay between model uncertainty and adversarial data in the context of data-driven control. This NSF CAREER project aims to address this critical research gap by developing a foundational framework of robust RL that bridges the gap between theory and practice, allowing for the reliable deployment of RL algorithms in applications with little tolerance for error, such as self-driving cars, robotics, and healthcare analytics. To achieve this, the intellectual merit of this project will involve building formal connections between the rich area of algorithmic robust statistics and ideas from RL, optimization, and control. The broader impacts of the project include a tight integration of the technical results with new data science courses, engagement with local high school teachers via the Science House outreach unit at the PI's institution, and industry outreach. This project comprises three interlinked thrusts that collectively address the technical challenge of providing non-asymptotic performance guarantees for decision-making algorithms subject to time-correlated, streaming data corrupted by heavy-tailed noise processes and outliers. Focusing on Markov Decision Processes (MDPs) with finite state-action spaces, the first thrust will identify information-theoretic fundamental limits on performance imposed by contaminated reward feedback. By leveraging tools from robust statistics, the next step will focus on design and analysis of novel RL algorithms with function approximation that achieve such limits. The second thrust will investigate the benefits of collaboration in multi-agent and federated RL under spatial corruption, subject to communication constraints. Finally, the third thrust will develop robust model-based and model-free data-driven control algorithms for unknown dynamical systems with continuous state-action spaces, and delineate the effects of corrupted data on system stability. The developed technical tools will advance the field of robust statistics by enabling robust mean and covariance estimation under correlated data, and that of RL, by building a finite-time theory for stochastic approximation with inexact updates. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria. NSF Award ID: 2542396 | Program: 01002627DB NSF RESEARCH & RELATED ACTIVIT | Principal Investigator: Aritra Mitra | Institution: North Carolina State University, RALEIGH, NC | Award Amount: $509,248 View on NSF Award Search: https://www.nsf.gov/awardsearch/show-award/?AWD_ID=2542396 View on Research.gov: https://www.research.gov/awardapi-service/v1/awards/2542396.html