Collaborative Research: Gradient-free optimization of matrix functions

Artificial intelligence models typically reduce to optimization problems: find the best solution according to a problem-specific metric. Sometimes, the nature of the problems means that the standard calculus-based tools cannot be applied. This setting is known as gradient-free optimization, and is particularly relevant for small businesses, academic research groups, and public-sector organizations that lack large-scale computing infrastructure yet still need to fine-tune machine learning models or optimize complex simulations. This project develops new mathematical and computational tools that make gradient-free optimization dramatically more efficient by exploiting hidden low-dimensional structure in these problems. This will lower the computational barrier to entry for a broad range of users. The project will train PhD students in these interdisciplinary methods, produce openly available software, and develop instructional materials connecting linear algebra to modern deep learning. Gradient-free optimization (GFO) has deep theoretical foundations, yet remains poorly understood in high dimensions. This project will establish mathematical and algorithmic tools that break worst-case GFO barriers by exploiting structure in matrix-valued gradients. Algorithms for objective functions whose gradients exhibit various kinds of low intrinsic dimensionality, such as sparsity, low rank, or sparsity-plus-low-rank will be developed. These gradient estimation techniques will be wrapped into modern algorithms such as muon which involve matrix factorizations, projections, or solving linear systems in the Fisher information matrix. Drawing upon computational linear algebra, novel techniques for performing these matrix operations in the gradient-free setting will be provided. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria. NSF Award ID: 2608660 | Program: 01002627DB NSF RESEARCH & RELATED ACTIVIT | Principal Investigator: Stephen Becker | Institution: University of Colorado at Boulder, BOULDER, CO | Award Amount: $150,000 View on NSF Award Search: https://www.nsf.gov/awardsearch/show-award/?AWD_ID=2608660 View on Research.gov: https://www.research.gov/awardapi-service/v1/awards/2608660.html