CAREER: Resilient and Efficient Automatic Control in Energy Infrastructure: An Expert-Guided Policy Optimization Framework

This Faculty Early Career Development (CAREER) award supports research that will leverage cutting-edge artificial intelligence technologies to significantly enhance the resilience and efficiency of automated control systems within a broad class of energy infrastructure systems. This initiative is crucial as it addresses several substantial limitations faced by existing learning-based decision-making frameworks used in practice.

This research will bridge these critical knowledge gaps by developing an analytically rigorous and practically implementable framework that integrates reinforcement learning with mathematical optimization, along with expert-in-the-loop guidance. The successful application of this research is anticipated to yield improvements in efficiency, stability, and security, empowering the energy infrastructure to respond rapidly and securely to uncertainty and disruptive events.

Integration of this research into the curriculum at University of Washington will foster training and learning opportunities in reinforcement learning for both graduate and undergraduate students. Educational and outreach activities are designed to increase awareness and interest among K-12 and college students through diverse initiatives, including an interactive artificial intelligence game training platform, video modules to supplement classroom lessons for local high schools, and research engagement with underrepresented students.

This project creatively applies the principles of distributionally robust optimization to policy gradient reinforcement learning methods for improving online policy sample efficiency and maintaining stability. The model’s superior numerical performance stems from its unrestricted policy distribution, rejection-free policy updates, as well as monotonic performance and global convergence guarantee through Wasserstein metric-based policy optimization.

The expert-in-the-loop reinforcement learning framework effectively leverages expert demonstrations and feedback to ensure safe system operation, accelerate learning, and enhance algorithm convergence. By modifying the advantage function in "susceptible" situations, the framework guides learning direction and addresses reinforcement learning’s weaknesses with limited samples.

This research will answer three key questions: How to effectively utilize expert feedback? How to identify states that require expert intervention? And how to achieve an optimal and stable policy independent of expert input?

The innovative mathematical models and algorithms generated by this work will contribute to addressing online decision-making challenges for better operations and management of complex energy infrastructure systems.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.