CAREER: Taming the size, complexity and longevity of OS kernels via enhanced OS kernel extensions

Operating system (OS) kernels are the most fundamental software component society depends on, underpinning the computing industry, from the clouds and datacenters that manage our data to the smart phones and other devices we all use daily. Unfortunately, they have grown to be complex and have not kept pace with advances in

programming languages and modular software design, instead struggling to meet modern security and reliability threats to our computing infrastructure. Moreover, there appears to be no clear path to incrementally evolve them and their unchecked complexity has become a significant barrier to entry for students and practitioners to learn

or innovate at the kernel level. This project seeks to remedy this issue by create a safe kernel extension framework within the legacy OS kernel that enables it to be replaced in part or in its entirety with safe components. Through this extension framework, the project will enable a practical, incrementally-adoptable mechanism for

undergraduate, graduate students and practitioners to evolve kernels to be safer and more reliable, improving national security and leading to higher productivity for society and the national economy. This project aims to achieve three goals for the OS kernel: security and robustness through language safety, robustness and incremental

update through componentization, and practical relevance through building upon existing kernel extension frameworks. The key insight in our approach is that a safe kernel extension framework can be created within the legacy OS kernel that enables any part of it to be replaced via bypass extensions, evolving the kernel towards a safe,

componentized architecture, or in its entirety with a safe and specialized in-situ kernel. Our integrated research and education plan involves three broad research thrusts. First, we will explore expressiveness/safety tradeoffs for kernel extensions, and will design a new Rust-based kernel extension environment that provides similar

safety guarantees with a more expressive programming environment than the popular eBPF kernel extension framework in Linux. Second, we will explore to what extent our enhanced safe kernel extensions can replace individual kernel components with bypass extensions and explore new component boundaries in Linux. Finally, we will explore to what extent

an entirely new in-situ kernel can be created from extensions, ultimately in the context of Linux. Our educational plan integrates the research in courses and other initiatives involving open source and industry.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.