CC* Data Storage: Software Defined Storage for Composable and HPC Workflows

High Performance Computing (HPC) resources on university campuses have traditionally provided multiple tiers of storage (scratch, project and archive) to accommodate the various I/O patterns, data sizes and retention requirements of scientific computing workflows. Traditionally, these have been provided by POSIX based storage systems and robotic tape libraries.

Recent campus level adoption of cloud native technologies like Kubernetes have produced scientific computing workflows that require additional data access methods not possible via traditional methods. Campus storage offerings must evolve to support non-POSIX data access methods and provisioning of purpose-built storage devices, providing additional storage types alongside traditional POSIX based ones, for a holistic offering.

This project closes the gap between campus and cloud storage through the deployment of a central, shared, multi-petabyte Ceph distributed storage system that: 1) Enhances campus shared storage capabilities by establishing a model for Software Defined Storage as a Service (STaaS) at the campus level, offering unified access to and on-demand provisioning of block, object and filesystem storage across composable and HPC resources. 2) Supports Science Domains via innovative storage infrastructure, providing new data access methods between instrumentation, campus composable and HPC resources and the public cloud. 3) Enables education and workforce development by engaging with undergraduate students in deployment, operation and integration of storage infrastructure and supports cloud native applications used by researchers and experiential and residential learning programs on the Purdue University campus.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.