POSE: Phase II: Expanding the data.table ecosystem for efficient big data manipulation in R

Professor Toby Hocking from Northern Arizona University is supported by an award from the Pathways to Enable Open-Source Ecosystems (POSE) program in the Directorate for Technology, Innovation and Partnerships (TIP). More data is being systematically gathered and recorded than at any previous time in human history, and efficient software packages are required to store and analyze these data using limited computational resources.

A leading, state-of-the-art example is data.table, which is free/open-source software for in-memory data manipulation/analysis, implemented as an R package with C code that is highly efficient in terms of both computation time and memory usage. Although data.table has a substantial number of users (including thousands of other R packages which import functionality from data.table), the growth of data.table is limited by (1) its flat/informal leadership structure with only one author at the top who can approve new code contributions, (2) lack of documentation/translations and community standards for promoting diversity/inclusion, and (3) lack of infrastructure for systematic software testing.

This project will expand the open-source ecosystem of users, contributors, and developers of data.table, by addressing these issues. In particular, this project will create (1) a written governance document with a new hierarchical leadership structure, (2) new documentation materials for onboarding new users/contributors, including translations and community standards to encourage diversity/inclusion, and (3) new testing software and infrastructure.

Furthermore, the project includes plans to systematically evaluate the broader impacts of this effort on the data.table ecosystem by measuring changes to important metrics (number of unique contributors, diversity of contributors, number of dependent packages, etc). The result of this project will be a self-sustaining open-source ecosystem for data.table, which will allow it to grow into a more powerful data analysis tool that will be used by more people, and more diverse people, in the future.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.