CCRI: ENS: Evolving the Defects4J Benchmark and Infrastructure to Enable Sustained Innovation and Reproducibility in Software Engineering Research

Software engineering research aims to increase the quality of the software that pervades modern technology. To make significant advances, reproducibility and comparability of empirical results, using realistic artifacts, are essential in many software engineering research areas, including software testing and debugging, program comprehension, software evolution, and machine learning in software engineering.

The Defects4J database of real software faults and a supporting infrastructure were developed to support the software engineering research community, in particular to further reproducibility, to enable faster and better completion of downstream research, and to free researchers from the burden of (re-)developing an experiment infrastructure.

This project evolves and extends Defects4J, which has been widely adopted by the software engineering research community. It focuses on enabling sustained innovation and reproducibility in software engineering research by making Defects4J more broadly available and meeting the growing interest and feature requests from the software engineering research community.

Specifically, this project aims to (1) grow the current set of artifacts in Defects4J, based on community feedback, (2) develop a fully automated framework for mining additional artifacts, (3) future-proof and containerize all artifacts for use with modern compilers and runtime environments, and (4) provide comprehensive tutorials and templates to support researchers and educators. A high degree of automation and structured pathways to community contributions will support sustained growth.

Additionally, this project advances scientific knowledge about defect patterns, prediction, and prevention, applicability and limitations of research techniques, and best practices for empirical evaluations. Finally, this project supports software engineering education at the graduate and undergraduate level. The benchmark and infrastructure allow students to reproduce published results, experiment with, and compare, existing tools and techniques, and quickly evaluate new ideas.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.