Beyond the Replication Crisis: Examining and Addressing the Group-to-Person Generalizability Problem in the Psychological and Social Sciences

Replicating scientific results is paramount for gaining confidence, validating scientific conclusions and estimating true effect sizes (Klein et al., 2014; McNutt, 2014).

With the publication of the psychology reproducibility project (Open Science Collaboration, 2015), the worrying suspicion that the majority of psychological research could not be replicated was worsened.

Specifically, the psychology reproducibility project attempted to directly replicate 100 studies from leading psychology journals. Overall, 36% of the replications demonstrated significant results (Open Science Collaboration, 2015).

Efforts to replicate 21 social science studies from the journals Nature and Science revealed an overall replication rate of 62% (Camerer et al., 2018). These replication projects have highlighted the need for improved research practices, methods, and norms.

Journals, editors, and reviewers now commonly require larger samples and encourage and/or recognize the importance of replication and open-science practices such as open-access data, pre-registrations and improved statistical approaches like the use of mixed-effects models in order to allow generalizations across stimuli (Korbmacher et al., 2023).

Encouragingly there are some initial signs that replication rates in psychological and behavioural sciences might be improving, in part, due to improved research practices.

Recently, Protzko et al. (2023) reported a 86% replication rate of newly discovered social-behavioural effects, employing the latest "rigour-enhancing" research and open science practices. While this seems encouraging, it should be noted that these replication efforts relate to group-level effects.

However, group-level effects cannot necessarily be generalised to the person-level (McManus, Young, & Sweetman, 2023).

That is, it may be possible to find replicable, group-level effects in the psychological and social sciences but these effects may not describe more than a (sometimes very) small minority of the sample/population (McManus et al.).

The proposed doctoral research aims to examine the extent of, and provide researchers with more tools for addressing, this group-to-person generalisability problem.