If you’ve followed science news at all over the past few years, you’ve probably heard at least a little bit about the ongoing “replication crisis” where findings from several scientific disciplines have proven difficult or impossible to replicate. Some of these replication failures are inherent to the nature of science (e.g., issues concerning unique populations or specific procedures). Some of the replication failures were the result of unethical decisions by researchers, including practices like “p-hacking,” meaning statistically manipulating non-significant data to produce apparently significant results. However, over the last several years of investigation, much of the replication crisis can be ascribed to issues of measurement – small sample sizes, biased or unreliable measures, and a lack of attention to the size of an intervention’s impact.
The replication crisis is still very much ongoing. In 2023 alone, more than 10,000 papers have been retracted, including work from some very important figures in both the medical and social sciences. So, why am I talking about it in a newsletter about leadership development? The answer is that the issues currently affecting leader development in and beyond higher education are very much the same ones as in the replication crisis. As Ryan and others discuss in Leadership Reckoning, many of the claims in leader development simply don’t hold up to empirical scrutiny, much like so many findings in the field of science. However, just as the current issues in leader development assessment parallel those of the replication crisis, so too will its solutions.
The primary response to the replication crisis, especially within both Ryan’s and my discipline of psychology, has been to take a step back and examine our research methods. Many of the proposed solutions to this crisis mirror the principles outlined in Leadership Reckoning and Measuring the Mist, including triangulation, checking the personal biases of researchers, and maximizing the reliability and validity of measurements and manipulations. One popular solution has been pre-registration, meaning outlining expectations and research practices ahead of time to minimize the impact of potential biases and lower the risk of unethical behavior. A benefit of pre-registration (and the reason I pre-register all of my studies!) is that it helps you step back and think about the research questions you’re asking, especially regarding research design and measurement, in a way similar to what we ask course participants to do in their final projects.
Although the replication crisis is still ongoing, these strategies have been observed to bear fruit, and there is much to be hopeful for in the future of science. However, in addition to checking biases or pre-registering analyses, there is another important way to move past the replication crises both in broader science and in leader development, and that is community. By providing feedback and support, by creating safe environments to explore and brainstorm, and by ensuring that measurement and evaluation do not occur in echo chambers, community has a rich potential for positive impact on the scientific process. It’s our hope as we roll out this Newsletter that we can take a first step toward creating such a community for all of you, and we look forward to having more exciting news to share with you throughout 2024.