This past semester, the Doerr Leader Development Team took a fresh look at our Catalyst module, “Conflict Isn’t a Dirty Word.” Doerr’s Catalyst programs are two-part workshops, with meetings two or more weeks apart, that are aimed at developing specific leadership skills. Each module has unique, specific goals and outcomes that we evaluate to determine the training’s impact.
Originally, this conflict-management module and its evaluation focused on identifying and teaching the “best” styles students could use for responding to conflict. However, research suggests that conflict resolution is less about a style than it is a mindset of openness and an intent to seek positive solutions to interpersonal problems. The catalyst module has been updated to reflect this, and so the Research & Evaluation Team wanted to evaluate the impact of this change for the spring of 2024.
Our first step was to meet with the Leader Development Team and determine the specific, measurable outcomes that would define whether the course is having the desired effect. We came up with three criteria for success:
1. Students should recognize how they typically react to conflicts.
2. Students should be able and willing to understand others’ viewpoints without getting defensive or making a situation worse.
3. Students should have confidence in their ability to manage conflict effectively, rather than instinctively avoiding conflict out of fear.
We had some cognitive and emotional measures that could be tweaked for measurement purposes, but we were left with two issues. First, none of our measures were behavioral, meaning one domain of experience was left unaddressed—and when it comes to conflict, this domain is crucial! Second, all our measures relied on self-reports, meaning our ability to triangulate the efficacy of this Catalyst module was very limited.
Because of these issues, we set our sights on updating our approach to evaluating this training, including adding a measure of behavior. We adapted an intellectual humility measure that contained a list of behaviors that students could rate themselves as having performed in a recent conflict situation – e.g., “The last time I was involved in a conflict, disagreement, or argument, I viewed the challenging of my ideas as an opportunity to grow and learn.” We also created a task in which students respond to vignettes, supposedly taken from an online post disparaging Rice University and its students. Students’ evaluations of and responses to the disparaging post, as well as attempts to take the perspective of the original poster, would thus serve as our behavioral measure in both pre and post-test evaluations. To this end, we worked with our Leader Development Team to create a detailed scoring guide for this measure so that it could be interpreted effectively and objectively.
However, in line with the principle of triangulation (discussed at length in Measuring the Mist), we also wanted a source of data aside from participants to address the potential for response-shift bias, among other such biases. Because we can’t watch students in action as they have real-world conflicts, we decided to use a peer-based feedback method instead, asking students to provide the email addresses of six or seven people who know them well. After this training is finished, we plan to send these peer evaluators a brief measure that assesses whether they’ve observed any changes in training participants’ conflict responses, as well as whether they’ve observed the participants becoming more creative. We included this creativity measure as a non-dependent variable, which is a useful feature of many training-related studies in which a non-trained group is not available for comparison. If we find changes in observed creativity, which is unrelated to the training (and thus should not change from pre- to post-training), it will indicate that we should be suspicious of any other changes reported by peers in our study.
Per the “Levels of Design for Program Assessment” rubric from the Demystifying Measurement course, the program assessment for this conflict training is a “Level 3”—a single group with pre and post measures that fit program objectives, plus measures coming from multiple data types and sources that cut across multiple domains of experience (emotion, cognition, behavior), and the addition of a non-dependent variable. This makes it a strong design, despite not being a true experiment with random assignment to groups (an RCT).
Even with this design boost, there is always room for improvement in our assessment system. If you have any thoughts, questions, or ideas about how to better assess conflict management skills, or about program evaluation/impact assessment at all, please don’t hesitate to let us know. We would love to hear from you!