Structuring Decision-Making to Support Equity and Transparency: A Case Study

Institutions strive to be data-driven, data-informed, or data-supported organizations. However, we often fall short of being “data-anything” for a variety of reasons, including the need to make decisions quickly, lack of the right data, or not having enough information. The desire to be data-informed is important as organizations work to become more efficient, effective, and equitable. Leaders also regularly hear the desire for more transparent decision making, as well as to allow for more input from a variety of groups. Many institutions struggle to use data to objectively prioritize the options we have or to determine where we should spend our resources such as staff time and funding, in addition to achieving all of these pieces listed above.

 

One strategy we have devised is to use a decision-making matrix. The value of this matrix is the objectivity and flexibility that it provides to all types of projects. Student affairs leaders often spend time trying to understand situations, gather information, and communicate consistent facts about their students and staff. We then hope our colleagues will use this information to adapt and improve their programs, services, and activities. While this may happen informally, our proposed decision-making matrix adds structure to this process to support professionals making significant and meaningful decisions. This structure helps to set priorities and enact organization values. Individuals and groups of all positions and roles within the organization can use this matrix as a foundation for making decisions using data.

 

Case Study

We will present the decision-making matrix as a case-study to provide a clear understanding of how it can be used in practice. The matrix was initially designed to support a large campus project that spanned every unit, division, college, and school. Project stakeholders were tasked with deciding how to prioritize more than thirty recommendations to improve efforts and initiatives on campus. The challenge is that all of the recommendations were identified as the most valuable and critical to support the needs of students, staff, and faculty. And yet, the challenge was deciding on the next steps and direction given the variety of perspectives and the importance of each recommendation. To implement this matrix we used the following components: criteria, scale points, analysis & interpretation, and reporting.

 

Criteria: to help support a structured decision-making strategy, we devised the use of a matrix that used several criteria.

• In this case study, the criteria we discussed initially included the following: the level of difficulty for implementation, the potential for impact to current students, the potential for impact to current faculty, the potential for impact to current staff, the urgency/need to address the issue, the collaboration required, and the cost/resources needed.
• The criteria can be adjusted to meet the needs of any stakeholders where projects may have all these criteria or they may focus on a few criteria. Upon further discussion with stakeholders, we determined the project in this case-study required us to focus on two. By eliminating the other criteria, we were able to identify what was most important to the next steps of this project. For example, the potential for impact and level of urgency became more important for our case-study project.
• Different types of criteria can be used depending on the project. For example, it may be relevant to include a scaffolding criteria that outlines decisions/projects that need to be taken prior to certain projects to ensure success.

 

Scale Points: it is recommended to use a six-point scale to allow for variation and spread among decisions. This means there is more opportunity for certain projects to rise to the top. Because criteria can be interpreted differently (e.g. how one defines urgency, impact, or high cost), each of the criteria and scale points were defined to promote consistency across evaluators. The following table outlines examples of the scale points used in the case-study.

 

Table 1: Original Decision-Making Matrix (full table, with all defined scale points available by emailing kruchen@colorado.edu)

• Criteria used in this example: potential impact to current students, urgency/ need to address the problem, collaboration/scope, cost/resources needed for full implementation.
• Scale points used in this example: low difficulty (1), slightly difficult (2), somewhat difficult (3), moderate difficulty (4), very difficult (5), extremely difficult (6).

 

To simplify the process for evaluators, the scale may be reduced to three points. In this case study, the six point scale presented a challenge in defining each scale point for each of the two criteria. This challenged our ability to promote inter-rater reliability and ultimately, we moved forward with a three point scale (1=low, 2=moderate, 3=high) for each of the two criteria. Additionally, if evaluators felt they did not have enough knowledge or information about a specific recommendation to be able to adequately assess its urgency and/or potential impact, they were given the opportunity to select “I am unable to assess.”

 

Table 2: Simplified Decision-Making Matrix (full table available by emailing kruchen@colorado.edu)

• Criteria used in this example: urgency (if swift action is not taken, negative results are expected to occur within the year) and impact (considers breadth and/or significance of impact on population.
• Scale points used in this example: low (1), moderate (2), high (3), unable to assess

 

Analysis: Data analysis for this project included several key metrics depicted in Table 3. Urgency and impact criteria frequencies and means were calculated for each of the recommendations. Additionally, a Total Score was assigned for each criteria and respective recommendation. For the case study, the Total Score was the sum of the numeric values for low (1), moderate (2), and high (3) ratings across all 19 participants. Ratings of “unable to assess” were assigned a numeric value of zero (0). The minimum and maximum scores were calculated and the range split into thirds to determine the threshold for each of the Overall Levels (low, moderate, or high).

 

Table 3: Decision Matrix Summary Statistics (available if you email kruchen@colorado.edu)

 

Finally, the total points were used to plot each recommendation by their level of urgency and potential impact to identify which recommendations rose to the top in terms of high urgency and high impact. The scatter plot, produced in Microsoft in Power BI, can be found in Figure 1.

 

Figure 1: Decision Matrix Scatter Plot: Urgency Total Score by Impact Total Score (available if you email kruchen@colorado.edu)

 

Reporting: The purpose of creating this decision-making matrix was to help campus stakeholders make decisions about how to prioritize 36 recommendations for initiatives and efforts on campus. With this in mind, stakeholders received a summary of the key findings. This summary identified the 10 recommendations that fell into the High Urgency and High Impact area of the scatter plot based on their total points in each criterion.

 

The list of recommendations provided to us were also grouped by category and population of impact (e.g. undergraduates, graduates, faculty, staff, finance, etc.). Hearing from stakeholders that they were considering moving forward with at least one priority from each category, the summary report also identified the top recommendations for each category. Appendices of the statistics table and the scatter plot were also included.

 

Final Note

Over the past decade, institutions have worked to collect and analyze data to understand and improve the student experience. The next challenge we face is using the information and findings to prioritize and support initiatives. A decision-making matrix provides a structured and documented approach to increase transparency, efficiency, and to become a data-driven organization. If your organization is striving to accomplish these three goals, establishing a decision-making matrix is necessary and visionary.  

 

Please contact us (kruchen@colorado.edu or crystal.cyr@colorado.edu) if you have questions about implementing this process.

 

Kimberly Kruchen-Spaulding, Student Affairs Director of Assessment, Research and Data Analytics at the University of Colorado Boulder. Crystal Cyr, Assessment Specialist in the Student Affairs Office of Assessment and Planning at the University of Colorado Boulder.