VIRTUAL JOURNAL CLUB

This article examines the impact of a holistic review screening process on advancing racial equity in graduate admissions. In this study, 547 applicants to a psychiatry residency program were evaluated for interview selection via three distinct screening rubrics: one holistic review approach and two non-holistic processes (Traditional and Traditional Modified). Relative to Traditional, Holistic Review significantly increased the odds of URM applicants being selected for interviews. Researchers found that assigning value to specific lived experiences (i.e., resilience and distance traveled) and de-emphasizing elements with documented racial biases (i.e., test scores) contributed to the significant increase in odds ratio of interview selection for URM applicants.

This article examines the impact of a holistic review screening process on advancing racial equity in graduate admissions. In this study, 547 applicants to a psychiatry residency program were evaluated for interview selection via three distinct screening rubrics: one holistic review approach and two non-holistic processes (Traditional and Traditional Modified). Relative to Traditional, Holistic Review significantly increased the odds of URM applicants being selected for interviews. Researchers found that assigning value to specific lived experiences (i.e., resilience and distance traveled) and de-emphasizing elements with documented racial biases (i.e., test scores) contributed to the significant increase in odds ratio of interview selection for URM applicants.

HOLISTIC REVIEW

According to the Association of American Medical Colleges: “a flexible, individualized way of assessing an applicant’s capabilities by which balanced consideration is given to experiences, attributes, and academic metrics…”

DISTANCE TRAVELLED

“trajectory relative to family or community-level barriers reflecting marginalization at a population or structural level, e.g., first-generation college graduate, raised in a community with high poverty/low educational resources” (p. 35)

RESILIENCE

“achievement in enduring adversity—e.g. personal setback, illness, discrimination” (p. 35)

RUBRIC

A tool used to guide the systematic evaluation of applicants on multiple dimensions. Rubrics typical contain multiple criteria, each of which has multiple score levels (ie. low, medium, high).

ODDS RATIO

An odds ratio represents the odds of a specific outcome given specific exposure to a specific intervention, relative to the odds of that outcome without the exposure to an intervention.

LEADERSHIP

Broader than academics, the researchers looked for evidence of three dimensions of leadership: duration, intensity, and achievements

Traditional interview selection methods systematically exclude URM applicants from consideration.

Despite the fact that URM applicants were more likely to describe having meaningfully participated in endeavors aligning with program priorities, the Traditional Modified approach (i.e., which added a search for diversity-related keywords in each application) was not sufficient to result in a significant change in the odds of interview selection for URM applicants.

In contrast, the probability of selecting URMs for an interview doubled when using a holistic review process, relative to the traditional. The holistic review process was designed to align with the program’s values, and included

(1) identifying and devaluing metrics that have known bias and limited predictive value for long-term clinical strength (e.g., test scores and induction into honor societies),

(2) reimagining and prioritizing personal qualities and professional characteristics that reflect program values, and

(3) actively considering applicants in a broader social context—including acknowledgment of how institutional racism, poverty, and family educational achievement can impact applicant trajectory.

• In this study, researchers found that the traditional candidate evaluation process in use by the admissions committee in the study systematically excluded URM applicants from the candidate pool. For graduate programs interested in advancing equity, a data dive that reviews and examines how URM applicants fare at various stages of the application and review process will be helpful for identifying areas for intervention.
• The holistic review process that was designed and implemented by the admissions committee in this study was operationalized to “embody diverse forms of excellence” (p.41) that are reflective of the students who were traditionally excluded from their interviews. An equity-minded and mission-driven rubric can ensure that faculty are utilizing a more expansive view of excellence when reviewing applicant admission files.
• In universities where the explicit consideration of race is not permissible by law, evidence in this paper suggests options for advancing racial diversity and equity in admissions include: 1) explicit consideration of applicants’ contributions to diversity in the discipline, in one’s department, etc.; 2) consideration of distance travelled and/or resilience.
• The admissions committee that was subject of review in this study employed a number of practices that are consistent with an equity-minded holistic review process. Those practices included: 1) training for all individuals that were participating in the evaluation process; 2) an analysis of the effectiveness of the review process for advancing equity at the beginning and end of the admissions cycle; 3) revising the selection process in ways that reflect the values of your program/discipline.

1. The method of evaluating applicants described in this study moves away from quota like recruitment of individual URM applicants (“how many were selected?”) and toward one that is aligned with the program’s values, including acknowledging that excellence may manifest itself in many ways.

• What strategies might you use to pursue such a change within your program?
• How might such a change affect other aspects of graduate education in your program?

2. The researchers in this study provided two attributes (Resilience & Distance-traveled) on which applicants can be assessed that align with the program’s diversity values.

• What other attributes might align with your program’s diversity values?
• How might you break these down into high, medium, and low for an admissions rubric?

1. Posselt et al. (2019): Metrics first, diversity later?

“faculty must revisit the narrow framing they have traditionally used in the admissions process to increase diversity in their graduate programs.”

2. Quinn (2020): Rubrics mitigate implicit racial bias in grading

“On a vague grade-level evaluation scale, teachers rated a student writing sample lower when it was randomly signaled to have a Black author, versus a White author. However, there was no evidence of racial bias when teachers used a rubric with more clearly defined evaluation criteria. [There was] no evidence that the magnitude of grading bias depends on teachers’ implicit or explicit racial attitudes”

3. Grabowski (2017): Effects of holistic review in medical admissions

“Using mission-driven, holistic admissions criteria comprised of applicant attributes and experiences in addition to academic metrics resulted in a more diverse interview pool than using academic metrics alone.”

4. Bastedo et al. (2018): Admissions officers’ views of holistic review

“…admissions officers with a ‘whole context’ view of holistic review were disproportionately likely to admit a low socioeconomic-status applicant.”