VIRTUAL JOURNAL CLUB

This study measured the relationships between common admissions metrics (i.e., undergraduate GPA and GRE scores) and PhD completion. The sample included nearly 4000 physics PhD students enrolled in 27 PhD programs from a variety of schools; the students were roughly representative of the student body in physics. The study looked at four analytic subsets of the data: All Students, US Only, US Women, and US Men. The work provides evidence that most of the commonly used admissions metrics have limited statistical validity, and those that have statistical validity, have limited practical benefit. It was also shown that scores on the GRE physics exams have strong gaps based on race, gender, and country of origin. Additional statistics were reported about the demographic composition of physics as a discipline, from undergraduate degrees awarded through PhDs completed.

GRE-QUANTITATIVE

An assessment “of basic mathematical skills, understanding of elementary mathematical concepts, and ability to reason quantitatively and to model and solve problems with quantitative methods. The content in these areas includes high school mathematics and statistics at a level that is generally no higher than a second course in algebra; it does not include trigonometry, calculus or other higher-level mathematics.”

GRE-VERBAL

An assessment of “ability to analyze and evaluate written material and synthesize information obtained from it, analyze relationships among component parts of sentences and recognize relationships among words and concepts.”

GRE-PHYSICS

“100 five-choice questions, some of which are grouped in sets and based on such materials as diagrams, graphs, experimental data and descriptions of physical situations. The aim of the test is to determine the extent of the examinees’ grasp of fundamental principles and their ability to apply these principles in the solution of problems.”

PHD COMPLETION RATE

Percentage of matriculants that earn the PhD in that program.

PREDICTIVE VALIDITY

The extent to which a score on a scale or test predicts some criterion or measure. In this case, we measure the extent to which undergrad GPA, GRE scores, and other variables studied predict PhD completion in Physics.

MULTIVARIATE LOGISTIC REGRESSION

Analysis method in which two or more independent variables are taken into consideration simultaneously to predict the value of a binary dependent variable.

BIVARIATE CORRELATION

A measure, r, with a value between +1 and −1 of the linear correlation between two variables. There are various opinions on qualifying the strength of a correlation, but the following seem to be reasonably widely accepted: |r|<0.1 = negligible; |r| is 0.1-0.3 = weak; |r| is 0.3-0.7 moderate; |r| >0.7 strong. Bivariate correlations suffer intrinsically from confounding, i.e., leaving out relevant independent variables.

TIER

A three-level categorical variable based on the National Research Council’s ranking of physics PhD programs.

• Cumulative undergraduate GPA was a statistically significant predictor of PhD completion for all analytic samples; the largest effect was for US Women, for whom, it is the only admissions metric with statistical significance.
• GRE-Quantitative was statistically significant in the aggregate All Students model (p=0.003) and just barely statistically significant in the US Only model (p=0.048). It was not statistically significant in the US Women or US Men analytic samples.
• GRE-Physics and GRE-Verbal were not statistically significant predictors of PhD completion for any of the groups analyzed. Gender was not statistically significant in the All Students or US Only models.
• Tier was a statistically significant parameter for nearly all of the models, suggesting that individuals at highly ranked programs are more likely to earn the PhD. As this work is correlational, the reason for this cannot be identified by this study, but several plausible explanations are given.

The combination of the GREs’ limited validity, their demographics-based score differences, and the prevalence with which minimum scores are used by physics admissions committees implies the community is doing more harm than good to itself. Continued overreliance on metrics that do not predict Ph.D. completion but have large gaps based on demographics works against both the fairness of admissions practices and the health of physics as a discipline.

This study implies an urgent need for additional research that improves admissions processes in physics (and beyond?). The community ought to reevaluate admissions criteria and practices to ensure that selection is both equitable and effective for identifying students who can be successful. This will necessitate programs and the community creative thinking and new research about a holistic approach to graduate admissions and identifying strategies that connect prospective students to graduate programs in which they will thrive.