Thursday, 31 August 2006

If you don't submit an SAT score, one will be imputed for you

This New York Times article (þ: Margaret Soltan) probably makes more of the vague trend towards deemphasis of the SAT in college admissions than is probably justified. Then again, I’m one of those weird social scientists who thinks that psychometrics are reliable and valid measures of student abilities, albeit—like all measures—subject to error. The real issue with the SAT is not its psychometric foundations or learning effects from “test prep,” but rather its wide error bounds, which make it too advantageous for students to repeat the exam. The scale of the numbers probably psychologically amplifies this effect; put the score on a range from 1.0 to 4.0, and I suspect you’d see retake rates plummet with absolutely no other changes to the exam.

Even though most admissions committees probably don’t do this in a very sophisticated way (at least, not yet, although one suspects that some of the SAT-optional trend can be attributed to admissions committee innumeracy or hostility towards numeric measures than any real problem with the SAT), the lack of SAT scores can be worked around with some fancy stats: you can just impute the missing data from the information you do have (mean SAT scores, likely available at the school or school district level; GPA; some measure of school quality; grades in math and English classes), albeit with an adjustment to account for an important selection effect: that the SAT score, which is probably known to the student, is more likely to have been reported if it is above the mean imputation (my gut suspicion is that reporting is distributed complementary log-log about the mean SAT score).

1 comment:

Any views expressed in these comments are solely those of their authors; they do not reflect the views of the authors of Signifying Nothing, unless attributed to one of us.
[Permalink] 1. Simon Spero wrote @ Fri, 1 Sep 2006, 11:22 am CDT:

An admissions committee for a SLAC is usually not a statistical superpower, but modern university information systems are getting closer to real world tools in terms of data mining capabilities. Whether or not this is done depends on which motivation is the real driving force behind making submitting scores optional (as opposed to mandatory, or prohibited).

Two motivations have been offered:
1) belief in limited validity of SAT-like measures as predictor of academic performance.
2) strategic behavior to improve US News & World Report rankings

If the second motivation is the dominant one, then I would expect to see the relationship between average scores and first year dropout rates to remain stable, even at optional schools; if the first motivation is dominant, then I would expect to see higher first year dropout rates at optional schools.

If the first year dropout rate does increase, it may be possible to impute a more accurate average SAT score for an optional school.

Hmmmmm. Any quants in the house?

Comments are now closed on this post.