Bayesian Fallacies, Part 1: Of Test Scores, Ski Jumps, and Diseases

For those of you who are not recent US high school students, let me provide some background. In the US, the PSAT is a nationally-normed test administered every fall in most high schools across the country. It has two distinct purposes. Firstly, it prepares students for the SAT, the most important test for college admissions. Secondly, students in the top 16,000 test takers (allocated proportionally by state, so cutoffs differ depending on what school you go to) qualify for National Merit Semifinalist status (which I’ll shorten to just “National Merit”), which can lead to college scholarship money. We’ll be focusing on this function of the test, because it provides a natural example of the statistical environment that produces the phenomena discussed above.

National Merit qualification has an interesting wrinkle: unlike the SAT or ACT, which most colleges encourage students to take as many times as they desire, a student can only qualify for National Merit in the test administered their third year of high school, which is a single test date picked by their school. This creates an interesting problem. For now, let’s pretend that PSAT scores don’t improve at all between sophomores and juniors: obviously this is false, but it’s a useful statistical cheat. Let’s also pretend that qualifying scores don’t change between years. Given these assumptions, let’s say a student takes the exam in 10th grade and gets a score that would qualify for National Merit if they were in 11th grade. What are the chances that student, retaking the test next year, actually does qualify for National Merit?

This problem feels like a classic case of regression to the mean. National Merit is a harsh cutoff: out of roughly 1.5 million eligible students who take it every year, only 16,000 qualify. That’s a little over 1%. Anyone who qualifies is likely to have had a good day and gotten lucky with what questions were asked. Given that, our intuition that the qualifiers on separate test administrations should heavily overlap is likely to be incorrect. We’ll put some hard numbers to this qualitative understanding and see how it holds up.

Let’s do a couple things to make this problem easier to model mathematically. Let’s assume that each individual test taker’s scores have two components: a mean performance \(\mu\) that represents what their average performance would be across many different test administrations, and a normally-distributed random variation with standard deviation \(\sigma\) that accounts for all of the things that differ between different test administrations: different questions, how people are feeling, etc.

You might wonder if PSAT scores really are normally distributed. Obviously this isn’t a perfect approximation (the actual scores have to be multiples of 10, if nothing else), but due to the way tests are normed and the beauty of the central limit theorem this is actually pretty good. Here’s a plot of a normal distribution against the actual percentiles from the past three years:

Not bad, huh?

What’s a good ballpark for the individual variability \(\sigma\)? Some really interesting math can happen if you allow this to vary between test takers, but this is already going to be long, so I’ll save that for another time and just set this as constant across test takers. The College Board, which administers the PSAT, gives the score range for any individual test as plus or minus 40 points. If we assume that this is plus or minus two standard deviations, that gives us a standard deviation of 20 points, which is what we’ll use for now.

The analysis doesn’t really depend on the exact choices of numbers, but because we’re using a case study from the real world let’s try and pin some ballpark figures down. The PSAT is scored on a scale between 320 and 1520. Although National Merit qualification uses a slightly different score, we’re going to stick with this one: our analysis won’t really change either way, and the raw PSAT score has more available data. The median score last year among juniors was 1000, which makes sense because 500 is the mean score that the PSAT and SAT are normed at. The mean score was 1014, and the standard deviation was 197. Note the distinction between the \(\sigma\) above, which is the variation within a single individual (alternatively, the error margins of the test as a measure of ability), and this standard deviation, which is the variability between different individuals.

An astute reader might notice that combining two normal distributions, one with standard deviation \(20\) and one with standard deviation \(197\), will actually produce a normal distribution with a larger variance than just the one normal distribution with \(\sigma = 197\). The formula for this behavior is that the standard deviation of the combined distribution is the geometric mean of the two original standard deviations: the variances add. So, in order to get a test score distribution that’s \(197\), we need to find the \(S\) such that \(S^2 + 20^2 + 197^2\). Solving this gives us \(195.98\), which I’ll just approximate as \(S = 196\).

One last thing we’ll do is use a clean nationwide 1% cutoff for National Merit: last year, this was 1460. This makes the math a lot easier, and it’s still the same underlying problem: the only thing that would change is the parameter values we set above. We’re interested in the abstract ideas, not anyone’s particular chances at National Merit.

With these assumptions, we can reframe this problem mathematically. We’ll use lowercase letters for individuals, and uppercase letters for the global parameters. An individual test taker’s score is a random variable \(s \sim N(\mu, \sigma)\), where \(\mu\) itself is distributed as \(\mu \sim N(M, S)\), which in this case is \(N(1014, 196)\). \(\sigma\), in this example, is \(20\).

We’re interested in a conditional probability question: given that a single score drawn with some fixed \(\mu\) is above \(1460\), what is the probability that a second score drawn with the same \(\mu\) is also above \(1460\), and how does that compare with the base 1% chance that any score hits the threshold? This might feel more reminiscent of the Bayesian statistics we did earlier.

We’re going to analyze the math for this in more detail, later. But, for now, let’s do a basic estimate by just simulating a bunch of different tests and test takers with a computer. We’ll simulate 1.5 million different test takers, have each of them take two tests, and compare the distribution of the second scores from those who qualified the first time and the general population:

86% of the simulated test takers who qualified the first time qualified the second, compared to about 1% of the general population. That’s a big difference! Why is this effect not as strong as other stories of regression to the mean? Another way of looking at the data is plotting the second scores against the first scores, where we also see a very strong correlation:

We can think of two distinct effects at play: a selection effect that makes the first-time qualifiers’ \(\mu\) values higher on average, and a reverse regression effect that makes the random variation on a second test administration unlikely to be as high as the first one. We can look at the \(\mu\) and random variations for both simulated test administrations to see this in action. First, the difference in \(\mu\) values showing the selection effect:

This effect is enormous, because it’s virtually impossible to get a 1460 without a mean score far above average, regardless of how lucky you get.

This graph shows the random variations from the qualifiers in the first test versus the general population. We can see a small difference, which is the regression effect: basically, the random variations for the first-time qualifiers on the second test administration should look like the lower graph instead of the higher one, and so there’s a slight decrease from the first scores. In our sample, this meant that the people who qualified on the first test had a 57% chance of scoring lower the second time: in this sense, regression to the mean is occurring. However, this effect clearly pales in comparison to the selection effect above, and so in this example we see that regression to the mean is a small effect.