A Whirlwind Tour of Statistical Tests

A very important thing to keep in mind is that statistical tests don’t prove that hypotheses are true. That is, they can’t by their nature confirm one specific pattern given the infinite possibilities inherent in any judgment like that. Instead, statistical tests disprove false hypotheses. They can say that a result would probably never occur by chance, but they can’t tell you why that result did happen. This is an oft-misunderstood fact about statistics. The key in making inferences is to set up your hypothesis in a way such that you can give strong evidence for it by disproving something else. For example, in our earlier example of testing population means, we can say that it’s likely that Americans are taller than Koreans because, assuming that the populations had the same height and I knew a random sample of both, the chance that I would observe whatever I do is really small.

This hypothesis—that whatever pattern I’m seeing is not due to the pattern I hypothesize but instead due to random chance—is called the null hypothesis. It’s really what all of these statistical tests are testing. I’ll make sure to keep it very clear what hypotheses match up with which tests.

This brings us to our next point: of course, there’s no way I’d actually know a random sample of either of those populations. Almost any statistical test has one or more assumptions about its data, and these are also often misunderstood or not applied correctly.

This means that we should really think of any errors we have in two ways: known unknowns, and unknown unknowns. Just like in American foreign policy, it’s the unknown unknowns that you have to watch out for. Known unknowns are the errors that are reported by our tests: the statistical errors that result from getting lucky or unlucky. Unknown unknowns are the errors introduced by bad methodology, untrue assumptions, or unaccounted-for statistical effects. (Elsewhere on this blog I’ve talked about regression to the mean: this is an excellent example of an effect that can make tests that seem worthwhile actually completely useless.)

Of course, we don’t live in a perfect mathematical world and some error is inevitable. However, keep in mind that the further we stretch our assumptions the less useful our tests are, and our tests won’t be able to report that uncertainty to us. Every significance level or test statistic is conditioned on our applying statistics properly, our data coming from a good source, etc. I stress this because of just how easy it is to blindly trust numbers that come out of the computers to 10 decimal places. Keep in mind that what we get out is only as good as what we put in, and Python isn’t magic: it can’t tell us that our data is bad or that our hypothesis is misleading or incorrect. What that out of the way, how do we interpret our results assuming that our inputs are good?

Tests will usually give a test statistic, some calculation based on our data, and that test statistic will usually follow a test distribution. Given those two things, we can determine how unlikely such a test statistic is given the assumptions of the null hypothesis. This, in turn, gives us a p-value: the likelihood of observing the results we did assuming the conditions of the null hypothesis. This is another commonly misunderstood but very important concept in statistics. This is definitely not the chance we’re right. It’s, at best, the chance we aren’t completely wrong: the chance that anything we’re seeing is entirely due to random variation. In the sciences, a value of \(0.05\) is generally considered significant, although this changes a lot depending on context. (One way of interpreting this is that, if you pick \(0.05\) as a cutoff, one out of every twenty tests you run will be wrong and you won’t know. This means that running 500 tests will produce some significant results purely by chance!)

OK, the philosophizing is over. Let’s get to the test themselves!

This problem is another classic statistics question. Let’s say I’m arguing with my friend and they claim that soccer games are lame because, on average, there’s only two goals a game. Let’s test this claim against some recent Champions League games as of this writing:

# from November 5-6, 2019
total_scores = [
    2, 3, 2, 3, 6, 
    1, 6, 4, 0, 2,
    5, 8, 3, 4, 2,
    5
]

The test we’re going to use is called the Student’s t-test, named for William Sealy Gosset who wrote under the pseudonym Student. It’s pretty simple: it tests the null hypothesis that the mean of the whole population we’re sampling is equal to the given value.

stat, p = stats.ttest_1samp(total_scores, 2)
print("p =", round(p, 3))

p = 0.012

This is indicative that our friend is wrong: the data doesn’t support it. Note that this is two-sided: it doesn’t claim whether the true mean is larger or smaller. It’s a good rule to use two-sided tests in most cases: it’s not usually fair to assume the mean is above 2 or below it.

This test can be extended to work for two distributions. There’s actually two distinct ways this often pops up that we should distinguish: paired and unpaired. In paired tests, the two datasets are the same size and correspond with each other. An example of this would be testing whether a tutor significantly improved the SAT scores of their tutees. In the unpaired kind, we have different populations (perhaps of different sizes) and we want to compare their means. An example of this would be trying to determine if Champions League games have more goals than Premier League games. This is pretty simple to do in Python and has a familiar form:

# "ind" for "independent" meaning "unpaired"
stats.ttest_ind(champions, premier)
# "rel" for "related" meaning "paired"
stats.ttest_rel(before, after)

There are some assumptions here. We’re assuming that the data have the same variance (regardless of whether we know it), and that they’re distributed in the same way. This generally holds up well in real life, and it isn’t that far off if you bend the rules a little, but it’s worth remembering.

Let’s say we have some paired data in different units and we want to figure out if they’re related. An example of this would be asking whether SAT scores correlate with freshman GPA in college. The example we’ll use is gas mileage vs. weight for some car models:

I’ve drawn in the regression line. Note that data can have all sorts of relationships, and it’s a very difficult question to answer how two sets of numbers relate in general. We’re going to limit ourselves to a single feasible version of this problem: do higher values of miles per gallon correlate with lower values of weight, as they seem to here?

We can use several tests for this: the most common is Pearson’s correlation coefficient, or \(r\). This assumes that the \(x\) and \(y\) datasets are normally distributed, but fudging this a little doesn’t usually have too many problems, at least when you combine it with a graphical test like the one above. Let’s see what comes out:

# assuming we've loaded our data into df
stats.pearsonr(df['mpg'], df['weight'])

(-0.831740933244335, 2.9727995640500577e-103)

The test statistic here is quite common and easy to interpret on its own: -1 means perfect negative correlation (the data is perfectly linear with a negative slope), 0 means no correlation whatsoever, and 1 means a perfect positive correlation (the data is perfectly linear with a positive slope.) This is often squared to get the common \(R^2\) statistic, which is usually interpreted as the percentage of the variance in the y-values that can be explained by the x-values.

How extreme is this \(r\) value? Well, the p-value has to be some sort of record! The chance that this would happen by chance if miles per gallon and weight really had no correlation (the null hypothesis) is astronomically low, which makes sense when we see the data.

I should really stress that this is best used in combination with visuals. You can get really nonsensical results otherwise:

stats.pearsonr(x, y)

(0.8165214368885029, 0.002164602347197218)

So there’s a relationship, right? Well, maybe not: