Hypothesis Testing: Summary

Here’s a reminder of the key points you should know about hypothesis testing. 

‘Statistics’ describe or summarise data from samples…

Whereas ‘parameters’ describe or summarise data from populations.

In the real world, we normally need to use statistics to estimate the values of parameters.

But because they’re estimates, it’s important that we never ‘accept’ those parameter values.

Instead, good statisticians keep testing their estimates, to make sure they’re good estimates.

For example, suppose we estimate that 22% of people in the UK have brown eyes. 

To test that estimate, we could take a new sample of 150 people and look at their eye colour.

It’s theoretically possible we’d see anywhere between 0 and 150 people with brown eyes.

But the probability of some of those outcomes is so low that, if we observe them, we should assume that our parameter estimate is incorrect. 

The question is: at what point do these outcomes, and therefore our parameter estimate, become too unlikely to believe?

Well, it varies depending on the context and the statistician.

But wherever they draw the line, statisticians call this region the ‘critical region’.

Any outcome in the critical region is so unlikely that, if that’s what we observe, we should reject the parameter estimate we started with.

There are some very unlikely outcomes on the left of the distribution, too. 

So we can also create critical regions with two ‘tails’.

…where the probability of getting an outcome in one tail is always about equal to the probability of getting an outcome from the other tail. 

Whether we create one or two tails depends on the context.

If we suspect the true parameter value might be lower than our estimate, we use a critical region like this.

If we suspect the true parameter value might be higher than our estimate, we use a critical region like this.

And if we suspect the real parameter value might be different to their estimate, but we’re not sure in what way, we use a critical region like this.

The first value inside a critical region is called a ‘critical value’.

And critical values allow us to describe critical regions with inequalities.

This region is called the ‘acceptance region’, even though we never actually ‘accept’ parameter estimates as the correct value.

Next, since we can’t know the true parameter value, we often make errors without realising.

If we fail to reject an estimate that’s secretly incorrect, that’s called a ‘type 2’ error.

If we do reject an estimate that’s secretly correct, that’s called a ‘type 1’ error.

And the probability of making a type 1 error is equal to the probability represented by the critical region.

Here’s a mnemonic which may help you remember which is which.

Making a type 1 error involves rejecting the parameter estimate. 

Making a type 2 error involves accepting the parameter estimate.

‘Reject’ has 1 consonant in the middle.

‘Accept’ has 2.

Any result we observe in the critical region is called a ‘significant’ result.

The significance level of an experiment determines how big our critical region is and is equal to the probability of making a type 1 error.

When we’re working with two tails, we need to split the probability given by the significance level evenly between the two tails.

And actually, there’s a difference between the stated significance level, and the exact probability of finding an outcome in the critical region, since those rarely line up.

The stated version is just called the ‘significance level’, whilst the actual probability is usually referred to as the ‘actual significance level’.

A statistician can choose any significance level they like.

But 1%, 5% and 10% are especially common.

There are two slightly different ways of creating the critical region.

We could ensure that our actual significance level is no greater than the stated one.

Or, we could ensure that our actual significance level is as close as possible to the stated significance level – even if that means going over a bit.

We find critical regions by using the binomial cumulative distribution function on our calculators. And we speed up the process if we create a table of values.

Now, the process we’ve just seen…

That is…

Taking parameter estimates…

Taking a sample…  

Seeing how the sample statistic compares with the parameter estimate…

And depending on how significant the statistic is, potentially rejecting the parameter estimate…

That’s collectively called ‘hypothesis testing’.

Although, there’s a bit more technical language to bring in.

First, another term for ‘parameter’ is ‘population parameter’.

And the statistic we get from our new sample is usually referred to as a ‘test statistic’.

Second, whenever we have a statement that we can test in some way, it’s called a hypothesis. 

In our example, we’re testing the hypothesis: “22% of people in the UK have brown eyes.”

Although actually, when we’re hypothesis testing in the conventional way, we need two hypotheses: the null hypothesis, and the alternative hypothesis.

The null hypothesis states what we currently believe to be true.

And the alternative hypothesis states some alternative.

Third, any hypothesis that can be tested by using a probability distribution can be referred to as a ‘statistical hypothesis’.

So you’ll sometimes see hypothesis testing referred to as statistical hypothesis testing instead.

Fourth, hypothesis tests in which the critical region has one tail are called one-tailed tests.

And hypothesis tests in which the critical region has two tails are called two-tailed tests.

Fifth, we represent the null and alternative hypothesis using mathematical notation. 

What goes on the right of the colon depends on whether we’re running a one-tailed test…

Or a two-tailed test.

Finally then, the exam can’t ask you to conduct an entire experiment yourself.

So instead, they may tell you a sample was taken, tell you the significance level, and ask you to perform the rest of the hypothesis test. 

Here’s what that looks like for a one-tailed test. 

First, write down the null and alternative hypotheses.

Second, write down the distribution we’re using.

Third, find and state the critical region.

Fourth, compare the test statistic to the critical region.

And fifth, write a conclusion.

An alternative method is to find the cumulative probability associated with the test statistic instead.

Then, compare that to the significance level and conclude.

In the exam you can generally do it either way.

That is, unless they specifically ask you to do it one way.

The working looks very similar for a two-tailed test.

First, write down the null and alternative hypotheses.

Second, write down the distribution we’re using.

Third, find and state the critical region, now with two tails.

Fourth, compare the test statistic to the critical region.

And fifth, write a conclusion.

Alternatively, find the cumulative probability associated with the test statistic instead.

69 out of 80 is 86.25%, so if it falls in either tail it would be the one on the left.

So, calculate this probability

Then, compare that to half the significance level, since this is a two-tailed test, and conclude.