Outliers and Standard Deviation Summary

Here’s a reminder of the key points you should know about outliers and standard deviation.

It’s possible for a dataset to have extreme values, called outliers. 

…Which can make some descriptive statistics misleading.

An outlier that is the result of an error is called an anomaly.

If an outlier is an anomaly, we should clean the data by removing it.

And if it isn’t, we need to leave it in, because it gives us important information, even if it skews our descriptive statistics.

To figure out if there are outliers in a data set, statisticians create cut-off values called fences.

Any data point lower than this… or greater than this… is an outlier.

Now, there are two strategies to determine where these fences should go.

One strategy is to use rules built from quartiles.

One for the lower fence.

[Q1-1.5(Q3-Q1), write formula on the lower fence]

And one for the upper fence.

            [Q3 + 1.5 (Q3-Q1), write formula on the upper fence]

Where this is the interquartile range. [Q_3 – Q_1, replace by IQR]

            [Q1-1.5(IQR)]

[Q3 + 1.5 (IQR)]

And this is a constant. [1.5]

…Which is often 1.5 but can vary, depending on the dataset.

A second strategy is to use rules in this form

Where this represents a measure of spread called the standard deviation. []

The standard deviation is the average distance between data points and the mean.

And to find the standard deviation of a dataset…

Start by finding all of the deviance scores, which is the distance between a given data point and the mean.

We can represent deviance scores using this notation.

Once we have the deviance scores for all data points..

Square them…

And add them.

We can also represent the sum of square deviations like this.

Finally, divide this by the total number of data points, [(x-x)2n] to find the variance. [variance=(x-x)2n]

Which is the standard deviation squared. [2=(x-x)2n]

So to find the standard deviation, just take the square root of this.[=(x-x)2n

Both standard deviation and variance are examples of measures of spread.

Now, it can take an unnecessarily long time to start by finding the sum of squared deviations.

Fortunately, there’s a shortcut we can use to find standard deviation.
First, calculate the total number of data points, the sum of all the data points, and the sum of squares.

Second, find the mean of the squares… and the square of the mean…

Third, use this formula to find the variance.
And take the square root if you want the standard deviation.

When the dataset is given in a frequency table, we can use the same shortcut to find the variance and standard deviation.

Though now, the number of data points is given by the sum of the frequency counts. [f]

And since we need to multiply each data point [highlight x] by its frequency count first [f], we represent the formula like this. [fx, fx2] 

[2  =fx2f- (fxf)2  ]

And when the dataset is given in a grouped frequency table, we can only estimate the variance and standard deviation.

First, find the midpoints of each class interval.

Finally, in your exam, you could be asked to compare two data sets.

And in that case, you need to compare one measure of central location, and one measure of spread.

If the datasets don’t have outliers, use the mean and standard deviation.

But if there are outliers, the mean and standard deviation can be misleading, so it’s more appropriate to use the median and interquartile range.