Cook's distance

In statistics, Cook's distance or Cook's D is a commonly used estimate of the influence of a data point when performing a least-squares regression analysis.^[1] In a practical ordinary least squares analysis, Cook's distance can be used in several ways: to indicate influential data points that are particularly worth checking for validity; or to indicate regions of the design space where it would be good to be able to obtain more data points. It is named after the American statistician R. Dennis Cook, who introduced the concept in 1977.^[2][3]

Definition

Data points with large residuals (outliers) and/or high leverage may distort the outcome and accuracy of a regression. Cook's distance measures the effect of deleting a given observation. Points with a large Cook's distance are considered to merit closer examination in the analysis. For the algebraic expression, first define

${\displaystyle {\underset {n\times 1}{\mathbf {y} }}={\underset {n\times p}{\mathbf {X} }}\quad {\underset {p\times 1}{\boldsymbol {\beta }}}\quad +\quad {\underset {n\times 1}{\boldsymbol {\epsilon }}}}$

where ${\displaystyle {\boldsymbol {\epsilon }}\sim {\mathcal {N}}\left(0,\sigma ^{2}\mathbf {I} \right)}$ is the error term, ${\displaystyle {\boldsymbol {\beta }}=\left[\beta _{0}\,\beta _{1}\dots \beta _{p-1}\right]^{\mathsf {T}}}$ is the coefficient matrix, and $\mathbf {X}$ is the design matrix including a constant. The least squares estimator then is ${\displaystyle \mathbf {b} =\left(\mathbf {X} ^{\mathsf {T}}\mathbf {X} \right)^{-1}\mathbf {X} ^{\mathsf {T}}\mathbf {y} }$, and consequently the fitted (predicted) values for the mean of $\mathbf {y}$ are

${\displaystyle \mathbf {\hat {y}} =\mathbf {X} \mathbf {b} =\mathbf {X} \left(\mathbf {X} ^{\mathsf {T}}\mathbf {X} \right)^{-1}\mathbf {X} ^{\mathsf {T}}\mathbf {y} =\mathbf {H} \mathbf {y} }$

where ${\displaystyle \mathbf {H} \equiv \mathbf {X} (\mathbf {X} ^{\mathsf {T}}\mathbf {X} )^{-1}\mathbf {X} ^{\mathsf {T}}}$ is the projection matrix (or hat matrix). The $i$-th diagonal element of $\mathbf{H} \,$, given by ${\displaystyle h_{i}\equiv \mathbf {x} _{i}^{\mathsf {T}}(\mathbf {X} ^{\mathsf {T}}\mathbf {X} )^{-1}\mathbf {x} _{i}}$,^[4] is known as the leverage of the $i$-th observation. Similarly, the $i$-th element of the residual vector ${\displaystyle \mathbf {e} =\mathbf {y} -\mathbf {\hat {y}} =\left(\mathbf {I} -\mathbf {H} \right)\mathbf {y} }$ is denoted by $e_{i}$. With this, we can define Cook's distance as

${\displaystyle D_{i}={\frac {e_{i}^{2}}{s^{2}p}}\left[{\frac {h_{i}}{(1-h_{i})^{2}}}\right],}$

where ${\displaystyle s^{2}\equiv \left(n-p\right)^{-1}\mathbf {e} ^{\top }\mathbf {e} }$ is the mean squared error of the regression model.^[5]

Detecting highly influential observations

There are different opinions regarding what cut-off values to use for spotting highly influential points. A simple operational guideline of $D_i>1$ has been suggested.^[6] Others have indicated that $D_i>4/n$, where $n$ is the number of observations, might be used.^[7]

A conservative approach relies on the fact that Cook's distance has the form W/p, where W is formally identical to the Wald statistic that one uses for testing that $H_0:\beta_i=\beta_0$ using some $\hat{\beta}_{[-i]}$. Recalling that W/p has an $F_{p,n-p}$ distribution (with p and n-p degrees of freedom), we see that Cook's distance is equivalent to the F statistic for testing this hypothesis, and we can thus use $F_{p,n-p, 1-\alpha}$ as a threshold.^[8]

Interpretation

Specifically $D_{i}$ can be interpreted as the distance one's estimates move within the confidence ellipsoid that represents a region of plausible values for the parameters. This is shown by an alternative but equivalent representation of Cook's distance in terms of changes to the estimates of the regression parameters between the cases, where the particular observation is either included or excluded from the regression analysis.

See also

• Outlier
• Leverage (statistics)
• Partial leverage
• DFFITS
• Studentized residual

References

1. Mendenhall, William; Sincich, Terry (1996). A Second Course in Statistics: Regression Analysis (5th ed.). Upper Saddle River, NJ: Prentice-Hall. p. 422. ISBN 0-13-396821-9. A measure of overall influence an outlying observation has on the estimated $\beta$ coefficients was proposed by R. D. Cook (1979). Cook's distance, D_i, is calculated...
2. Cook, R. Dennis (February 1977). "Detection of Influential Observations in Linear Regression". Technometrics. American Statistical Association. 19 (1): 15–18. doi:10.2307/1268249. JSTOR 1268249. MR 0436478.
3. Cook, R. Dennis (March 1979). "Influential Observations in Linear Regression". Journal of the American Statistical Association. American Statistical Association. 74 (365): 169–174. doi:10.2307/2286747. JSTOR 2286747. MR 0529533.
4. Hayashi, Fumio (2000). Econometrics. Princeton University Press. pp. 21–23.
5. "Statistics 512: Applied Linear Models" (PDF). Purdue University.
6. Cook, R. Dennis; Weisberg, Sanford (1982). Residuals and Influence in Regression. New York, NY: Chapman & Hall. ISBN 0-412-24280-X.
7. Bollen, Kenneth A.; Jackman, Robert W. (1990). Fox, John; Long, J. Scott, eds. Regression Diagnostics: An Expository Treatment of Outliers and Influential Cases. Modern Methods of Data Analysis. Newbury Park, CA: Sage. pp. 257–91. ISBN 0-8039-3366-5.
8. Aguinis, Herman; Gottfredson, Ryan K.; Joo, Harry (2013). "Best-Practice Recommendations for Defining Identifying and Handling Outliers" (PDF). Organizational Research Methods. Sage. 16 (2): 270–301. doi:10.1177/1094428112470848. Retrieved 4 December 2015.

Further reading

• Atkinson, Anthony; Riani, Marco (2000). "Deletion Diagnostics". Robust Diagnostics and Regression Analysis. New York: Springer. pp. 22–25. ISBN 0-387-95017-6.
• Heiberger, Richard M.; Holland, Burt (2013). "Case Statistics". Statistical Analysis and Data Display. Springer Science & Business Media. pp. 312–27. ISBN 9781475742848.
• Krasker, William S.; Kuh, Edwin; Welsch, Roy E. (1983). "Estimation for dirty data and flawed models". Handbook of Econometrics. 1. Elsevier. pp. 651–698. doi:10.1016/S1573-4412(83)01015-6.
• Aguinis, Herman; Gottfredson, Ryan K.; Joo, Harry (2013). "Best-Practice Recommendations for Defining Identifying and Handling Outliers" (PDF). Organizational Research Methods. Sage. 16 (2): 270–301. doi:10.1177/1094428112470848. Retrieved 4 December 2015.