Efficiency (statistics)

In the comparison of various statistical procedures, efficiency is a measure of quality of an estimator, of an experimental design,^[1] or of a hypothesis testing procedure.^[2] Essentially, a more efficient estimator, experiment, or test needs fewer observations than a less efficient one to achieve a given performance. This article primarily deals with efficiency of estimators.

The relative efficiency of two procedures is the ratio of their efficiencies, although often this concept is used where the comparison is made between a given procedure and a notional "best possible" procedure. The efficiencies and the relative efficiency of two procedures theoretically depend on the sample size available for the given procedure, but it is often possible to use the asymptotic relative efficiency (defined as the limit of the relative efficiencies as the sample size grows) as the principal comparison measure.

Efficiencies are often defined using the variance or mean square error as the measure of desirability.^[1]

Estimators

The efficiency of an unbiased estimator, T, of a parameter θ is defined as ^[3]

e(T) = \frac{1/\mathcal{I}(\theta)}{\mathrm{var}(T)}

where $\mathcal{I}(\theta)$ is the Fisher information of the sample. Thus e(T) is the minimum possible variance for an unbiased estimator divided by its actual variance. The Cramér–Rao bound can be used to prove that e(T) ≤ 1.

Efficient estimators

Main article: Efficient estimator

If an unbiased estimator of a parameter θ attains $e(T) = 1$ for all values of the parameter, then the estimator is called efficient.

Equivalently, the estimator achieves equality in the Cramér–Rao inequality for all θ.

An efficient estimator is also the minimum variance unbiased estimator (MVUE). This is because an efficient estimator maintains equality on the Cramér–Rao inequality for all parameter values, which means it attains the minimum variance for all parameters (the definition of the MVUE). The MVUE estimator, even if it exists, is not necessarily efficient, because "minimum" does not mean equality holds on the Cramér–Rao inequality.

Thus an efficient estimator need not exist, but if it does, it is the MVUE.

Asymptotic efficiency

Some estimators can attain efficiency asymptotically and are thus called asymptotically efficient estimators. This can be the case for some maximum likelihood estimators or for any estimators that attain equality of the Cramér–Rao bound asymptotically.

Example

Consider a sample of size $N$ drawn from a normal distribution of mean $\mu$ and unit variance, i.e., $X_n \sim \mathcal{N}(\mu, 1).$

The sample mean, ${\overline {X}}$ , of the sample $X_1, X_2, \ldots, X_N$ , defined as

\overline{X} = \frac{1}{N} \sum_{n=1}^{N} X_n \sim \mathcal{N}\left(\mu, \frac{1}{N}\right).

The variance of the mean, 1/N (the square of the standard error) is equal to the reciprocal of the Fisher information from the sample and thus, by the Cramér–Rao inequality, the sample mean is efficient in the sense that its efficiency is unity (100%).

Now consider the sample median, $\widetilde{X}$ . This is an unbiased and consistent estimator for $\mu$ . For large $N$ the sample median is approximately normally distributed with mean $\mu$ and variance ${\pi}/{2N},$ i.e.,^[4]

{\widetilde {X}}\sim {\mathcal {N}}\left(\mu ,{\frac {\pi }{2N}}\right).

The efficiency for large $N$ is thus

e\left({\widetilde {X}}\right)=\left({\frac {1}{N}}\right)\left({\frac {\pi }{2N}}\right)^{-1}=2/\pi \approx 64\%.

Note that this is the asymptotic efficiency — that is, the efficiency in the limit as sample size $N$ tends to infinity. For finite values of $N,$ the efficiency is higher than this (for example, a sample size of 3 gives an efficiency of about 74%).

The sample mean is thus more efficient than the sample median in this example. However, there may be measures by which the median performs better. For example, the median is far more robust to outliers, so that if the Gaussian model is questionable or approximate, there may advantages to using the median (see Robust statistics).

Dominant estimators

If $T_{1}$ and $T_{2}$ are estimators for the parameter $\theta$ , then $T_{1}$ is said to dominate $T_{2}$ if:

its mean squared error (MSE) is smaller for at least some value of $\theta$
the MSE does not exceed that of $T_{2}$ for any value of θ.

Formally, $T_{1}$ dominates $T_{2}$ if

\operatorname {E} [(T_{1}-\theta )^{2}]\leq \operatorname {E} [(T_{2}-\theta )^{2}]

holds for all $\theta$ , with strict inequality holding somewhere.

Relative efficiency

The relative efficiency of two estimators is defined as

e(T_{1},T_{2})={\frac {\operatorname {E} [(T_{2}-\theta )^{2}]}{\operatorname {E} [(T_{1}-\theta )^{2}]}}

Although $e$ is in general a function of $\theta$ , in many cases the dependence drops out; if this is so, $e$ being greater than one would indicate that $T_{1}$ is preferable, whatever the true value of $\theta$ .

An alternative to relative efficiency for comparing estimators, is the Pitman closeness criterion. This replaces the comparison of mean-squared-errors with comparing how often one estimator produces estimates closer to the true value than another estimator.

Estimators of u.i.d. variables

In estimating the mean of uncorrelated, identically distributed variables we can take advantage of the fact that the variance of the sum is the sum of the variances. In this case efficiency can be defined as the square of the coefficient of variation, i.e.,^[5]

\operatorname {E} \equiv \left({\frac {s}{i}}gma\mu \right)^{2}

Relative efficiency of two such estimators can thus be interpreted as the relative sample size of one required to achieve the certainty of the other. Proof:

{\frac {\operatorname {E} _{1}}{\operatorname {E} _{2}}}={\frac {s_{1}^{2}}{s_{2}^{2}}}.

Now because $s_1^2 = n_1 \sigma^2, \, s_2^2 = n_2 \sigma^2$ we have ${\frac {\operatorname {E} _{1}}{\operatorname {E} _{2}}}={\frac {n_{1}}{n_{2}}}$ so the relative efficiency expresses the relative sample size of the first estimator needed to match the variance of the second.

Robustness

Efficiency of an estimator may change significantly if the distribution changes, often dropping. This is one of the motivations of robust statistics – an estimator such as the sample mean is an efficient estimator of the population mean of a normal distribution, for example, but can be an inefficient estimator of a mixture distribution of two normal distributions with the same mean and different variances. For example, if a distribution is a combination of 98% N(μ, σ) and 2% N(μ, 10σ), the presence of extreme values from the latter distribution (often "contaminating outliers") significantly reduces the efficiency of the sample mean as an estimator of μ. By contrast, the trimmed mean is less efficient for a normal distribution, but is more robust (less affected) by changes in distribution, and thus may be more efficient for a mixture distribution. Similarly, the shape of a distribution, such as skewness or heavy tails, can significantly reduce the efficiency of estimators that assume a symmetric distribution or thin tails.

Uses of inefficient estimators

Further information: L-estimator § Applications

While efficiency is a desirable quality of an estimator, it must be weighed against other considerations, and an estimator that is efficient for certain distributions may well be inefficient for other distributions. Most significantly, estimators that are efficient for clean data from a simple distribution, such as the normal distribution (which is symmetric, unimodal, and has thin tails) may not be robust to contamination by outliers, and may be inefficient for more complicated distributions. In robust statistics, more importance is placed on robustness and applicability to a wide variety of distributions, rather than efficiency on a single distribution. M-estimators are a general class of solutions motivated by these concerns, yielding both robustness and high relative efficiency, though possibly lower efficiency than traditional estimators for some cases. These are potentially very computationally complicated, however.

A more traditional alternative are L-estimators, which are very simple statistics that are easy to compute and interpret, in many cases robust, and often sufficiently efficient for initial estimates. See applications of L-estimators for further discussion.

Hypothesis tests

For comparing significance tests, a meaningful measure of efficiency can be defined based on the sample size required for the test to achieve a given task power.^[6]

Pitman efficiency^[7] and Bahadur efficiency (or Hodges–Lehmann efficiency)^[8]^[9] relate to the comparison of the performance of statistical hypothesis testing procedures. The Encyclopedia of Mathematics provides a brief exposition of these three criteria.

Experimental design

Further information: Optimal design

For experimental designs, efficiency relates to the ability of a design to achieve the objective of the study with minimal expenditure of resources such as time and money. In simple cases, the relative efficiency of designs can be expressed as the ratio of the sample sizes required to achieve a given objective.^[10]

Notes

1 2 Everitt 2002, p. 128.
↑ Nikulin, M.S. (2001), "Efficiency of a statistical procedure", in Hazewinkel, Michiel, Encyclopedia of Mathematics, Springer, ISBN 978-1-55608-010-4
↑ Fisher, R (1921). "On the Mathematical Foundations of Theoretical Statistics". Philosophical Transactions of the Royal Society of London. Series A. 222: 309–368. JSTOR 91208.
↑ Williams, D. (2001) Weighing the Odds, CUP. ISBN 052100618X (p.165)
↑ Grubbs, Frank (1965). Statistical Measures of Accuracy for Riflemen and Missile Engineers. pp. 26–7.
↑ Everitt 2002, p. 321.
↑ Nikitin, Ya.Yu. (2001), "Efficiency, asymptotic", in Hazewinkel, Michiel, Encyclopedia of Mathematics, Springer, ISBN 978-1-55608-010-4
↑ Arcones M. A. "Bahadur efficiency of the likelihood ratio test" preprint
↑ Canay I. A. & Otsu, T. "Hodges–Lehmann Optimality for Testing Moment Condition Models"
↑ Dodge, Y. (2006) The Oxford Dictionary of Statistical Terms, OUP. ISBN 0-19-920613-9

References

Everitt, Brian S. (2002). The Cambridge Dictionary of Statistics. Cambridge University Press. ISBN 0-521-81099-X.
Lehmann, Erich L. (1998). Elements of Large-Sample Theory. New York: Springer Verlag. ISBN 978-0-387-98595-4.
Nikitin, Ya.Yu. (2001), "Efficiency, asymptotic", in Hazewinkel, Michiel, Encyclopedia of Mathematics, Springer, ISBN 978-1-55608-010-4

Statistics

Descriptive statistics

Continuous data

Center	Mean arithmetic geometric harmonic Median Mode

Dispersion	Variance Standard deviation Coefficient of variation Percentile Range Interquartile range

Shape	Moments Skewness Kurtosis L-moments

Count data

Index of dispersion

Summary tables

Dependence

Graphics

Data collection

Study design	Population Statistic Effect size Statistical power Sample size determination Missing data

Survey methodology	Sampling Standard error stratified cluster Opinion poll Questionnaire

Controlled experiments	Design control optimal Controlled trial Randomized Random assignment Replication Blocking Interaction Factorial experiment

Uncontrolled studies	Observational study Natural experiment Quasi-experiment

Statistical inference

Statistical theory

Frequentist inference

Point estimation	Estimating equations Maximum likelihood Method of moments M-estimator Minimum distance Unbiased estimators Mean-unbiased minimum-variance Rao–Blackwellization Lehmann–Scheffé theorem Median unbiased Plug-in

Interval estimation	Confidence interval Pivot Likelihood interval Prediction interval Tolerance interval Resampling Bootstrap Jackknife

Testing hypotheses	1- & 2-tails Power Uniformly most powerful test Permutation test Randomization test Multiple comparisons

Parametric tests	Likelihood-ratio Wald Score

Specific tests

Z (normal) Student's t-test F

Goodness of fit	Chi-squared Kolmogorov–Smirnov Anderson–Darling Normality (Shapiro–Wilk) Likelihood-ratio test Model selection Cross validation AIC BIC

Rank statistics	Sign Sample median Signed rank (Wilcoxon) Hodges–Lehmann estimator Rank sum (Mann–Whitney) Nonparametric anova 1-way (Kruskal–Wallis) 2-way (Friedman) Ordered alternative (Jonckheere–Terpstra)

Bayesian inference

Correlation	Pearson product–moment Partial correlation Confounding variable Coefficient of determination

Regression analysis	Errors and residuals Regression model validation Mixed effects models Simultaneous equations models Multivariate adaptive regression splines (MARS)

Linear regression	Simple linear regression Ordinary least squares General linear model Bayesian regression

Non-standard predictors	Nonlinear regression Nonparametric Semiparametric Isotonic Robust Heteroscedasticity Homoscedasticity

Generalized linear model	Exponential families Logistic (Bernoulli) / Binomial / Poisson regressions

Partition of variance	Analysis of variance (ANOVA, anova) Analysis of covariance Multivariate ANOVA Degrees of freedom

Categorical / Multivariate / Time-series / Survival analysis

Categorical

Multivariate

Time-series

General	Decomposition Trend Stationarity Seasonal adjustment Exponential smoothing Cointegration Structural break Granger causality

Specific tests	Dickey–Fuller Johansen Q-statistic (Ljung–Box) Durbin–Watson Breusch–Godfrey

Time domain	Autocorrelation (ACF) partial (PACF) Cross-correlation (XCF) ARMA model ARIMA model (Box–Jenkins) Autoregressive conditional heteroskedasticity (ARCH) Vector autoregression (VAR)

Frequency domain	Spectral density estimation Fourier analysis Wavelet

Survival

Survival function	Kaplan–Meier estimator (product limit) Proportional hazards models Accelerated failure time (AFT) model First hitting time

Hazard function	Nelson–Aalen estimator

Test	Log-rank test

Applications

Biostatistics	Bioinformatics Clinical trials / studies Epidemiology Medical statistics

Engineering statistics	Chemometrics Methods engineering Probabilistic design Process / quality control Reliability System identification

Social statistics	Actuarial science Census Crime statistics Demography Econometrics National accounts Official statistics Population statistics Psychometrics

Spatial statistics	Cartography Environmental statistics Geographic information system Geostatistics Kriging

Category
Portal
Commons
WikiProject

This article is issued from Wikipedia - version of the 11/30/2016. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.