Power and Size Advantages of Exact Parametric Methods

XPro procedures are based on generalized tests and confidence intervals. This not only provides superior power for most problems of ANOVA under unequal variances, but also provides substantially better size performance compared to classical procedures. Two examples are shown below; additional examples can be found in the references listed on this page.

Example 1: Power Implications of Generalized F-Test

Suppose you have a data set such as the one shown below for comparing the mean effects of two treatments and a placebo. After a preliminary analysis, you believe that the differences in treatment means are statistically significant. Although these data were indeed generated from a normal population with unequal means and variances, application of the classical F-test will not support your intuition in this situation at all, because the p- value of the usual F-test is as large as .2323.

XPro computes the p-value for testing the equality of treatment means both under the (unreasonable) assumption of equal variances and without that assumption. The p-values of the two sets computed using XPro are as follows:

P-value without the equal variances assumption (the generalized F-test of XPro): .0388

P-value under the equal variances assumption (the classical F-test from any software package): .2323

The discrepancy of the p-values in this example is quite dramatic. It clearly demonstrates the serious weakness of the classical F-test in the presence of heterogeneity. Because it ignores the problem of heterogeneity, the classical F-test failes to detect significant differences in treatments, despite the fact that the data provides sufficient information to do so.

Are you willing to risk your imporatnt findings from an experiment when the lack of power of a test is so low? In applications such as biomedical experiments, weakness of a procedure at this magnitude is unacceptable, especially when experiments are expensive and time consuming. This example shows how exact methods can help conserve resources by concluding experiments in a timely manner at a reduced cost.

Example 2: Size Performance, a comparison with widely used MLE based Methods

Variance components analytical tools available from some widely used general purpose software packages are based on outdated asymptotic methods (e.g. MLE based methods discussed below) that deliver poor size performance, even with large samples. Presented below are results from an extensive simulation study comparing the size performance of XPro methods versus MLE based asymptotic methods. This comparison and underlying exact methods have been presented at ASA courses and technical sessions. Although the study findings have implications for all higher-way balanced mixed models, we present here the results as applied to a simple a x n one-way layout (with n being the number of repetitions), so that one can easily reproduce the results and check our claims. In this simulation, the error variance is fixed at 1 and the variance component of the random factor is set to three values around 1. The following table shows Type I error of each test when the test is carried out at the .05 confidence level (using 10,000 simulated samples). This also represents the rate at which the corresponding 95% confidence intervals did not contain the true value of the variance component.

Type I (false positive) Error of Competing .05-Size Tests

It is evident from this simulation study that regardless of how large n is, the Type I error rates of the asymptotic test range between approximately .2 and .6, unless the number of levels of the random factor is unusually large. XPro computes exact an/or size guaranteed p-values and confidence intervals on variance components of random effects and mixed models. It also allows one to compare variance components from two balanced models.

References

[1]. Krutchkoff, R.G. (1988). "One-Way Fixed Effects Analysis of Variance When the Error Variances May be Unequal", Journal of Statistics, Computational Simulation, 30, 177-183.

[2]. Thursby, J.G. (1992). "A Comparison of Several Exact and Approximate Tests for Structured Shift Under Hetereoscedasticity", Journal of Econometrics, 53, 363-386.

[3]. Weerahandi, S. (1995). Exact Statistical Methods for Data Analysis, Springer-erlag, New York.

[4]. Weerahandi, S. (1995). "ANOVA Under Unequal Error Variances", Biometrics, 51, 589-599.

[5]. Zhou, L. and Mathew, T. (1994), "Some Tests for Variance Components Using Generalized p-Values", Technometrics, 4, 394-402.

[6]. Weerahandi, S. (1991), "Testing Variance Components in Mixed Models with Generalized p Values," Journal of the American Statistical Association, 86, 151-153.

[7]. Chi, E. and Weerahandi, S (1998), "Comparing Treatments Under Growth Curve Models with Compound-Symmetric Covariance Structure: Exact Tests Using Generalized p-Values,", To appear in JSPI.