Analysis of variance

(Condensed from the "NIST Engineering Statistics Handbook": Section 5.7. A Glossary of DOE Terminology.)[17]

Balanced design

An experimental design where all cells (i.e. treatment combinations) have the same number of observations.

Blocking

A schedule for conducting treatment combinations in an experimental study such that any effects on the experimental results due to a known change in raw materials, operators, machines, etc., become concentrated in the levels of the blocking variable. The reason for blocking is to isolate a systematic effect and prevent it from obscuring the main effects. Blocking is achieved by restricting randomization.

Design

A set of experimental runs which allows the fit of a particular model and the estimate of effects.

DOE

Design of experiments. An approach to problem solving involving collection of data that will support valid, defensible, and supportable conclusions.[18]

Effect

How changing the settings of a factor changes the response. The effect of a single factor is also called a main effect.

Error

Unexplained variation in a collection of observations. DOE's typically require understanding of both random error and lack of fit error.

Experimental unit

The entity to which a specific treatment combination is applied.

Factors

Process inputs that an investigator manipulates to cause a change in the output.

Lack-of-fit error

Error that occurs when the analysis omits one or more important terms or factors from the process model. Including replication in a DOE allows separation of experimental error into its components: lack of fit and random (pure) error.

Model

Mathematical relationship which relates changes in a given response to changes in one or more factors.

Random error

Error that occurs due to natural variation in the process. Random error is typically assumed to be normally distributed with zero mean and a constant variance. Random error is also called experimental error.

Randomization

A schedule for allocating treatment material and for conducting treatment combinations in a DOE such that the conditions in one run neither depend on the conditions of the previous run nor predict the conditions in the subsequent runs.[nb 1]

Replication

Performing the same treatment combination more than once. Including replication allows an estimate of the random error independent of any lack of fit error.

Responses

The output(s) of a process. Sometimes called dependent variable(s).

Treatment

A treatment is a specific combination of factor levels whose effect is to be compared with other treatments.

The fixed-effects model (class I) of analysis of variance applies to situations in which the experimenter applies one or more treatments to the subjects of the experiment to see whether the response variable values change. This allows the experimenter to estimate the ranges of response variable values that the treatment would generate in the population as a whole.

Random-effects model (class II) is used when the treatments are not fixed. This occurs when the various factor levels are sampled from a larger population. Because the levels themselves are random variables, some assumptions and the method of contrasting the treatments (a multi-variable generalization of simple differences) differ from the fixed-effects model.[19]

A mixed-effects model (class III) contains experimental factors of both fixed and random-effects types, with appropriately different interpretations and analysis for the two types.

Example: Teaching experiments could be performed by a college or university department to find a good introductory textbook, with each text considered a treatment. The fixed-effects model would compare a list of candidate texts. The random-effects model would determine whether important differences exist among a list of randomly selected texts. The mixed-effects model would compare the (fixed) incumbent texts to randomly selected alternatives.

Defining fixed and random effects has proven elusive, with competing definitions arguably leading toward a linguistic quagmire.[20]

The analysis of variance can be presented in terms of a linear model, which makes the following assumptions about the probability distribution of the responses:[21][22][23][24]

• Independence of observations – this is an assumption of the model that simplifies the statistical analysis.
• Normality – the distributions of the residuals are normal.
• Equality (or "homogeneity") of variances, called homoscedasticity — the variance of data in groups should be the same.

The separate assumptions of the textbook model imply that the errors are independently, identically, and normally distributed for fixed effects models, that is, that the errors (${\displaystyle \varepsilon }$) are independent and

${\displaystyle \varepsilon \thicksim N(0,\sigma ^{2}).\,}$

In a randomized controlled experiment, the treatments are randomly assigned to experimental units, following the experimental protocol. This randomization is objective and declared before the experiment is carried out. The objective random-assignment is used to test the significance of the null hypothesis, following the ideas of C. S. Peirce and Ronald Fisher. This design-based analysis was discussed and developed by Francis J. Anscombe at Rothamsted Experimental Station and by Oscar Kempthorne at Iowa State University.[25] Kempthorne and his students make an assumption of unit treatment additivity, which is discussed in the books of Kempthorne and David R. Cox.

In its simplest form, the assumption of unit-treatment additivity[nb 2] states that the observed response ${\displaystyle y_{i,j}}$ from experimental unit ${\displaystyle i}$ when receiving treatment ${\displaystyle j}$ can be written as the sum of the unit's response ${\displaystyle y_{i}}$ and the treatment-effect ${\displaystyle t_{j}}$, that is [26][27][28]

${\displaystyle y_{i,j}=y_{i}+t_{j}.}$

The assumption of unit-treatment additivity implies that, for every treatment ${\displaystyle j}$, the ${\displaystyle j}$th treatment has exactly the same effect ${\displaystyle t_{j}}$ on every experiment unit.

The assumption of unit treatment additivity usually cannot be directly falsified, according to Cox and Kempthorne. However, many consequences of treatment-unit additivity can be falsified. For a randomized experiment, the assumption of unit-treatment additivity implies that the variance is constant for all treatments. Therefore, by contraposition, a necessary condition for unit-treatment additivity is that the variance is constant.

The use of unit treatment additivity and randomization is similar to the design-based inference that is standard in finite-population survey sampling.

Kempthorne uses the randomization-distribution and the assumption of unit treatment additivity to produce a derived linear model, very similar to the textbook model discussed previously.[29] The test statistics of this derived linear model are closely approximated by the test statistics of an appropriate normal linear model, according to approximation theorems and simulation studies.[30] However, there are differences. For example, the randomization-based analysis results in a small but (strictly) negative correlation between the observations.[31][32] In the randomization-based analysis, there is no assumption of a normal distribution and certainly no assumption of independence. On the contrary, the observations are dependent!

The randomization-based analysis has the disadvantage that its exposition involves tedious algebra and extensive time. Since the randomization-based analysis is complicated and is closely approximated by the approach using a normal linear model, most teachers emphasize the normal linear model approach. Few statisticians object to model-based analysis of balanced randomized experiments.

However, when applied to data from non-randomized experiments or observational studies, model-based analysis lacks the warrant of randomization.[33] For observational data, the derivation of confidence intervals must use subjective models, as emphasized by Ronald Fisher and his followers. In practice, the estimates of treatment-effects from observational studies generally are often inconsistent. In practice, "statistical models" and observational data are useful for suggesting hypotheses that should be treated very cautiously by the public.[34]

The normal-model based ANOVA analysis assumes the independence, normality and homogeneity of variances of the residuals. The randomization-based analysis assumes only the homogeneity of the variances of the residuals (as a consequence of unit-treatment additivity) and uses the randomization procedure of the experiment. Both these analyses require homoscedasticity, as an assumption for the normal-model analysis and as a consequence of randomization and additivity for the randomization-based analysis.

However, studies of processes that change variances rather than means (called dispersion effects) have been successfully conducted using ANOVA.[35] There are no necessary assumptions for ANOVA in its full generality, but the F-test used for ANOVA hypothesis testing has assumptions and practical limitations which are of continuing interest.

Problems which do not satisfy the assumptions of ANOVA can often be transformed to satisfy the assumptions. The property of unit-treatment additivity is not invariant under a "change of scale", so statisticians often use transformations to achieve unit-treatment additivity. If the response variable is expected to follow a parametric family of probability distributions, then the statistician may specify (in the protocol for the experiment or observational study) that the responses be transformed to stabilize the variance.[36] Also, a statistician may specify that logarithmic transforms be applied to the responses, which are believed to follow a multiplicative model.[27][37] According to Cauchy's functional equation theorem, the logarithm is the only continuous transformation that transforms real multiplication to addition.

ANOVA uses traditional standardized terminology. The definitional equation of sample variance is ${\displaystyle s^{2}=\textstyle {\frac {1}{n-1}}\sum (y_{i}-{\bar {y}})^{2}}$, where the divisor is called the degrees of freedom (DF), the summation is called the sum of squares (SS), the result is called the mean square (MS) and the squared terms are deviations from the sample mean. ANOVA estimates 3 sample variances: a total variance based on all the observation deviations from the grand mean, an error variance based on all the observation deviations from their appropriate treatment means, and a treatment variance. The treatment variance is based on the deviations of treatment means from the grand mean, the result being multiplied by the number of observations in each treatment to account for the difference between the variance of observations and the variance of means.

The fundamental technique is a partitioning of the total sum of squares SS into components related to the effects used in the model. For example, the model for a simplified ANOVA with one type of treatment at different levels.

${\displaystyle SS_{\text{Total}}=SS_{\text{Error}}+SS_{\text{Treatments}}}$

The number of degrees of freedom DF can be partitioned in a similar way: one of these components (that for error) specifies a chi-squared distribution which describes the associated sum of squares, while the same is true for "treatments" if there is no treatment effect.

${\displaystyle DF_{\text{Total}}=DF_{\text{Error}}+DF_{\text{Treatments}}}$

See also Lack-of-fit sum of squares.

The F-test is used for comparing the factors of the total deviation. For example, in one-way, or single-factor ANOVA, statistical significance is tested for by comparing the F test statistic

${\displaystyle F={\frac {\text{variance between treatments}}{\text{variance within treatments}}}}$

${\displaystyle F={\frac {MS_{\text{Treatments}}}{MS_{\text{Error}}}}={{SS_{\text{Treatments}}/(I-1)} \over {SS_{\text{Error}}/(n_{T}-I)}}}$

where MS is mean square, ${\displaystyle I}$ = number of treatments and ${\displaystyle n_{T}}$ = total number of cases

to the F-distribution with ${\displaystyle I-1}$, ${\displaystyle n_{T}-I}$ degrees of freedom. Using the F-distribution is a natural candidate because the test statistic is the ratio of two scaled sums of squares each of which follows a scaled chi-squared distribution.

The expected value of F is ${\displaystyle 1+{n\sigma _{\text{Treatment}}^{2}}/{\sigma _{\text{Error}}^{2}}}$ (where ${\displaystyle n}$ is the treatment sample size) which is 1 for no treatment effect. As values of F increase above 1, the evidence is increasingly inconsistent with the null hypothesis. Two apparent experimental methods of increasing F are increasing the sample size and reducing the error variance by tight experimental controls.

There are two methods of concluding the ANOVA hypothesis test, both of which produce the same result:

• The textbook method is to compare the observed value of F with the critical value of F determined from tables. The critical value of F is a function of the degrees of freedom of the numerator and the denominator and the significance level (α). If F ≥ F_Critical, the null hypothesis is rejected.
• The computer method calculates the probability (p-value) of a value of F greater than or equal to the observed value. The null hypothesis is rejected if this probability is less than or equal to the significance level (α).

The ANOVA F-test is known to be nearly optimal in the sense of minimizing false negative errors for a fixed rate of false positive errors (i.e. maximizing power for a fixed significance level). For example, to test the hypothesis that various medical treatments have exactly the same effect, the F-test's p-values closely approximate the permutation test's p-values: The approximation is particularly close when the design is balanced.[30][41] Such permutation tests characterize tests with maximum power against all alternative hypotheses, as observed by Rosenbaum.[nb 3] The ANOVA F-test (of the null-hypothesis that all treatments have exactly the same effect) is recommended as a practical test, because of its robustness against many alternative distributions.[42][nb 4]

ANOVA consists of separable parts; partitioning sources of variance and hypothesis testing can be used individually. ANOVA is used to support other statistical tools. Regression is first used to fit more complex models to data, then ANOVA is used to compare models with the objective of selecting simple(r) models that adequately describe the data. "Such models could be fit without any reference to ANOVA, but ANOVA tools could then be used to make some sense of the fitted models, and to test hypotheses about batches of coefficients."[43] "[W]e think of the analysis of variance as a way of understanding and structuring multilevel models—not as an alternative to regression but as a tool for summarizing complex high-dimensional inferences ..."[43]

In the design of an experiment, the number of experimental units is planned to satisfy the goals of the experiment. Experimentation is often sequential.

Early experiments are often designed to provide mean-unbiased estimates of treatment effects and of experimental error. Later experiments are often designed to test a hypothesis that a treatment effect has an important magnitude; in this case, the number of experimental units is chosen so that the experiment is within budget and has adequate power, among other goals.

Reporting sample size analysis is generally required in psychology. "Provide information on sample size and the process that led to sample size decisions."[50] The analysis, which is written in the experimental protocol before the experiment is conducted, is examined in grant applications and administrative review boards.

Besides the power analysis, there are less formal methods for selecting the number of experimental units. These include graphical methods based on limiting the probability of false negative errors, graphical methods based on an expected variation increase (above the residuals) and methods based on achieving a desired confidence interval.[51]

Power analysis is often applied in the context of ANOVA in order to assess the probability of successfully rejecting the null hypothesis if we assume a certain ANOVA design, effect size in the population, sample size and significance level. Power analysis can assist in study design by determining what sample size would be required in order to have a reasonable chance of rejecting the null hypothesis when the alternative hypothesis is true.[52][53][54][55]

Several standardized measures of effect have been proposed for ANOVA to summarize the strength of the association between a predictor(s) and the dependent variable or the overall standardized difference of the complete model. Standardized effect-size estimates facilitate comparison of findings across studies and disciplines. However, while standardized effect sizes are commonly used in much of the professional literature, a non-standardized measure of effect size that has immediately "meaningful" units may be preferable for reporting purposes.[56]

It is prudent to verify that the assumptions of ANOVA have been met. Residuals are examined or analyzed to confirm homoscedasticity and gross normality.[57] Residuals should have the appearance of (zero mean normal distribution) noise when plotted as a function of anything including time and modeled data values. Trends hint at interactions among factors or among observations. One rule of thumb: "If the largest standard deviation is less than twice the smallest standard deviation, we can use methods based on the assumption of equal standard deviations and our results will still be approximately correct."[58]

A statistically significant effect in ANOVA is often followed up with one or more different follow-up tests. This can be done in order to assess which groups are different from which other groups or to test various other focused hypotheses. Follow-up tests are often distinguished in terms of whether they are planned (a priori) or post hoc. Planned tests are determined before looking at the data and post hoc tests are performed after looking at the data.

Often one of the "treatments" is none, so the treatment group can act as a control. Dunnett's test (a modification of the t-test) tests whether each of the other treatment groups has the same mean as the control.[59]

Post hoc tests such as Tukey's range test most commonly compare every group mean with every other group mean and typically incorporate some method of controlling for Type I errors. Comparisons, which are most commonly planned, can be either simple or compound. Simple comparisons compare one group mean with one other group mean. Compound comparisons typically compare two sets of groups means where one set has two or more groups (e.g., compare average group means of group A, B and C with group D). Comparisons can also look at tests of trend, such as linear and quadratic relationships, when the independent variable involves ordered levels.

Following ANOVA with pair-wise multiple-comparison tests has been criticized on several grounds.[56][60] There are many such tests (10 in one table) and recommendations regarding their use are vague or conflicting.[61][62]

Below we make clear the connection between multi-way ANOVA and linear regression.

Linearly re-order the data so that ${\displaystyle k^{\text{th}}}$ observation is associated with a response ${\displaystyle y_{k}}$ and factors ${\displaystyle Z_{k,b}}$ where ${\displaystyle b\in \{1,2,\ldots ,B\}}$ denotes the different factors and ${\displaystyle B}$ is the total number of factors. In one-way ANOVA ${\displaystyle B=1}$ and in two-way ANOVA ${\displaystyle B=2}$. Furthermore, we assume the ${\displaystyle b^{th}}$ factor has ${\displaystyle I_{b}}$ levels, namely ${\displaystyle \{1,2,\ldots ,I_{b}\}}$. Now, we can one-hot encode the factors into the ${\displaystyle \sum _{b=1}^{B}I_{b}}$ dimensional vector ${\displaystyle v_{k}}$.

The one-hot encoding function ${\displaystyle g_{b}:\{1,2,\ldots ,I_{b}\}\mapsto \{0,1\}^{I_{b}}}$ is defined such that the ${\displaystyle i^{th}}$ entry of ${\displaystyle g_{b}(Z_{k,b})}$ is

$${\displaystyle g_{b}(Z_{k,b})_{i}={\begin{cases}1&{\text{if }}i=Z_{k,b}\\0&{\text{otherwise}}\end{cases}}}$$

The vector ${\displaystyle v_{k}}$ is the concatenation of all of the above vectors for all ${\displaystyle b}$. Thus, ${\displaystyle v_{k}=[g_{1}(Z_{k,1}),g_{2}(Z_{k,2}),\ldots ,g_{B}(Z_{k,B})]}$. In order to obtain a fully general ${\displaystyle B}$-way interaction ANOVA we must also concatenate every additional interaction term in the vector ${\displaystyle v_{k}}$ and then add an intercept term. Let that vector be ${\displaystyle X_{k}}$.

With this notation in place, we now have the exact connection with linear regression. We simply regress response ${\displaystyle y_{k}}$ against the vector ${\displaystyle X_{k}}$. However, there is a concern about identifiability. In order to overcome such issues we assume that the sum of the parameters within each set of interactions is equal to zero. From here, one can use F-statistics or other methods to determine the relevance of the individual factors.

We can consider the 2-way interaction example where we assume that the first factor has 2 levels and the second factor has 3 levels.

Define ${\displaystyle a_{i}=1}$ if ${\displaystyle Z_{k,1}=i}$ and ${\displaystyle b_{i}=1}$ if ${\displaystyle Z_{k,2}=i}$, i.e. ${\displaystyle a}$ is the one-hot encoding of the first factor and ${\displaystyle b}$ is the one-hot encoding of the second factor.

With that,

$${\displaystyle X_{k}=[a_{1},a_{2},b_{1},b_{2},b_{3},a_{1}\times b_{1},a_{1}\times b_{2},a_{1}\times b_{3},a_{2}\times b_{1},a_{2}\times b_{2},a_{2}\times b_{3},1]}$$

where the last term is an intercept term. For a more concrete example suppose that

$${\displaystyle {\begin{aligned}Z_{k,1}&=2\\Z_{k,2}&=1\end{aligned}}}$$

Then,

$${\displaystyle X_{k}=[0,1,1,0,0,0,0,0,1,0,0,1]}$$