Welch's t-test

Student's t-test assumes that the sample means (test statistics) of two population distributions being compared are normally distributed with equal variance. Welch's t-test is designed for unequal sample distribution variance, but the assumption of sample distribution normality is maintained.[1] Welch's t-test is an approximate solution to the Behrens–Fisher problem.

Welch's t-test defines the statistic t by the following formula:

${\displaystyle t\quad =\quad {\;{\overline {X}}_{1}-{\overline {X}}_{2}\; \over {\sqrt {\;{s_{1}^{2} \over N_{1}}\;+\;{s_{2}^{2} \over N_{2}}\quad }}}\,}$

where ${\displaystyle {\overline {X}}_{j}}$, ${\displaystyle s_{j}}$ and ${\displaystyle N_{j}}$ are the ${\displaystyle j^{th}}$ sample mean, sample standard deviation and sample size, respectively, ${\displaystyle j\in \{1,2\}}$. Unlike in Student's t-test, the denominator is not based on a pooled variance estimate.

The degrees of freedom ${\displaystyle \nu }$  associated with this variance estimate is approximated using the Welch–Satterthwaite equation:

${\displaystyle \nu \quad \approx \quad {{\left(\;{s_{1}^{2} \over N_{1}}\;+\;{s_{2}^{2} \over N_{2}}\;\right)^{2}} \over {\quad {s_{1}^{4} \over N_{1}^{2}\nu _{1}}\;+\;{s_{2}^{4} \over N_{2}^{2}\nu _{2}}\quad }}}$

Here ${\displaystyle \nu _{1}=N_{1}-1}$, the degrees of freedom associated with the first variance estimate. ${\displaystyle \nu _{2}=N_{2}-1}$, the degrees of freedom associated with the 2nd variance estimate.

The statistic is approximately from the t-distribution since we have an approximation of the chi-square distribution. This approximation is better done when both ${\displaystyle N_{1}}$ and ${\displaystyle N_{2}}$ are larger than 5.[4][5]

Once t and ${\displaystyle \nu }$ have been computed, these statistics can be used with the t-distribution to test one of two possible null hypotheses:

• that the two population means are equal, in which a two-tailed test is applied; or
• that one of the population means is greater than or equal to the other, in which a one-tailed test is applied.

The approximate degrees of freedom are rounded down to the nearest integer.

Welch's t-test is more robust than Student's t-test and maintains type I error rates close to nominal for unequal variances and for unequal sample sizes under normality. Furthermore, the power of Welch's t-test comes close to that of Student's t-test, even when the population variances are equal and sample sizes are balanced.[2] Welch's t-test can be generalized to more than 2-samples,[6] which is more robust than one-way analysis of variance (ANOVA).

It is not recommended to pre-test for equal variances and then choose between Student's t-test or Welch's t-test.[7] Rather, Welch's t-test can be applied directly and without any substantial disadvantages to Student's t-test as noted above. Welch's t-test remains robust for skewed distributions and large sample sizes.[8] Reliability decreases for skewed distributions and smaller samples, where one could possibly perform Welch's t-test.[9]

The following three examples compare Welch's t-test and Student's t-test. Samples are from random normal distributions using the R programming language.

For all three examples, the population means were ${\displaystyle \mu _{1}=20}$ and ${\displaystyle \mu _{2}=22}$.

The first example is for equal variances (${\displaystyle \sigma _{1}^{2}=\sigma _{2}^{2}=4}$) and equal sample sizes (${\displaystyle N_{1}=N_{2}=15}$). Let A1 and A2 denote two random samples:

${\displaystyle A_{1}=\{27.5,21.0,19.0,23.6,17.0,17.9,16.9,20.1,21.9,22.6,23.1,19.6,19.0,21.7,21.4\}}$

${\displaystyle A_{2}=\{27.1,22.0,20.8,23.4,23.4,23.5,25.8,22.0,24.8,20.2,21.9,22.1,22.9,20.5,24.4\}}$

The second example is for unequal variances (${\displaystyle \sigma _{1}^{2}=16}$, ${\displaystyle \sigma _{2}^{2}=1}$) and unequal sample sizes (${\displaystyle N_{1}=10}$, ${\displaystyle N_{2}=20}$). The smaller sample has the larger variance:

${\displaystyle {\begin{aligned}A_{1}&=\{17.2,20.9,22.6,18.1,21.7,21.4,23.5,24.2,14.7,21.8\}\\A_{2}&=\{21.5,22.8,21.0,23.0,21.6,23.6,22.5,20.7,23.4,21.8,20.7,21.7,21.5,22.5,23.6,21.5,22.5,23.5,21.5,21.8\}\end{aligned}}}$

The third example is for unequal variances (${\displaystyle \sigma _{1}^{2}=1}$, ${\displaystyle \sigma _{2}^{2}=16}$) and unequal sample sizes (${\displaystyle N_{1}=10}$, ${\displaystyle N_{2}=20}$). The larger sample has the larger variance:

${\displaystyle {\begin{aligned}A_{1}&=\{19.8,20.4,19.6,17.8,18.5,18.9,18.3,18.9,19.5,22.0\}\\A_{2}&=\{28.2,26.6,20.1,23.3,25.2,22.1,17.7,27.6,20.6,13.7,23.2,17.5,20.6,18.0,23.9,21.6,24.3,20.4,24.0,13.2\}\end{aligned}}}$

Reference p-values were obtained by simulating the distributions of the t statistics for the null hypothesis of equal population means (${\displaystyle \mu _{1}-\mu _{2}=0}$). Results are summarised in the table below, with two-tailed p-values:

	Sample A1			Sample A2			Student's t-test				Welch's t-test
Example	${\displaystyle N_{1}}$	${\displaystyle {\overline {X}}_{1}}$	${\displaystyle s_{1}^{2}}$	${\displaystyle N_{2}}$	${\displaystyle {\overline {X}}_{2}}$	${\displaystyle s_{2}^{2}}$	${\displaystyle t}$	${\displaystyle \nu }$	${\displaystyle P}$	${\displaystyle P_{\mathrm {sim} }}$	${\displaystyle t}$	${\displaystyle \nu }$	${\displaystyle P}$	${\displaystyle P_{\mathrm {sim} }}$
1	15	20.8	7.9	15	23.0	3.8	−2.46	28	0.021	0.021	−2.46	24.9	0.021	0.017
2	10	20.6	9.0	20	22.1	0.9	−2.10	28	0.045	0.150	−1.57	9.9	0.149	0.144
3	10	19.4	1.4	20	21.6	17.1	−1.64	28	0.110	0.036	−2.22	24.5	0.036	0.042

Welch's t-test and Student's t-test gave identical results when the two samples have identical variances and sample sizes (Example 1). But note that if you sample data from populations with identical variances, the sample variances will differ, as will the results of the two t-tests. So with actual data, the two tests will almost always give somewhat different results.

For unequal variances, Student's t-test gave a low p-value when the smaller sample had a larger variance (Example 2) and a high p-value when the larger sample had a larger variance (Example 3). For unequal variances, Welch's t-test gave p-values close to simulated p-values.

• Student's t-test
• Z-test
• Factorial experiment
• One-way analysis of variance
• Hotelling's two-sample T-squared statistic, a multivariate extension of Welch's t-test