Module 08 · Comparing two groups
Choose a method that matches the study design, then estimate the difference between two means or two proportions.
Begin with the unit of observation and how the groups were formed. Different customers assigned to two website designs produce independent groups. Measuring the same store before and after a change produces paired observations. Recording whether an order arrives on time produces a binary outcome, summarized as a proportion.
Fix the subtraction order before interpreting results. This lesson always uses A − B. A positive difference means a larger mean or success proportion in group A. Whether larger is better depends on the outcome: a higher conversion rate may be desirable, while a longer service time may not be. Random assignment supports a causal treatment comparison; observational group differences can reflect confounding.
Use independent numeric observations, matched numeric pairs, or success counts. Every result compares A − B and tests a zero difference with a two-sided alternative.
The point is the observed difference; the horizontal line is its two-sided confidence interval. The vertical reference marks no difference. A confidence interval is an estimation procedure, not a probability distribution for a fixed parameter.
Reject the zero-difference null at α = 5%. Estimated standard error = 0.590; degrees of freedom = 13.292. There are 8 observations in A and 8 in B. Assess the size of the difference alongside its uncertainty and the study design.
The estimated A minus B difference is 1.750. Its 95% interval is 0.478 to 3.022. t = 2.966, p-value 0.0107. Estimated standard error = 0.590; degrees of freedom = 13.292. There are 8 observations in A and 8 in B. Reject the zero-difference null at α = 5%.
For independent samples, estimate μA − μB with x̄A − x̄B. Welch’s method uses SE = √(sA²/nA + sB²/nB) and does not require equal population variances. The null of equal means gives t = (x̄A − x̄B) / SE. Its t reference uses Welch–Satterthwaite degrees of freedom, which need not be an integer.
Let vA = sA²/nA and vB = sB²/nB. The degrees of freedom are (vA + vB)² / [vA²/(nA − 1) + vB²/(nB − 1)]. A two-sided interval is the observed difference ± t* SE. The approximation still depends on independent samples and suitable population shapes. Unequal variances are accommodated; severe outliers and dependence are not repaired by the formula.
For matched data, compute di = Ai − Bi for each pair. Then analyze these differences as one sample: estimate the population mean difference with d̄, use SE = sd / √n, and compare t = d̄ / SE with a t distribution having n − 1 degrees of freedom. Here n is the number of pairs, not the combined number of individual measurements.
The variation of the differences incorporates the within-pair relationship. Treating the two lists as independent discards that structure. Conversely, equal sample sizes alone do not justify pairing. Pair by store, person, shipment, or another meaningful design feature; keep the same order in both input lists. Examine the distribution of the differences when assessing the normality assumption.
For binary outcomes, estimate pA − pB with p̂A − p̂B. When testing H₀: pA = pB, pool the successes to estimate the common null proportion: p̂pool = (xA + xB)/(nA + nB). The null standard error is √[p̂pool(1 − p̂pool)(1/nA + 1/nB)], and the two-sided test uses the standard normal reference.
The displayed Wald confidence interval uses the unpooled standard error √[p̂A(1 − p̂A)/nA + p̂B(1 − p̂B)/nB]. The interval estimates an unknown difference and does not assume it is zero. This lab checks for at least 10 expected successes and failures per group under the null for the test, and at least 10 observed successes and failures per group for the interval. Sparse samples need other methods, such as exact tests or score intervals. Because the test and Wald interval use different standard errors, their conclusions can differ close to a threshold.
The default illustrative service-time samples have means 12 and 10.25. Welch’s standard error is about 0.590 and its degrees of freedom about 13.292. The difference is 1.75, with t ≈ 2.966, p ≈ 0.0107, and a 95% interval of approximately [0.478, 3.022]. Under the assumptions, these data provide evidence of a mean difference. The interval also shows the range of effect sizes consistent with this estimation procedure.
For an illustrative conversion experiment with 70 successes in 200 observations for A and 50 in 200 for B, the observed proportions are 0.35 and 0.25. The difference is 0.10, or 10 percentage points. The pooled proportion is 0.30, giving a test standard error of about 0.04583, z ≈ 2.182, and p ≈ 0.0291. The unpooled 95% interval is approximately [0.0107, 0.1893], or 1.07 to 18.93 percentage points. Ten percentage points is different from a 10% relative increase.
Try each question before revealing the solution.
Use n = 15 pairs and the standard deviation of the 15 within-store differences. The degrees of freedom are 14. Do not use n = 30 or substitute the standard deviation of all sales measurements.
The equality null specifies a shared population success probability, so pooled successes estimate that common value for the test. The interval allows a range of nonzero differences and estimates each group’s variability separately. The two calculations answer related but different questions.
The interval estimates A’s population mean to be 1 to 4 minutes lower than B’s, under the model assumptions. It excludes zero for the corresponding two-sided t procedure. Whether that reduction justifies a change depends on costs and goals; causation also depends on the study design.
Further reading