Module 07 · Hypothesis testing
Compare an observed effect with the variation expected under a null model. Learn what a p-value says—and what it cannot establish.
A fulfillment manager wants to know whether mean processing time differs from a 50-minute benchmark. Define μ as the mean for the target process and period. The null hypothesis is H₀: μ = 50; the two-sided alternative is Hₐ: μ ≠ 50. These statements concern the population, not the observed sample mean.
If only an increase would answer the research question, use Hₐ: μ > 50, with the null μ ≤ 50 evaluated at its boundary. For a decrease, use Hₐ: μ < 50 with null μ ≥ 50. Choose the direction before seeing the results. Switching to whichever one-sided alternative matches the data changes the error rate of the procedure.
Change the sample summary or the alternative. The shaded tail area is the p-value under the null t distribution.
The curve is a reference distribution for a test statistic under the null, not a distribution of the population mean. Tail probabilities include areas beyond the displayed range.
Do not reject H₀ at α = 5%. The data do not provide enough evidence against the null at this threshold; this does not establish equality. The observed difference of 2.000 still needs a practical interpretation.
The observed difference is 2.000. t = 1.000, with 24 degrees of freedom and p 0.3273. At α = 0.05, do not reject h₀.
The raw difference x̄ − μ₀ is easier to assess after comparing it with its estimated sampling variation. With unknown population standard deviation, the test statistic is t = (x̄ − μ₀) / (s / √n). Under independent sampling from a normal population and the boundary null, this statistic has a t distribution with n − 1 degrees of freedom.
A positive t puts the sample mean above the null value; a negative t puts it below. Its magnitude tells us how many estimated standard errors separate the two. Larger samples can make the same raw difference more statistically detectable. Summary values alone cannot reveal outliers, skewness, clustering, or a poor sampling design, so inspect the original data before relying on this calculation.
The p-value is the probability, under the null model and its assumptions, of a test statistic at least as extreme as the observed one in the direction specified by Hₐ. For a two-sided t test, both tails beyond |t| count. For Hₐ: μ > μ₀, use the right tail; for Hₐ: μ < μ₀, use the left tail.
Compare p with a significance level α selected in advance. Here we reject when p ≤ α. Otherwise, we do not reject. A p-value is not the probability that the null is true, and failure to reject does not prove equality. Changing α changes the decision threshold; it does not change the p-value for the same data and alternative.
For an illustrative independent sample, suppose n = 25, x̄ = 52 minutes, and s = 10 minutes. Testing μ₀ = 50 gives SE = 10 / √25 = 2 and t = (52 − 50) / 2 = 1, with 24 degrees of freedom. The two-sided p-value is approximately 0.3273, so we do not reject at α = 0.05.
This result says the sample does not supply sufficiently strong evidence of a difference under that rule. It does not say the mean is exactly 50. For a prespecified greater-than alternative, p ≈ 0.1636; for the opposite less-than alternative, p ≈ 0.8364. The direction matters. A companion 95% t interval is approximately [47.87, 56.13] minutes and includes 50.
A Type I error rejects a true null. A valid procedure controls that risk at α. A Type II error fails to reject when a particular alternative is true; its probability is β, and power is 1 − β. Power depends on the true effect, variation, sample size, and decision rule. It is not 1 minus the observed p-value.
Before collecting data, define the smallest difference that would affect a decision and plan a sample capable of detecting it. Afterward, report the estimate and confidence interval along with the p-value. A tiny difference may be statistically significant in a large sample yet operationally unimportant. Repeated unplanned tests or stopping whenever p becomes small can inflate false positives; a single-test α does not automatically protect the whole search.
Try each question before revealing the solution.
At α = 0.05, reject the null. At α = 0.01, do not reject. The p-value remains 0.03: it depends on the data and the test, not the chosen threshold. α should be set before examining the result.
SE = 8 / √16 = 2, so t = (104 − 100) / 2 = 2 with 15 degrees of freedom. A two-sided p-value is approximately 0.0639; a prespecified greater-than p-value is approximately 0.0320.
Examine the confidence interval, the cost of the change, the scale of operations, and the study design. Statistical significance alone does not establish a useful effect or a causal interpretation. Decide whether plausible effects are large enough to matter operationally.
Further reading