3.4.3 One Way ANOVA

One Way ANOVA Concept and Purpose One Way ANOVA (Analysis of Variance) is a statistical method used to test whether the means of three or more independent groups are equal. It answers the question: - “Is at least one group mean different from the others?” Typical examples: - Comparing average cycle time across three machines - Comparing defect rates across four suppliers - Comparing satisfaction scores for three different process designs The central idea: - Total variation in the data is split into: - Between-group variation: due to differences among group means - Within-group variation: due to random variation inside each group If between-group variation is large compared to within-group variation, the group means are unlikely to be equal. --- Basic Concepts and Notation Groups, Factor, and Response - Factor: The categorical input variable with different levels (e.g., Machine A/B/C) - Levels: The distinct categories of the factor (e.g., 3 machines → 3 levels) - Response: The continuous outcome measured (e.g., time, cost, length) Notation (standard): - (k): number of groups (factor levels) - (n_i): sample size in group (i) - (N = \sum{i=1}^{k} ni): total sample size - (\bar{X}_i): sample mean of group (i) - (\bar{X}): overall (grand) mean across all data points - (X_{ij}): (j)-th observation in group (i) --- Hypotheses in One Way ANOVA Null and Alternative Hypotheses The One Way ANOVA test compares multiple means simultaneously. - Null hypothesis (H_0): All group means are equal [ H0: \mu1 = \mu2 = \dots = \muk ] - Alternative hypothesis (H_1): At least one group mean is different [ H1: \text{Not all } \mui \text{ are equal} ] ANOVA itself does not indicate which means differ, only that at least one difference exists. --- ANOVA Model and Assumptions Statistical Model A simple One Way ANOVA model can be written as: [ X{ij} = \mu + \taui + \varepsilon_{ij} ] Where: - (\mu): overall mean - (\tau_i): effect of group (i) (deviation of group mean from overall mean) - (\varepsilon_{ij}): random error for observation (j) in group (i) The test focuses on: - (H0: \tau1 = \tau2 = \dots = \tauk = 0) Key Assumptions One Way ANOVA relies on these assumptions: - Independence: - Observations are independent within and across groups - Achieved by proper randomization and sampling - Normality: - The response in each group is approximately normally distributed - Especially important when sample sizes are small - Homogeneity of variances (homoscedasticity): - Population variances of all groups are equal [ \sigma1^2 = \sigma2^2 = \dots = \sigma_k^2 ] When these assumptions are reasonably met, the F-test in ANOVA is valid and reliable. --- Partitioning Variation: Sums of Squares Total, Between, and Within Variation One Way ANOVA decomposes the total variation in the data into between and within components. - Total Sum of Squares (SST): - Measures overall variation around the grand mean [ SST = \sum{i=1}^k \sum{j=1}^{ni} (X_{ij} - \bar{X})^2 ] - Between-Groups Sum of Squares (SSB or SSA): - Measures variation of group means around the grand mean [ SSB = \sum{i=1}^k ni (\bar{X}i - \bar{X})^2 ] - Within-Groups Sum of Squares (SSW or SSE): - Measures variation of individual observations around their own group mean [ SSW = \sum{i=1}^k \sum{j=1}^{ni} (X{ij} - \bar{X}i)^2 ] Relationship: [ SST = SSB + SS_W ] Degrees of Freedom - Between-groups degrees of freedom: - (df_B = k - 1) - Within-groups degrees of freedom: - (df_W = N - k) - Total degrees of freedom: - (df_T = N - 1) - Satisfies (dfT = dfB + df_W) --- Mean Squares and the F-statistic Mean Squares Mean squares are sums of squares divided by their degrees of freedom: - Between-groups Mean Square (MSB): [ MSB = \frac{SSB}{dfB} = \frac{SSB}{k - 1} ] - Within-groups Mean Square (MSW): [ MSW = \frac{SSW}{dfW} = \frac{SSW}{N - k} ] Interpretation: - (MS_W) estimates the common error variance (\sigma^2) - If (H0) is true, (MSB) also estimates (\sigma^2) - If (H0) is false, (MSB) becomes larger than (\sigma^2) F-statistic and Distribution The F-statistic compares between and within variation: [ F = \frac{MSB}{MSW} ] Under (H_0): - (F) follows an F-distribution with: - Numerator df = (k - 1) - Denominator df = (N - k) Decision rule at significance level (\alpha): - Compute (F) and corresponding p-value - Reject (H_0) if: - (p\text{-value} \leq \alpha), or - (F \geq F_{\alpha,,k-1,,N-k}) (critical value) --- The ANOVA Table Standard ANOVA Table Structure A typical One Way ANOVA table contains: - Source: Between, Within (Error), Total - SS: Sum of Squares - df: Degrees of Freedom - MS: Mean Squares - F: F-statistic - p-value: Significance level of the F-statistic Conceptually: - Source: Between - SS = (SS_B) - df = (k - 1) - MS = (MSB = SSB / (k - 1)) - F = (MSB / MSW) - p-value from F-distribution - Source: Within (Error) - SS = (SS_W) - df = (N - k) - MS = (MSW = SSW / (N - k)) - Source: Total - SS = (SS_T) - df = (N - 1) Interpret the table by focusing on: - F-statistic under Between - Its p-value --- Assumption Checking Normality Assessment To support the normality assumption: - Graphical checks: - Histogram or density plot of residuals - Normal probability plot (Q-Q plot) of residuals - What to look for: - Roughly symmetric residual distribution - Q-Q plot points near the straight line For moderate to large group sizes, ANOVA is robust to small deviations from normality. Homogeneity of Variances To check equal variances: - Graphical checks: - Boxplots of responses by group: similar spread across groups - Residuals vs. fitted values: roughly constant spread with no funnel shape - Rule-of-thumb: - Sample standard deviations across groups should be reasonably similar - Extremely different group spreads suggest unequal variances When variances differ substantially and sample sizes are unbalanced, the standard One Way ANOVA can give misleading results. Independence Considerations Independence usually depends on study design: - Use appropriate randomization when assigning units to groups - Avoid clustering that creates correlation (e.g., multiple measures from the same unit treated as independent) Violations of independence cannot be fixed by simple transformations; they must be handled at the design or modeling level. --- Effect Size and Practical Significance Measuring Effect Size ANOVA’s p-value shows statistical significance, but not the magnitude of the effect. Effect sizes provide a measure of practical importance. Common for One Way ANOVA: - Eta squared ((\eta^2)): [ \eta^2 = \frac{SSB}{SST} ] - Proportion of total variation explained by the factor - Omega squared ((\omega^2)) (less biased estimator): [ \omega^2 = \frac{SSB - (k - 1) MSW}{SST + MSW} ] - Estimates the proportion of variance explained in the population Interpretation: - Values near 0: very small factor effect - Larger values: factor explains a larger share of variance Effect size supports decisions about whether statistically significant differences are meaningful in practice. --- Multiple Comparisons and Post-hoc Tests Why Post-hoc Tests Are Needed ANOVA tells whether at least one mean differs, but not which means differ. If the overall F-test is significant: - Follow-up pairwise comparisons are used to identify specific group differences - These are often called post-hoc tests In the context of One Way ANOVA, typical goals: - Compare each pair of group means - Control the overall Type I error rate when doing multiple tests Common Multiple Comparison Approaches While many methods exist, the conceptual understanding needed includes: - Multiple testing inflates the chance of false positives - Post-hoc procedures adjust to keep the overall error rate controlled Key ideas: - Family-wise error rate (FWER): - Probability of making at least one Type I error across all comparisons - ANOVA-based procedures aim to keep this at or below a chosen (\alpha) - Balanced vs. unbalanced designs: - Some methods assume equal sample sizes; modern software usually handles both In practice, once ANOVA shows significance: - Use an appropriate multiple comparison method (e.g., adjusted pairwise tests) to interpret which specific group means are different. --- Design and Data Considerations Balanced and Unbalanced Designs - Balanced design: - All groups have the same sample size ((n_i) equal) - Properties: - Simpler calculations - Greater robustness to assumption violations - Unbalanced design: - Groups have different sample sizes - Properties: - ANOVA is still valid if assumptions hold - Sensitivity to unequal variances increases - Interpretation can be more complex When planning data collection, balanced designs are preferred where feasible. Sample Size Considerations For useful One Way ANOVA results: - Each group should have enough observations to: - Estimate variance reliably - Check normality reasonably - Very small group sizes (e.g., (n_i < 5)) limit: - Power to detect differences - Reliability of assumption checks Power increases with: - Larger group sizes - Larger differences between group means - Smaller within-group variation - Higher significance level (larger (\alpha), though this increases Type I error) --- Interpretation and Common Pitfalls Interpreting Significant and Non-significant Results When the ANOVA F-test is significant (small p-value): - Evidence that at least one group mean differs from the others - Effect size should be examined to gauge practical importance - Post-hoc comparisons help locate specific differences When the ANOVA F-test is not significant: - Data are consistent with equal means, given sample size and variability - Does not prove the means are exactly equal, only that there is insufficient evidence of difference - Low power (small data, high variability) may hide real differences Typical Misinterpretations Avoid: - Concluding all means are unequal from ANOVA alone - Ignoring assumption violations when they are severe - Interpreting p-values as the probability that the null hypothesis is true - Equating statistical significance with practical importance Good practice: - Check assumptions - Report F, df, p-value, and effect size - Interpret findings in the context of the process or system studied --- Summary One Way ANOVA is a method to test whether three or more independent group means are equal, using a single categorical factor and a continuous response. It decomposes total variability into between-group and within-group components, compares their mean squares through an F-statistic, and evaluates significance via the F-distribution. Key elements include: - Clear formulation of hypotheses about multiple means - Understanding sums of squares, degrees of freedom, mean squares, and the F-ratio - Proper use and interpretation of the ANOVA table - Verification of assumptions: independence, normality, equal variances - Distinguishing statistical from practical significance through effect size - Using post-hoc multiple comparisons only after a significant overall F-test Mastering these concepts enables correct application, calculation, and interpretation of One Way ANOVA in situations where multiple group means must be compared rigorously.