4.2.4 Residuals Analysis

Residuals Analysis Residuals analysis is the primary tool for checking whether a statistical model, especially regression or designed experiment models, is appropriate for the data. It focuses on the differences between the observed data and the values predicted by the model. Mastery of residuals analysis is essential for validating conclusions based on regression and ANOVA in process improvement work. --- Foundations of Residuals What Is a Residual? A residual is the difference between an observed response and the value predicted by the model: - Residual = observed value − predicted value For an observation (i): - ( y_i ) = observed response - ( \hat{y}_i ) = predicted response from the model - ( ei = yi - \hat{y}_i ) = residual Residuals measure the unexplained part of the response after accounting for the model’s predictors (factors). Role of Residuals in Model Validation Residuals analysis is used to: - Check model assumptions - Detect model misspecification - Identify outliers and influential points - Assess goodness of fit beyond summary statistics (R², p-values) If model assumptions are seriously violated, statistical tests and confidence intervals can become misleading. --- Assumptions Checked by Residuals Most linear regression and ANOVA models rely on four main assumptions about residuals: Linearity The relationship between predictors and response should be linear in the model form. - Residuals should be randomly scattered around zero for all values of predictors. - Systematic patterns (curves, waves) suggest missing terms such as: - Quadratic effects (e.g., (x^2)) - Interaction effects (e.g., (x1 x2)) - Transformations (e.g., log, square root) Independence Residuals from different observations should be independent. - No pattern over time or sequence. - No clustering by batch, operator, machine, or location if not modeled. Violations often occur in time-ordered data (autocorrelation) or nested data (e.g., measurements inside units). Constant Variance (Homoscedasticity) Residuals should have constant spread (variance) across the range of predicted values or predictors. - Residuals should show similar vertical spread for low and high predictions. - Funnel shapes (spread increasing or decreasing) indicate non-constant variance. Heteroscedasticity affects: - Accuracy of standard errors - Reliability of hypothesis tests and confidence intervals Normality Residuals should be approximately normally distributed when: - Making inferences (p-values, confidence/prediction intervals) - Using small to moderate sample sizes Severe non-normality can impact: - Validity of t-tests and F-tests - Precision of prediction intervals Normality is less critical for large sample sizes but should still be checked, especially for extreme outliers or skewness. --- Types of Residuals Raw (Ordinary) Residuals Raw residuals are the direct differences: - ( ei = yi - \hat{y}_i ) They are the starting point for most diagnostics but do not account for differences in leverage or variance across observations. Standardized Residuals Standardized residuals scale raw residuals by an estimate of their standard deviation: - Standardized residual = residual ÷ estimated standard deviation of residual They allow comparison across observations on a common scale. Use standardized residuals to: - Identify relatively large deviations - Compare observations within a single model Studentized Residuals Studentized residuals further account for the influence of each data point: - Internally studentized: use error variance estimated with all data - Externally studentized: use error variance estimated after removing the point being evaluated Externally studentized residuals are more sensitive for detecting outliers. Rules of thumb: - Values larger than about ±2 (or ±3) merit investigation. - Extreme studentized residuals can indicate outlying or incorrectly recorded data. Residuals in Designed Experiments In factorial and response surface designs, residuals still represent observed minus fitted values, but special attention is given to: - Interactions and curvature (which can show up as patterns in residuals) - Replicate runs (to estimate pure error and compare against model error using residuals) --- Essential Residual Plots Residual plots are visual tools to detect assumption violations and model problems. Residuals vs Fitted Values Plot residuals on the vertical axis and fitted (predicted) values on the horizontal axis. What to look for: - Random band around zero: good sign - Curved pattern: suggests nonlinearity or missing terms - Funnel shape (widening or narrowing spread): non-constant variance - Clusters: unmodeled groups or missing factors Use this plot to check: - Linearity - Constant variance - General model fit Residuals vs Predictor (X) Variables Plot residuals against each predictor (or important process variable): - Detect nonlinear relationships not captured by the model. - Identify regions where model performs poorly. - Check for missing transformations (e.g., log, square root) or polynomial terms. Patterns that suggest refinement: - Curved shapes: add polynomial terms or transformations. - Different spreads across X levels: consider variance-stabilizing transformation or weighted regression. Residuals vs Time or Run Order For data collected over time or in experimental run order: - Plot residuals versus time or run order index. Use to detect: - Autocorrelation (oscillating or trending residuals) - Process shifts or drifts (step changes in mean of residuals) - Warm-up or learning effects (early residuals differ from later ones) If significant time-related patterns appear, consider: - Including time or batch as a factor. - Using time-series methods instead of standard regression if dependence is strong. Normal Probability Plot of Residuals A normal probability plot compares the ordered residuals to the expected order statistics from a normal distribution. Indicators: - Approximate straight line: residuals are roughly normal. - Curved S-shape: skewness. - Heavy tails (points bending away at both ends): more extreme values than normal. - Isolated extreme points: possible outliers. This plot is often preferred over raw histograms because it is more sensitive to deviations from normality. --- Diagnostic Tests Related to Residuals Residual plots are primary tools, but some numeric tests are commonly associated with residual analysis. Tests for Normality Applied to residuals: - Shapiro–Wilk - Anderson–Darling - Kolmogorov–Smirnov Usage: - Support visual assessment from the normal probability plot. - Indicate whether non-normality is statistically significant. Interpretation: - A significant p-value plus strong visual deviation suggests serious non-normality. - For moderate non-normality with large sample sizes, model results may still be practically acceptable. Tests for Independence (Autocorrelation) Applied mainly when data have a natural order (time, sequence): - Durbin–Watson statistic: tests for first-order autocorrelation in residuals. Signs of autocorrelation: - Residuals that alternate slowly above and below zero. - Clusters of positive or negative residuals. If autocorrelation is present: - Standard regression and ANOVA inferences become unreliable. - Consider adding lagged terms, using time-series models, or rethinking sampling/design. Leverage and Influence (Connected to Residuals) Though not residuals themselves, leverage and influence measures rely on residual behavior. - Leverage (hᵢᵢ): how far a predictor combination is from the overall center of X. - Influence: combined effect of leverage and large residuals on fitted model. Common measures: - Cook’s distance: large values indicate influential points. - DFFITS, DFBETAS: show how much a point affects fitted values or coefficients. Use: - Identify points that unduly control the regression line or ANOVA results. - Decide whether such points represent real process behavior or data problems. --- Common Patterns and Their Interpretation Random, Structureless Residuals Characteristics: - Centered around zero - No obvious pattern vs fitted values, predictors, or time - Constant spread Interpretation: - Model and assumptions are reasonably adequate. - Proceed with interpretation of coefficients, p-values, and intervals. Curvature in Residuals Example pattern: - Residuals form a U shape or inverted U when plotted vs fitted values or a predictor. Interpretation: - Relationship is not fully captured by a straight line. - Consider: - Adding polynomial terms (e.g., (x^2), (x^3)) - Applying transformations (e.g., log, square root) - Using response surface methods if factors are continuous Non-Constant Variance Example patterns: - Funnel shape: residual spread increases with fitted values. - Wedge: residuals are tight in the middle, spread at extremes. Interpretation: - Variance of errors depends on level of response or predictors. Possible remedies: - Transform the response (e.g., log, square root, Box–Cox). - Use weighted least squares (assigning weights inversely proportional to variance). - Model separate variances for distinct groups, if appropriate. Outliers and Extreme Residuals Signs: - Individual residuals far from zero. - Large standardized or studentized residuals. - Points deviating strongly on normal probability plot. Investigation steps: - Check for data recording or measurement errors. - Assess whether point represents a special cause event. - Evaluate impact on model: - Refit without the point to see if results change substantially. - Decide whether to: - Keep it (if it reflects true process behavior), - Model it (add a factor for that condition), - Or remove it with justification (e.g., clear measurement error). Non-Normal Residuals Signs: - Pronounced S-shape on normal probability plot. - Skewed or heavy-tailed residuals. Interpretation: - Non-normal error distribution. - May affect statistical inferences, especially with small samples or heavy tails. Response options: - Transform the response (e.g., log for right-skewed data). - Use robust methods or nonparametric approaches when appropriate. - In large samples, minor deviations often have limited practical impact. Correlated Residuals Over Time Signs: - Runs of consecutive positive or negative residuals. - Residuals vs time plot shows trend or oscillation. Interpretation: - Process is not stable over time, or model omits key dynamic factors. Possible actions: - Add time, batch, or other relevant covariates. - Arrange data collection to reduce temporal dependence (e.g., randomization). - For strong autocorrelation, apply time-series models instead of simple regression. --- Residuals in Designed Experiments and ANOVA Residuals are central to validating models in designed experiments (DOE) and ANOVA. Checking Model Adequacy in DOEs For factorial, fractional factorial, and response surface designs: - Plot residuals vs: - Fitted values - Each factor and key interaction - Run order Signs of inadequacy: - Patterned residuals vs factors: missing interactions, curvature, or transformations. - Large residuals for specific combinations: potential unmodeled effects or experimental errors. When curvature is suspected: - Add center points (if not already present) and compare: - Mean at center vs mean of factorial points - Inspect residuals for lack of fit that correlates with mid-range levels. Residuals and Lack-of-Fit Tests In replicated experiments: - Total residual variation can be split into: - Pure error: variability among replicates at the same factor setting. - Lack of fit: additional variation due to model not capturing the true relationship. A significant lack-of-fit test: - Indicates the model form is inadequate relative to the replication-based pure error. - Suggests need to: - Add terms (interactions, curvature) - Use transformations - Consider alternative model structures Block Effects and Residuals When blocks (e.g., shifts, days, machines) are included in DOE: - Residual plots should show: - No remaining systematic differences between blocks. - Similar residual distributions across blocks. If residuals differ widely by block: - Block factor may be incomplete or mis-specified. - Additional blocking or random effects may be needed. --- Practical Residual Analysis Workflow Step 1: Fit the Model - Choose an initial regression or ANOVA model. - Include terms based on process knowledge and experimental design. Step 2: Examine Core Residual Plots Focus on four primary views: - Residuals vs fitted values - Residuals vs key predictors - Residuals vs time or run order (if applicable) - Normal probability plot of residuals Ask: - Are residuals centered at zero? - Is spread roughly constant? - Are there curved patterns or clusters? - Are there outliers or non-normal behavior? Step 3: Compute Diagnostic Statistics Check: - Standardized or studentized residuals for outliers. - Leverage and influence (Cook’s distance, etc.) for influential points. - Normality tests for residuals (if needed). - Durbin–Watson or other autocorrelation checks for time-based data. Step 4: Refine the Model as Needed Depending on residual patterns, consider: - Adding or removing predictors. - Adding interaction or polynomial terms. - Applying transformations to the response or predictors. - Addressing non-constant variance or correlation structures. After each refinement: - Refit the model. - Re-examine residuals to confirm improvement and avoid overfitting. Step 5: Document and Justify Decisions For each major modeling decision: - State observed residual issues (e.g., curvature, outliers, non-constant variance). - Describe changes applied to the model. - Show how residual diagnostics improved. This ensures transparency and supports confidence in final conclusions. --- Summary Residuals analysis is the central method for evaluating whether regression and ANOVA models adequately represent process behavior. Residuals are the differences between observed and fitted values, and their patterns reveal whether key assumptions hold: - Linearity - Independence - Constant variance - Normality Key tools include: - Residual plots vs fitted values, predictors, and time - Normal probability plots of residuals - Standardized and studentized residuals - Leverage and influence measures - Lack-of-fit and normality tests, and checks for autocorrelation By systematically interpreting residual patterns and refining models accordingly, the resulting analyses provide reliable estimates, valid hypothesis tests, and sound predictions for process improvement and control.