4.4.2 Linear & Quadratic Mathematical Models

Linear & Quadratic Mathematical Models Introduction Linear and quadratic models are core tools for describing, analyzing, and improving relationships between variables. They are widely used for: - Modeling cause–effect relationships - Predicting process performance - Quantifying the impact of input changes on outputs - Supporting optimization and decision-making This article builds from basic form and interpretation through estimation, diagnostics, and practical use in improvement projects, focusing only on linear and quadratic models and closely related concepts. --- Foundations of Mathematical Modeling Variables, Parameters, and Error A mathematical model expresses how a response (output) depends on one or more predictors (inputs): - Response variable (Y): outcome of interest (e.g., cycle time, defect rate) - Predictor variable (X): factor thought to influence Y (e.g., temperature, speed) - Parameters: unknown constants to be estimated from data (e.g., slopes, intercept) - Error term (ε): random deviation of observations from the model A general model form is: - Deterministic part: describes the systematic relationship - Example: ( f(X) = \beta0 + \beta1 X + \beta_2 X^2 ) - Stochastic part: random variation not explained by the predictors - Complete model: ( Y = f(X) + \varepsilon ) Functional vs Statistical Models - Functional model: exact relationship assumed, no random error - Example: geometric formulas, physical laws - Statistical model: includes random variation - Example: regression models with error term ε Linear and quadratic regression models are statistical models: parameters are estimated from data, and predictions include uncertainty. --- Linear Models Form of a Simple Linear Model A simple linear model with one predictor X has the form: [ Y = \beta0 + \beta1 X + \varepsilon ] - (\beta_0) (intercept): predicted value of Y when X = 0 - (\beta_1) (slope): change in mean Y for a 1-unit increase in X - (\varepsilon): random error term, typically assumed to have: - Mean 0 - Constant variance - Independence - Often approximated as normally distributed for inference For multiple predictors (X1, X2, \dots, X_k): [ Y = \beta0 + \beta1 X1 + \dots + \betak X_k + \varepsilon ] Estimation by Least Squares Parameters are typically estimated by Ordinary Least Squares (OLS): - Objective: choose (\hat{\beta}0, \hat{\beta}1, \dots) to minimize the sum of squared residuals: [ \text{SSE} = \sum{i=1}^{n} (yi - \hat{y}_i)^2 ] - Residual for point i: [ ei = yi - \hat{y}_i ] Key ideas: - The fitted line passes through the “center” of the data in an optimal way - Residuals measure the unexplained portion of each observation - Sum of residuals is zero for a model with intercept Interpretation of Linear Coefficients For the simple linear model: [ \hat{Y} = \hat{\beta}0 + \hat{\beta}1 X ] - Intercept (\hat{\beta}_0): - Predicted Y when X = 0 - Often extrapolative and may have limited physical meaning - Slope (\hat{\beta}_1): - Estimated change in the mean of Y for a one-unit increase in X - Sign indicates direction: - Positive: Y increases as X increases - Negative: Y decreases as X increases - Magnitude indicates strength of effect per unit change In multiple linear regression, each slope is interpreted holding other predictors constant. Goodness of Fit: R² and Adjusted R² To assess how well a linear model fits: - Total Sum of Squares (SST): [ \text{SST} = \sum (y_i - \bar{y})^2 ] - Regression Sum of Squares (SSR): [ \text{SSR} = \sum (\hat{y}_i - \bar{y})^2 ] - Error Sum of Squares (SSE): [ \text{SSE} = \sum (yi - \hat{y}i)^2 ] - Decomposition: [ \text{SST} = \text{SSR} + \text{SSE} ] Coefficient of determination (R²): [ R^2 = \frac{\text{SSR}}{\text{SST}} = 1 - \frac{\text{SSE}}{\text{SST}} ] - Proportion of variation in Y explained by the model - Between 0 and 1; higher values indicate better explanatory power Adjusted R²: - Adjusts R² for the number of predictors and sample size - Prevents artificial inflation of R² by adding non-informative variables - Used mainly in multiple regression to compare models with different numbers of predictors Linear Model Assumptions and Diagnostics Core assumptions: - Linearity: mean of Y is a linear function of predictors - Independence: errors are independent - Homoscedasticity: errors have constant variance for all fitted values - Normality (for inference): errors are approximately normally distributed Common diagnostics: - Residual vs fitted plot: - Should show random scatter around zero - Systematic patterns (curvature, funnel shapes) suggest: - Nonlinearity - Non-constant variance - Normal Q–Q plot of residuals: - Points should follow a straight line - Strong deviations suggest non-normality - Outlier and leverage analyses: - Identify unusual observations or influential points When residual plots show curvature, a linear model may be inadequate; a quadratic model can often provide a better fit. --- Quadratic Models Form of a Quadratic Model A quadratic model in one predictor X has the form: [ Y = \beta0 + \beta1 X + \beta_2 X^2 + \varepsilon ] - (\beta_0): intercept - (\beta_1): linear effect of X - (\beta_2): curvature term; determines the shape of the parabola Properties: - If (\beta_2 > 0): parabola opens upward (U-shaped) - If (\beta_2 < 0): parabola opens downward (inverted U) - Allows modeling of: - Diminishing returns - Optimum points (minima or maxima) - Curved response patterns often found in real processes Relationship to Linear Regression Even though the model includes (X^2), it is still a linear model in the parameters: [ Y = \beta0 + \beta1 X + \beta_2 X^2 + \varepsilon ] Define new predictors: - (Z_1 = X) - (Z_2 = X^2) Then: [ Y = \beta0 + \beta1 Z1 + \beta2 Z_2 + \varepsilon ] This means: - OLS methods for linear regression apply directly - Software handles quadratic models as a special case of multiple linear regression with transformed predictors Vertex and Optimum Point The quadratic model has a vertex where the predicted response is minimum (if (\beta2 > 0)) or maximum (if (\beta2 < 0)). For: [ \hat{Y} = \hat{\beta}0 + \hat{\beta}1 X + \hat{\beta}_2 X^2 ] - X-coordinate of the vertex: [ X^* = -\frac{\hat{\beta}1}{2\hat{\beta}2} ] - Predicted optimum response: [ \hat{Y}^ = \hat{\beta}0 + \hat{\beta}1 X^ + \hat{\beta}_2 {X^*}^2 ] Interpretation: - If (\hat{\beta}_2 < 0): vertex is a maximum; (X^*) gives the peak predicted response - If (\hat{\beta}_2 > 0): vertex is a minimum; (X^*) gives the lowest predicted response This is central for identifying operating conditions that optimize performance. Interpretation of Quadratic Coefficients For: [ \hat{Y} = \hat{\beta}0 + \hat{\beta}1 X + \hat{\beta}_2 X^2 ] - (\hat{\beta}_0): - Predicted Y at X = 0 (may be extrapolative) - (\hat{\beta}_1): - Local linear effect near X = 0 - Not constant across all X when (\hat{\beta}_2 \ne 0) - (\hat{\beta}_2): - Curvature: magnitude indicates degree of bending - Sign determines whether the response curves up or down The instantaneous rate of change of Y with respect to X is the derivative: [ \frac{d\hat{Y}}{dX} = \hat{\beta}1 + 2\hat{\beta}2 X ] - Shows that the effect of X on Y changes with the level of X - Rate of change is zero at the vertex (X^* = -\hat{\beta}1 / (2\hat{\beta}2)) --- Building Linear and Quadratic Models from Data Data Requirements and Preparation Key considerations: - Measurement quality: - Reliable, valid measurements of Y and X - Stable measurement system - Range of X: - Must include enough spread to detect linear or quadratic patterns - For quadratic effects, include values on both sides of the potential optimum if feasible - Sample size: - Should be sufficient to estimate parameters and assess model adequacy - More parameters (e.g., adding (X^2)) require more data Data preparation: - Check for: - Missing values - Obvious errors or outliers - Understand units and scaling: - Strongly different scales across predictors can affect computations and diagnostics - Centering or standardizing X can sometimes improve interpretability and reduce multicollinearity with (X^2) Model Selection: Linear vs Quadratic Choosing between a linear and quadratic model involves: - Subject-matter knowledge: - Physical or process understanding may suggest curvature or optimum - Graphical examination: - Scatter plot of Y vs X - Overlaid linear and quadratic fits for comparison - Statistical comparison (for nested models): - Linear model: (Y = \beta0 + \beta1 X + \varepsilon) - Quadratic model: (Y = \beta0 + \beta1 X + \beta_2 X^2 + \varepsilon) - Test significance of (\beta_2): - If (\beta_2) is statistically nonzero, curvature is supported - Model parsimony: - Prefer the simpler model if it fits adequately - Use quadratic when it clearly improves fit and interpretation Parameter Estimation and Inference OLS estimation provides: - Point estimates: - (\hat{\beta}0, \hat{\beta}1, \hat{\beta}_2, \dots) - Standard errors: - Measure of estimation uncertainty - t-tests for individual coefficients: - Hypotheses: - (H0: \betaj = 0) - (Ha: \betaj \ne 0) - Large |t| and small p-values suggest the coefficient is significantly different from 0 - Confidence intervals for parameters: - Ranges of plausible values for each coefficient For the overall model: - ANOVA F-test: - Tests whether the model explains a significant portion of the variation in Y - Compares model with predictors against a null model with only an intercept --- Model Validation and Diagnostics Residual Analysis for Linear and Quadratic Fits Residuals (ei = yi - \hat{y}_i) are central to assessing model adequacy. Key plots: - Residuals vs fitted values: - Look for: - Random scatter around zero: acceptable - Curved patterns: may indicate that a linear model is insufficient - Fanning out or narrowing: signals non-constant variance - Residuals vs X: - Useful specifically for checking overlooked curvature - If linear model shows systematic curvature, a quadratic model may be appropriate - Normal Q–Q plot of residuals: - Evaluate approximate normality - Important for valid p-values and confidence intervals Comparing Linear and Quadratic Residuals When both models are fit: - Compare: - Residual patterns - SSE values - R² and adjusted R² - Assess: - Whether the quadratic significantly reduces SSE compared to linear - Whether residual plots for the quadratic show improved randomness and constant variance If adding the quadratic term: - Substantially improves fit - Removes systematic patterns in residuals - Yields significant (\beta_2) then the quadratic model is generally preferred. Overfitting and Appropriate Complexity Model complexity should match data support and purpose: - Overfitting: - Model captures random noise instead of the underlying relationship - Symptoms: - Very high R² with poor predictive performance on new data - Unstable coefficient estimates and wide confidence intervals - Balanced approach: - Use linear model when it adequately describes the data - Move to quadratic when there is clear evidence of curvature - Avoid unnecessarily high-degree polynomials without strong justification --- Using Linear and Quadratic Models for Prediction Point Predictions Given fitted model (\hat{Y} = \hat{\beta}0 + \hat{\beta}1 X) or (\hat{Y} = \hat{\beta}0 + \hat{\beta}1 X + \hat{\beta}_2 X^2): - For a specified X value: - Compute the fitted value (\hat{Y}) by substitution - In multiple regression: - Substitute the set of predictor values (X1, X2, \dots) These point predictions represent the estimated mean response at that X (and other predictors). Prediction Intervals and Uncertainty Predictions have uncertainty from: - Estimation error in the coefficients - Random error in new observations Key intervals: - Confidence interval for mean response at X: - Reflects uncertainty in the expected value of Y at that X - Narrower than prediction intervals - Prediction interval for an individual Y at X: - Includes both: - Uncertainty in the model parameters - Random variation in a specific observation - Wider than the confidence interval for the mean As X moves far from the center of the observed data, both intervals typically widen, reflecting increased uncertainty in extrapolation. Extrapolation Risks Using the model outside the data range can be misleading: - For linear models: - Assuming the linear trend continues indefinitely may be unrealistic - For quadratic models: - Predicted values can grow very large (positive or negative) outside the observed X range - Apparent optimum may be outside the practical or physically meaningful range To manage risk: - Restrict interpretation primarily to the range of X used for fitting - Use domain knowledge to judge plausibility of extrapolated predictions - When optimal conditions are near the edge of the studied range, consider collecting more data beyond that edge if feasible --- Practical Interpretation and Application Understanding Effect Size and Sensitivity For a linear model: [ \hat{Y} = \hat{\beta}0 + \hat{\beta}1 X ] - Sensitivity of Y to X: - (\hat{\beta}_1) units of change in Y per one unit of X - Improvement thinking: - How much must X change to achieve a desired change in Y? - Is this change operationally feasible? For a quadratic model: [ \hat{Y} = \hat{\beta}0 + \hat{\beta}1 X + \hat{\beta}_2 X^2 ] - Sensitivity depends on X: [ \frac{d\hat{Y}}{dX} = \hat{\beta}1 + 2\hat{\beta}2 X ] - Interpretation: - Near the vertex, the rate of change may be small (plateau) - Far from the vertex, the rate of change may be larger (more sensitive) Identifying Optimum Operating Conditions Quadratic models are especially useful for locating settings that optimize performance. Steps: - Fit quadratic model: ( \hat{Y} = \hat{\beta}0 + \hat{\beta}1 X + \hat{\beta}_2 X^2 ) - Compute: [ X^* = -\frac{\hat{\beta}1}{2\hat{\beta}2} ] - Evaluate: - Whether (\hat{\beta}2 < 0) (for a maximum) or (\hat{\beta}2 > 0) (for a minimum), depending on the objective - If (X^*) lies within the studied and practically feasible range - Calculate predicted optimum response: [ \hat{Y}^ = \hat{\beta}0 + \hat{\beta}1 X^ + \hat{\beta}_2 {X^*}^2 ] If the optimum is near the boundary of the tested region: - Consider extending the range of X and collecting additional data - Confirm that the observed optimum is not an artifact of extrapolation Multiple Predictors and Second-Order Models When more than one predictor is involved, a second-order model can include: [ Y = \beta0 + \sum{i} \betai Xi + \sum{i} \beta{ii} X_i^2 + \sum{i