4.2.3 Confidence & Prediction Intervals

Confidence & Prediction Intervals Introduction Confidence intervals and prediction intervals quantify uncertainty in data-based decisions. They are central to estimating process parameters, comparing groups, and predicting future observations. This article explains: - What confidence and prediction intervals are - How they relate to sampling distributions - Key formulas and assumptions - How to interpret and use them in practice - Common pitfalls and practical tips All content is focused on the knowledge required to understand and correctly apply confidence and prediction intervals in data analysis and improvement projects. --- Foundations: Sampling and Variation Sampling Distributions and Standard Error A confidence or prediction interval is built on a sampling distribution. - Population: The full set of all possible observations of interest. - Sample: A subset of the population actually measured. - Statistic: A number computed from the sample (for example, sample mean). When samples are repeatedly drawn from the same population: - The sample statistic varies from sample to sample. - The distribution of that statistic is the sampling distribution. For the sample mean: - If (X1, X2, ..., X_n) are independent observations with mean (\mu) and variance (\sigma^2), - The sample mean is (\bar{X} = \frac{1}{n}\sum X_i). - Its standard deviation is the standard error of the mean: [ SE(\bar{X}) = \frac{\sigma}{\sqrt{n}} ] In practice, (\sigma) is usually unknown and is estimated by the sample standard deviation (s): [ SE(\bar{X}) \approx \frac{s}{\sqrt{n}} ] Central Limit Theorem (CLT) The central limit theorem underpins most interval formulas: - For sufficiently large sample size (n), the sampling distribution of (\bar{X}) is approximately normal, even when the population is not normal. - For small (n), normality of the underlying population (or near-normal residuals) is more important. This allows use of normal or t distributions for interval calculations. --- Confidence Intervals: Concept and Interpretation What a Confidence Interval Is A confidence interval (CI) provides a range of plausible values for a population parameter (for example, mean or proportion), based on sample data. - Parameter: Fixed but unknown (for example, true mean (\mu)). - Interval: Random because it depends on random sample data. For a 95% confidence interval: - If one could repeat the sampling procedure many times and compute a 95% CI from each sample: - About 95% of those intervals would contain the true parameter. - About 5% would not. Key point: - The confidence level (for example, 95%) describes the long-run performance of the interval construction method, not the probability that a specific computed interval contains the parameter (the parameter is not random). Confidence Level and Alpha The confidence level is related to alpha ((\alpha)): - (\text{Confidence level} = 1 - \alpha) - 95% CI → (\alpha = 0.05) - 99% CI → (\alpha = 0.01) Alpha represents the probability that the constructed interval does not contain the true parameter in the long run. Trade-offs: - Higher confidence level → - Wider intervals - More certainty that the interval includes the parameter - Lower confidence level → - Narrower intervals - Less certainty --- Confidence Intervals for a Mean One-Sample Mean, Sigma Unknown (Typical Case) Most real situations do not know the population standard deviation. Use the t distribution. Given: - Sample size (n) - Sample mean (\bar{x}) - Sample standard deviation (s) - Desired confidence level, with critical value (t_{\alpha/2, , n-1}) from the t distribution with (n - 1) degrees of freedom The 2-sided confidence interval for the population mean (\mu) is: [ \bar{x} \pm t_{\alpha/2, , n-1} \cdot \frac{s}{\sqrt{n}} ] Key behaviors: - Larger (n) → smaller (SE(\bar{X})) → narrower CI. - Larger variability (s) → wider CI. - Higher confidence level (for example, 99% instead of 95%) → larger critical value → wider CI. One-Sample Mean, Sigma Known (Theoretical Case) If population standard deviation (\sigma) is known: [ \bar{x} \pm z_{\alpha/2} \cdot \frac{\sigma}{\sqrt{n}} ] - (z_{\alpha/2}) is the critical value from the standard normal distribution. - In practice, this situation is rare; t-based intervals are standard. Assumptions for Mean Confidence Intervals Main assumptions: - Data are independent observations. - For small samples: - Population (or residuals) is approximately normal. - For large samples: - The CLT makes the sampling distribution of (\bar{X}) approximately normal, even if the population is not. Practical checks: - Plot the data (histogram, boxplot, normal probability plot). - Look for strong skewness, outliers, or multimodality that may distort the CI. --- Confidence Intervals for a Proportion One-Sample Proportion Interval Let: - (n) = sample size - (x) = number of “successes” (for example, defects, passes) - (\hat{p} = \frac{x}{n}) = sample proportion For large (n), the approximate 2-sided confidence interval for the true proportion (p) is: [ \hat{p} \pm z_{\alpha/2} \cdot \sqrt{\frac{\hat{p}(1-\hat{p})}{n}} ] Assumptions: - Binary outcome (success/failure). - Independent trials. - Sample size large enough such that: - (n\hat{p} \geq 5) and (n(1-\hat{p}) \geq 5) (common rule of thumb). For small samples or extreme proportions (near 0 or 1), exact methods (for example, Clopper-Pearson) or adjusted formulas are preferable, but the conceptual interpretation remains the same. --- Confidence Intervals for Difference in Means Two-Sample Means (Independent Samples) Often the goal is to compare two processes or groups. Let: - Group 1: (n1, \bar{x}1, s_1) - Group 2: (n2, \bar{x}2, s_2) - Parameter of interest: (\mu1 - \mu2) The point estimate is: [ \widehat{\mu1 - \mu2} = \bar{x}1 - \bar{x}2 ] The standard error depends on whether equal variances are assumed. - Unequal variances (Welch’s t), most general: [ SE(\bar{x}1 - \bar{x}2) = \sqrt{\frac{s1^2}{n1} + \frac{s2^2}{n2}} ] The confidence interval is: [ (\bar{x}1 - \bar{x}2) \pm t{\alpha/2, , df} \cdot SE(\bar{x}1 - \bar{x}_2) ] Degrees of freedom (df) are approximated (Welch-Satterthwaite formula), usually given by software. Assumptions: - Samples are independent. - Within each group, data are independent and approximately normal (especially important for small samples). - For large samples, normality is less strict due to CLT. Interpretation: - If a 95% CI for (\mu1 - \mu2) does not include 0, this aligns with a significant difference in means at (\alpha = 0.05). --- Confidence Intervals and Hypothesis Tests Relationship Between CIs and Tests Confidence intervals and hypothesis tests are mathematically linked. - A 2-sided ((1 - \alpha)) confidence interval corresponds to a 2-sided hypothesis test at significance level (\alpha). - If a hypothesized parameter value (for example, (\mu0)) lies outside the ((1 - \alpha)) CI, the null hypothesis (H0: \mu = \mu_0) would be rejected at level (\alpha). Advantages of using CIs: - Provide a range of plausible values, not just a reject/accept decision. - Convey both statistical significance and practical magnitude. --- Confidence Intervals in Regression Confidence Intervals for Regression Coefficients In simple or multiple linear regression, parameters include: - Intercept: (\beta_0) - Slope(s): (\beta1, \beta2, ...) Each estimated coefficient (\hat{\beta}_j) has: - Standard error (SE(\hat{\beta}_j)) - A t-based confidence interval: [ \hat{\beta}j \pm t{\alpha/2, , df} \cdot SE(\hat{\beta}_j) ] Interpretation: - The CI for a slope expresses the range of plausible values for the true effect of the predictor on the response, holding other predictors constant. - If the CI for a slope includes 0, the predictor may not be useful in explaining variation in the response (at that confidence level). Assumptions for linear regression CIs: - Linearity between predictors and response. - Independence of residuals. - Constant variance (homoscedasticity). - Normally distributed residuals (especially important for small n). --- Prediction Intervals: Concept and Use What a Prediction Interval Is A prediction interval (PI) estimates the range in which a single new observation is likely to fall, given what has been learned from sample data. Key differences from CIs: - A CI is about a parameter (for example, mean (\mu)). - A PI is about a future individual observation. Prediction intervals are always wider than corresponding confidence intervals because they include: - Uncertainty in estimating the mean. - Natural individual-to-individual variability around that mean. --- Prediction Interval for a Future Observation (Normal Data) One-Sample Case Assume: - Data are from a normal distribution. - The goal is to predict a single future observation (X_{\text{new}}). Given: - Sample size (n) - Sample mean (\bar{x}) - Sample standard deviation (s) A 2-sided prediction interval for a new observation from the same process is: [ \bar{x} \pm t_{\alpha/2, , n-1} \cdot s \cdot \sqrt{1 + \frac{1}{n}} ] Explanation: - The term (\sqrt{1 + \frac{1}{n}}) accounts for both: - Uncertainty in estimating the mean. - Natural variation of individual observations around the mean. As (n) becomes very large, the (\frac{1}{n}) term becomes negligible: - The prediction interval width approaches (t_{\alpha/2} \cdot s) (still much wider than the CI for the mean, which shrinks like (s/\sqrt{n})). Assumptions: - Observations are independent and normally distributed. - Future observation comes from the same process. --- Prediction Interval for Regression Predicting the Mean vs Predicting an Individual In the context of linear regression: - Confidence interval for mean response at (x_0): - Range of plausible values for the average response (E[Y|X = x_0]). - Prediction interval for an individual response at (x_0): - Range where a single future observation (Y{\text{new}}) at (X = x0) is likely to fall. Both intervals are centered at the predicted value: [ \hat{y}0 = \hat{\beta}0 + \hat{\beta}1 x0 + \cdots ] But the formulas differ: - Mean response CI at (x_0): [ \hat{y}0 \pm t{\alpha/2, , df} \cdot SE(\hat{y}_0) ] - Individual response PI at (x_0): [ \hat{y}0 \pm t{\alpha/2, , df} \cdot \sqrt{SE(\hat{y}_0)^2 + s^2} ] where: - (SE(\hat{y}0)) is the standard error of the predicted mean at (x0). - (s^2) is the residual variance (estimate of error variance). Consequences: - Prediction intervals are always wider than CIs for the mean response at the same (x_0). - The farther (x_0) is from the center of the observed predictor values, the wider both intervals become. Assumptions (same as regression CI assumptions): - Correct model form (linearity). - Independent residuals. - Constant variance of residuals. - Approximately normal residuals. --- Choosing Between Confidence and Prediction Intervals Which Interval to Use? The choice depends on the decision question: - Estimate a process parameter: - Use a confidence interval for the mean or proportion. - Examples: - Estimating average cycle time. - Estimating defect rate. - Compare groups or processes: - Use a confidence interval for the difference in means or proportions. - Example: - Comparing average performance before and after a change. - Forecast a future average (for many units): - Use a confidence interval for the mean response. - Example: - Estimating average yield in the next production run of large size. - Forecast a single future value: - Use a prediction interval. - Examples: - Predicting how long one specific transaction will take. - Predicting a single future measurement from a machine. Clarifying the decision question is critical to selecting the correct interval type. --- Assumptions, Diagnostics, and Robustness Common Assumptions For most confidence and prediction intervals considered here: - Independence: - Observations are not systematically related (no strong time-series correlation unless modeled). - Random sampling or representative data: - The sample must adequately represent the process or population of interest. - Approximate normality: - For small samples, the population (or residuals) is close to normal. - For large samples, the CLT often makes mean-based intervals robust to moderate non-normality. - Constant variance: - Particularly important in regression. Practical Diagnostics Before relying on intervals: - Examine data or residual plots: - Histograms or normal probability plots for normality. - Residuals vs fits or residuals vs predictors for constant variance and independence. - Watch for: - Extreme outliers or heavy skew that may invalidate standard intervals. - Clearly non-representative samples. When assumptions are seriously violated, alternative methods (transformations, nonparametric methods, or different models) may be required. --- Common Pitfalls and Misinterpretations Misinterpreting Confidence Intervals Avoid these misunderstandings: - “There is a 95% chance that the true mean is in this specific interval.” - The parameter is fixed; the interval method has a 95% long-run success rate. - “If two 95% CIs overlap, there is definitely no significant difference.” - Overlap does not automatically mean non-significance; formal comparison uses a CI for the difference or a hypothesis test. - “A narrow CI always means good data.” - A narrow CI can result from large sample size but may still be biased if the sampling is flawed. Misinterpreting Prediction Intervals Common pitfalls: - Treating a CI for the mean as if it applies to individual observations. - Individual values typically fall well outside the CI for the mean. - Ignoring that PIs are conditional on the model and assumptions holding for the future: - If the process changes, historical intervals may no longer apply. --- Practical Guidance for Application Steps to Construct and Interpret Intervals - Clarify the parameter or quantity of interest: - Mean, proportion, difference in means, regression coefficient, mean response, individual response. - Select the appropriate interval type: - CI for parameter estimation or group comparison. - PI for individual future outcomes. - Check sample size and assumptions: - Reasonable n? - Approximate normality (for mean-based intervals) or appropriate conditions for proportions? - Independence? - Compute point estimate and standard error: - Mean, proportion, or regression-based prediction. - Select confidence level: - Common choices: 90%, 95%, or 99%. - Obtain critical value: - t (typically) or z (proportions and large samples). - Construct the interval and interpret it in context: - Focus on both the range and its implications for decisions (for example, capability, compliance, cost impact). --- Summary Confidence and prediction intervals quantify uncertainty in different but complementary ways. - Confidence intervals: - Provide a range of plausible values for population parameters (means, proportions, differences, regression coefficients, mean responses). - Are built from point estimates, standard errors, and t or z critical values. - Depend on assumptions of independence, approximate normality (for means), and representative sampling. - Connect directly to hypothesis testing via the chosen confidence level. - Prediction intervals: - Provide a range where a single new observation is likely to fall. - Are always wider than the corresponding confidence intervals for the mean because they include both estimation uncertainty and natural individual variability. - Are especially important for forecasting individual outcomes from a stable process or regression model. Mastery involves: - Distinguishing clearly between estimating a parameter and predicting an individual value. - Selecting, constructing, and interpreting the appropriate interval type. - Understanding how assumptions, sample size, and variability affect interval width and reliability.