4.3.2 Experimental Methods

Experimental Methods Introduction Experimental methods in process improvement focus on deliberately changing inputs (factors) to observe their effect on outputs (responses). The goal is to learn cause-and-effect with minimal cost, time, and data. This article walks through the essential concepts and tools needed to plan, execute, and interpret experiments in a structured way. --- Purpose of Experimental Methods Experimental methods aim to: - Identify critical factors affecting a response. - Quantify how much each factor influences performance. - Find optimal factor settings to improve outcomes. - Validate whether observed improvements are statistically reliable. They are used when: - You suspect multiple factors affect a response. - You need more than simple observation or correlation. - You must choose among competing solutions or settings. --- Fundamental Concepts Factors, Levels, and Responses - Factor: An input you can deliberately change in an experiment. - Level: A specific setting of a factor (for example, low/high; 50 °C/70 °C). - Response: The output you measure to evaluate performance. Key distinctions: - Quantitative factors: Measured on a numeric scale (temperature, speed). - Qualitative factors: Categorical (machine A/B, method 1/2). Experimental Units and Randomization - Experimental unit: The smallest entity that can receive a treatment independently (part, batch, person, machine run). - Randomization: Assigning the run order randomly to protect against hidden time-related or environmental biases. Good practice: - Randomize run order whenever practical. - If full randomization is impossible, document restrictions (for example, blocking by day or batch). Replication and Reproducibility - Replication: Repeating the same treatment combination on different experimental units. - Reproducibility: Getting consistent results when the experiment is repeated under similar conditions, possibly by different operators or at different times. Reasons to replicate: - Estimate pure experimental error. - Increase precision of effect estimates. - Check stability of the process during the experiment. --- One-Factor-at-a-Time vs Designed Experiments One-Factor-at-a-Time (OFAT) Approach OFAT changes one factor while holding others constant. Limitations: - Does not reveal interactions between factors. - Often requires more runs for the same information. - May miss optimal combinations of factors. Designed Experiments Designed experiments change multiple factors simultaneously according to a structured plan. Advantages: - Estimate main effects and interactions efficiently. - Reduce the number of runs for the same knowledge. - Support systematic modeling and optimization. --- Main Effects and Interactions Main Effects A main effect is the average change in the response when a factor goes from one level to another, holding other factors averaged out. Interpretation: - Large main effect → factor strongly impacts the response. - Sign direction tells whether increasing the factor improves or worsens the response. Interactions An interaction occurs when the effect of one factor depends on the level of another factor. Examples: - Temperature only improves quality at a specific pressure. - A training method only works well at a certain workload. Why interactions matter: - Ignoring interactions can lead to wrong conclusions. - Optimal settings often result from favorable interaction combinations. --- Screening Designs Purpose of Screening Screening designs help identify the few important factors from many candidates using a relatively small number of runs. Typical goals: - Separate vital few factors from trivial many. - Decide which factors deserve detailed follow-up experiments. Plackett-Burman and Resolution III Designs Key ideas: - Use a large number of factors, each at two levels. - Small number of runs relative to factors. - Some interactions are confounded with main effects. Implications: - Good for early-stage exploration. - Results guide which factors to include in more detailed, higher-resolution designs. --- Full Factorial Designs Structure of Full Factorials A full factorial design includes all possible combinations of factor levels. For two-level designs: - With k factors, total runs = 2^k. Example: - 3 factors (A, B, C), each low/high → 2^3 = 8 runs. - All combinations: (A-,B-,C-), (A+,B-,C-), …, (A+,B+,C+). Information from Full Factorials Full factorials allow: - Estimation of all main effects. - Estimation of all interactions (two-way, three-way, etc.). - Clear understanding of factor interplay, if enough runs and replications are used. Drawback: - Number of runs grows quickly with factors. --- Fractional Factorial Designs Basic Idea A fractional factorial uses only a fraction of all possible combinations. Motivation: - Reduce experimental runs. - Focus on main effects and low-order interactions. Example: - 4 factors at 2 levels → full factorial: 2^4 = 16 runs. - A half-fraction design: 2^(4–1) = 8 runs. Resolution and Confounding - Confounding: When two or more effects (for example, a main effect and an interaction) are indistinguishable based on the data. - Design resolution indicates which effects are confounded: - Resolution III: Main effects may be aliased with two-factor interactions. - Resolution IV: Main effects are clear of two-factor interactions; two-factor interactions may be aliased with each other. - Resolution V: Main effects and two-factor interactions are clear of each other; two-factor interactions may be aliased with three-factor interactions. Choice guidelines: - Use at least Resolution IV when possible to keep main effects interpretable. - Use Resolution III mainly for early screening with many factors. --- Blocking and Randomization Blocking Blocking groups experimental runs into homogeneous subsets (blocks) to control nuisance variation. Examples of blocks: - Day of production. - Machine used. - Operator. Benefits: - Reduces noise from known but unwanted sources of variation. - Increases sensitivity to detect factor effects. Key concept: - Block effects are accounted for, but usually not of interest. - Some treatment combinations may be confounded with block effects. Restricted Randomization Sometimes full randomization is not practical due to setup or cost. Strategies: - Randomize within blocks or batches. - Keep a random pattern subject to operational constraints. - Document all restrictions and consider them in analysis. --- Response Surface Methods (RSM) Purpose of RSM Response surface methods are used after identifying key factors, when you want to: - Model the relationship between factors and response in more detail. - Identify optimal levels of factors. - Understand curvature (non-linear behavior) in the response. Second-Order Models RSM often uses second-order (quadratic) models including: - Linear terms (A, B, C). - Interaction terms (AB, AC, BC). - Quadratic terms (A², B², C²). These models can describe: - Curvature (optima, minima, saddle points). - Combined effects of factors on the response surface. --- Central Composite and Box-Behnken Designs Central Composite Designs (CCD) CCD are common RSM designs that build on a factorial or fractional factorial core. Components: - Factorial points: All combinations at low/high levels. - Axial (star) points: Points beyond the low/high levels on each factor axis. - Center points: Repeated runs at the midpoint of all factors. Reasons to use CCD: - Efficient estimation of quadratic models. - Flexibility in scaling factor ranges. - Can be “rotatable,” making prediction precision uniform at equal distances from the center. Box-Behnken Designs Box-Behnken designs are RSM designs that: - Use combinations of factors at midpoints and extremes, but avoid running all factors at extremes simultaneously. - Typically require fewer runs than CCD for a similar number of factors. - Keep all design points within a safe or practical region when extremes are risky. Use cases: - When experimental region must stay within moderate bounds. - When a spherical or near-spherical design region is desirable. --- Center Points and Curvature Role of Center Points Center points are runs where all factors are set to mid-levels. Functions: - Estimate pure error (if replicated). - Detect curvature by comparing mean response at center to mean of factorial corner points. Testing for Curvature Interpretation: - If center point mean differs significantly from factorial mean, curvature is present. - Evidence of curvature suggests a linear model is inadequate and quadratic terms are needed. --- Replication, Repetition, and Random Error Replication vs Repetition - Replication: Independent runs with full reset of conditions (new batch, new unit). - Repetition: Multiple measurements on the same experimental unit under essentially identical conditions. Purposes: - Replication: Estimate variability between units and over time. - Repetition: Assess measurement system variation. Random Error and Power - Random error: Natural variability in response not explained by factors. - Statistical power: Probability of detecting a true effect. To improve power: - Increase replications. - Reduce noise sources (stabilize process, improve measurement). - Focus factors within a relevant range where effects are large enough to detect. --- Analysis of Experimental Data Descriptive Tools Common tools used before formal modeling: - Main effects plots: Show average response at low vs high levels of each factor. - Interaction plots: Show whether the effect of one factor changes across levels of another. - Residual plots: Assess model adequacy and check assumptions. These help visualize structure before deeper analysis. ANOVA for Designed Experiments Analysis of variance (ANOVA) partitions total variability into components due to: - Factors and interactions (explained variation). - Error (unexplained variation). Key outputs: - F-statistics and p-values for each effect. - Estimates of variance components. - Overall model significance. Interpretation: - Effects with small p-values are statistically significant. - Non-significant higher-order interactions can often be removed to simplify the model. Model Adequacy Checking After fitting a model: - Check residual plots for patterns or non-constant variance. - Examine normal probability plots of residuals. - Confirm no obvious time trends or block-related issues. If assumptions are violated: - Consider transforming the response. - Investigate missing factors or non-linearities. - Re-express the model or perform follow-up experiments. --- Sequential Experimentation and Confirmation Sequential Strategy Experiments are often best planned in stages: - Initial screening to find important factors. - Follow-up factorial or fractional factorial on key factors. - RSM to refine and optimize. - Confirmation runs to verify the chosen settings. Benefits: - Reduces risk of large up-front investment in the wrong factors. - Allows learning to guide subsequent designs. Confirmation Experiments After identifying optimal or improved settings: - Run confirmation experiments at predicted optimum. - Compare predicted vs observed response. Confirmation evaluates: - Practical significance of improvement. - Stability of gains under normal operating conditions. --- Robustness and Noise Factors Noise and Control Factors - Control factors: Inputs you can deliberately set and control. - Noise factors: Inputs that vary but are difficult or costly to control (ambient temperature, customer behavior). Goal: - Choose control factor settings that keep the response stable and acceptable despite noise variation. Robust Experimental Strategies To study robustness: - Intentionally vary noise factors during experimentation. - Evaluate how sensitive the response is to noise under different control settings. - Prefer solutions with smaller variability and acceptable mean performance. This leads to processes less sensitive to real-world fluctuations. --- Practical Planning Considerations Choosing Factor Ranges and Levels Consider: - Engineering or process limits. - Expected non-linear behavior (use wide enough range to reveal it, but stay safe). - Operational relevance (settings must be feasible in practice). Use: - At least two levels for initial screening. - Three or more distinct settings (including center levels) when curvature is expected. Run Size and Resource Constraints Balance: - Amount of information needed about effects and interactions. - Available time, cost, and material. - Need for replication and blocking. Plan: - Start with lean designs. - Reserve resources for follow-up stages. --- Summary Experimental methods provide a structured way to learn cause-and-effect in processes by: - Systematically varying factors and observing responses. - Using designs such as screening, full factorial, fractional factorial, and response surface methods. - Managing randomization, blocking, replication, and center points to control and understand variation. - Analyzing data with plots and ANOVA to estimate main effects, interactions, and curvature. - Applying sequential experimentation and confirmation to move from exploration to optimization and robust performance. Mastering these concepts enables effective design, execution, and interpretation of experiments that drive reliable, data-based process improvements.