Planning and Executing Credible Experiments

Covers experiment planning, execution, analysis, and reporting
 
This single-source resource guides readers in planning and conducting credible experiments for engineering, science, industrial processes, agriculture, and business. The text takes experimenters all the way through conducting a high-impact experiment, from initial conception, through execution of the experiment, to a defensible final report. It prepares the reader to anticipate the choices faced during each stage.
 
Filled with real-world examples from engineering science and industry, Planning and Executing Credible Experiments: A Guidebook for Engineering, Science, Industrial Processes, Agriculture, and Business offers chapters that challenge experimenters at each stage of planning and execution and emphasizes uncertainty analysis as a design tool in addition to its role for reporting results. Tested over decades at Stanford University and internationally, the text employs two powerful, free, open-source software tools: GOSSET to optimize experiment design, and R for statistical computing and graphics. A website accompanies the text, providing additional resources and software downloads.
* A comprehensive guide to experiment planning, execution, and analysis
* Leads from initial conception, through the experiment's launch, to final report
* Prepares the reader to anticipate the choices faced throughout an experiment
* Hones the motivating question
* Employs principles and techniques from Design of Experiments (DoE)
* Selects experiment designs to obtain the most information from fewer experimental runs
* Offers chapters that propose questions that an experimenter will need to ask and answer during each stage of planning and execution
* Demonstrates how uncertainty analysis guides and strengthens each stage
* Includes examples from real-life industrial experiments
* Accompanied by a website hosting open-source software
 
Planning and Executing Credible Experiments is an excellent resource for graduates and senior undergraduates--as well as professionals--across a wide variety of engineering disciplines.

About the Authors xxi Preface xxiii Acknowledgments xxvii About the Companion Website xxix 1 Choosing Credibility 1 1.1 The Responsibility of an Experimentalist 2 1.2 Losses of Credibility 2 1.3 Recovering Credibility 3 1.4 Starting with a Sharp Axe 3 1.5 A Systems View of Experimental Work 4 1.6 In Defense of Being a Generalist 5 Panel 1.1 The Bundt Cake Story 6 References 6 Homework 6 2 The Nature of Experimental Work 7 2.1 Tested Guide of Strategy and Tactics 7 2.2 What Can Be Measured and What Cannot? 8 2.2.1 Examples Not Measurable 8 2.2.2 Shapes 9 2.2.3 Measurable by the Human Sensory System 10 2.2.4 Identifying and Selecting Measurable Factors 11 2.2.5 Intrusive Measurements 11 2.3 Beware Measuring Without Understanding: Warnings from History 12 2.4 How Does Experimental Work Differ from Theory and Analysis? 13 2.4.1 Logical Mode 13 2.4.2 Persistence 13 2.4.3 Resolution 13 2.4.4 Dimensionality 15 2.4.5 Similarity and Dimensional Analysis 15 2.4.6 Listening to Our Theoretician Compatriots 16 Panel 2.1 Positive Consequences of the Reproducibility Crisis 17 Panel 2.2 Selected Invitations to Experimental Research, Insights from Theoreticians 18 Panel 2.3 Prepublishing Your Experiment Plan 21 2.4.7 Surveys and Polls 22 2.5 Uncertainty 23 2.6 Uncertainty Analysis 23 References 24 Homework 25 3 An Overview of Experiment Planning 27 3.1 Steps in an Experimental Plan 27 3.2 Iteration and Refinement 28 3.3 Risk Assessment/Risk Abatement 28 3.4 Questions to Guide Planning of an Experiment 29 Homework 30 4 Identifying the Motivating Question 31 4.1 The Prime Need 31 Panel 4.1 There's a Hole in My Bucket 32 4.2 An Anchor and a Sieve 33 4.3 Identifying the Motivating Question Clarifies Thinking 33 4.3.1 Getting Started 33 4.3.2 Probe and Focus 34 4.4 Three Levels of Questions 35 4.5 Strong Inference 36 4.6 Agree on the Form of an Acceptable Answer 36 4.7 Specify the Allowable Uncertainty 37 4.8 Final Closure 37 Reference 38 Homework 38 5 Choosing the Approach 39 5.1 Laying Groundwork 39 5.2 Experiment Classifications 40 5.2.1 Exploratory 40 5.2.2 Identifying the Important Variables 40 5.2.3 Demonstration of System Performance 41 5.2.4 Testing a Hypothesis 41 5.2.5 Developing Constants for Predetermined Models 41 5.2.6 Custody Transfer and System Performance Certification Tests 42 5.2.7 Quality-Assurance Tests 42 5.2.8 Summary 43 5.3 Real or Simplified Conditions? 43 5.4 Single-Sample or Multiple-Sample? 43 Panel 5.1 A Brief Summary of "Dissertation upon Roast Pig" 44 Panel 5.2 Consider a Spherical Cow 44 5.5 Statistical or Parametric Experiment Design? 45 5.6 Supportive or Refutative? 47 5.7 The Bottom Line 47 References 48 Homework 48 6 Mapping for Safety, Operation, and Results 51 6.1 Construct Multiple Maps to Illustrate and Guide Experiment Plan 51 6.2 Mapping Prior Work and Proposed Work 51 6.3 Mapping the Operable Domain of an Apparatus 53 6.4 Mapping in Operator's Coordinates 57 6.5 Mapping the Response Surface 59 6.5.1 Options for Organizing a Table 59 6.5.2 Options for Presenting the Response on a Scatter-Plot-Type Graph 61 Homework 64 7 Refreshing Statistics 65 7.1 Reviving Key Terms to Quantify Uncertainty 65 7.1.1 Population 65 7.1.2 Sample 66 7.1.3 Central Value 67 7.1.4 Mean, or ? 67 7.1.5 Residual 67 7.1.6 Variance, 2 or S2 68 7.1.7 Degrees of Freedom, Df 68 7.1.8 Standard Deviation, Y or SY 68 7.1.9 Uncertainty of the Mean, 69 7.1.10 Chi?Squared, 2 69 7.1.11 p?Value 70 7.1.12 Null Hypothesis 70 7.1.13 F?value of Fisher Statistic 71 7.2 The Data Distribution Most Commonly Encountered The Normal Distribution for Samples of Infinite Size 71 7.3 Account for Small Samples: The t?Distribution 72 7.4 Construct Simple Models by Computer to Explain the Data 73 7.4.1 Basic Statistical Analysis of Quantitative Data 73 7.4.2 Model Data Containing Categorical and Quantitative Factors 75 7.4.3 Display Data Fit to One Categorical Factor and One Quantitative Factor 76 7.4.4 Quantify How Each Factor Accounts for Variation in the Data 76 7.5 Gain Confidence and Skill at Statistical Modeling Via the R Language 77 7.5.1 Model and Plot Results of a Single Variable Using the Example Data diceshoe.csv 77 7.5.2 Evaluate Alternative Models of the Example Data hiloy.csv 78 7.5.2.1 Inspect the Data 78 7.5.3 Grand Mean 78 7.5.4 Model by Groups: Group?Wise Mean 78 7.5.5 Model by a Quantitative Factor 78 7.5.6 Model by Multiple Quantitative Factors 78 7.5.7 Allow Factors to Interact (So Each Group Gets Its Own Slope) 79 7.5.8 Include Polynomial Factors (a Statistical Linear Model Can Be Curved) 80 7.6 Report Uncertainty 80 7.7 Decrease Uncertainty (Improve Credibility) by Isolating Distinct Groups 81 7.8 Original Data, Summary, and R 82 References 83 Homework 83 8 Exploring Statistical Design of Experiments 87 8.1 Always Seeking Wiser Strategies 87 8.2 Evolving from Novice Experiment Design 87 8.3 Two?Level and Three?Level Factorial Experiment Plans 88 8.4 A Three?Level, Three?Factor Design 89 8.5 The Plackett-Burman 12?Run Screening Design 93 8.6 Details About Analysis of Statistically Designed Experiments 95 8.6.1 Model Main Factors to Original Raw Data 95 8.6.2 Model Main Factors to Original Data Around Center of Each Factor 96 8.6.3 Model Including All Interaction Terms 97 8.6.4 Model Including Only Dominant Interaction Terms 97 8.6.5 Model Including Dominant Interaction Term Plus Quadratic Term 98 8.6.6 Model All Factors of Example 2, Centering Each Quantitative Factor 99 8.6.7 Refine Model of Example 2 Including Only Dominant Terms 100 8.7 Retrospect of Statistical Design Examples 101 8.8 Philosophy of Statistical Design 101 8.9 Statistical Design for Conditions That Challenge Factorial Designs 102 8.10 A Highly Recommended Tool for Statistical Design of Experiments 103 8.11 More Tools for Statistical Design of Experiments 103 8.12 Conclusion 103 Further Reading 104 Homework 104 9 Selecting the Data Points 107 9.1 The Three Categories of Data 107 9.1.1 The Output Data 107 9.1.2 Peripheral Data 108 9.1.3 Backup Data 108 9.1.4 Other Data You May Wish to Acquire 108 9.2 Populating the Operating Volume 109 9.2.1 Locating the Data Points Within the Operating Volume 109 9.2.2 Estimating the Topography of the Response Surface 109 9.3 Example from Velocimetry 109 9.3.1 Sharpen Our Approach 110 9.3.2 Lessons Learned from Velocimetry Example 111 9.4 Organize the Data 112 9.4.1 Keep a Laboratory Notebook 112 9.4.2 Plan for Data Security 112 9.4.3 Decide Data Format 112 9.4.4 Overview Data Guidelines 112 9.4.5 Reasoning Through Data Guidelines 113 9.5 Strategies to Select Next Data Points 114 9.5.1 Overview of Option 1: Default Strategy with Intensive Experimenter Involvement 115 9.5.1.1 Choosing the Data Trajectory 115 9.5.1.2 The Default Strategy: Be Bold 115 9.5.1.3 Anticipate, Check, Course Correct 116 9.5.1.4 Other Aspects to Keep in Mind 116 9.5.1.5 Endpoints 117 9.5.2 Reintroducing Gosset 118 9.5.3 Practice Gosset Examples (from Gosset User Manual) 119 9.6 Demonstrate Gosset for Selecting Data 120 9.6.1 Status Quo of Experiment Planning and Execution (Prior to Selecting More Samples) 120 9.6.1.1 Specified Motivating Question 120 9.6.1.2 Identified Pertinent Candidate Factors 121 9.6.1.3 Selected Initial Sample Points Using Plackett-Burman 121 9.6.1.4 Executed the First 12 Runs at the PB Sample Conditions 122 9.6.1.5 Analyzed Results. Identified Dominant First-Order Factors. Estimated First-Order Uncertainties of Factors 123 9.6.1.6 Generated Draft Predictive Equation 124 9.6.2 Use Gosset to Select Additional Data Samples 125 9.6.2.1 Example Gosset Session: User Input to Select Next Points 125 9.6.2.2 Example Gosset Session: How We Chose User Input 126 9.6.2.3 Example Gosset Session: User Input Along with Gosset Output 128 9.6.2.4 Example Gosset Session: Convert the Gosset Design to Operator Values 131 9.6.2.5 Results of Example Gosset Session: Operator Plots of Total Experiment Plan 132 9.6.2.6 Execute Stage Two of the Experiment Plan: User Plus Gosset Sample Points 132 9.7 Use Gosset to Analyze Results 133 9.8 Other Options and Features of Gosset 133 9.9 Using Gosset to Find Local Extrema in a Function of Several Variables 134 9.10 Summary 137 Further Reading 137 Homework 137 10 Analyzing Measurement Uncertainty 143 10.1 Clarifying Uncertainty Analysis 143 10.1.1 Distinguish Error and Uncertainty 144 10.1.1.1 Single-Sample vs. Multiple-Sample 145 10.1.2 Uncertainty as a Diagnostic Tool 146 10.1.2.1 What Can Uncertainty Analysis Tell You? 146 10.1.2.2 What is Uncertainty Analysis Good For? 148 10.1.2.3 Uncertainty Analysis Can Redirect a Poorly Conceived Experiment 148 10.1.2.4 Uncertainty Analysis Improves the Quality of Your Work 148 10.1.2.5 Slow Sampling and "Randomness" 149 10.1.2.6 Uncertainty Analysis Makes Results Believable 150 10.1.3 Uncertainty Analysis Aids Management Decision-Making 150 10.1.3.1 Management's Task: Dealing with Warranty Issues 150 10.1.4 The Design Group's Task: Setting Tolerances on Performance Test Repeatability 152 10.1.5 The Performance Test Group's Task: Setting the Tolerances on Measurements 152 10.2 Definitions 153 10.2.1 True Value 153 10.2.2 Corrected Value 153 10.2.3 Data Reduction Program 153 10.2.4 Accuracy 153 10.2.5 Error 154 10.2.6 XXXX Error 154 10.2.7 Fixed Error 154 10.2.8 Residual Fixed Error 154 10.2.9 Random Error 154 10.2.10 Variable (but Deterministic) Error 155 10.2.11 Uncertainty 155 10.2.12 Odds 155 10.2.13 Absolute Uncertainty 155 10.2.14 Relative Uncertainty 155 10.3 The Sources and Types of Errors 156 10.3.1 Types of Errors: Fixed, Random, and Variable 156 10.3.2 Sources of Errors: The Measurement Chain 156 10.3.2.1 The Undisturbed Value 158 10.3.2.2 The Available Value 158 10.3.2.3 The Achieved Value 158 10.3.2.4 The Observed Value 159 10.3.2.5 The Corrected Value 159 10.3.3 Specifying the True Value 160 10.3.3.1 If the Achieved Value is Taken as the True Value 160 10.3.3.2 If the Available Value is Taken as the True Value 163 10.3.3.3 If the Undisturbed Value is Taken as the True Value 166 10.3.3.4 If the Mixed Mean Gas Temperature is Taken as the True Value 167 10.3.4 The Role of the End User 167 10.3.4.1 The End-Use Equations Implicitly Define the True Value 167 10.3.5 Calibration 168 10.4 The Basic Mathematics 170 10.4.1 The Root-Sum-Squared (RSS) Combination 170 10.4.2 The Fixed Error in a Measurement 171 10.4.3 The Random Error in a Measurement 172 10.4.4 The Uncertainty in a Measurement 173 10.4.5 The Uncertainty in a Calculated Result 174 10.4.5.1 The Relative Uncertainty in a Result 176 10.5 Automating the Uncertainty Analysis 178 10.5.1 The Mathematical Basis 178 10.5.2 Example of Uncertainty Analysis by Spreadsheet 179 10.6 Single-Sample Uncertainty Analysis 181 10.6.1 Assembling the Necessary Inputs 184 10.6.2 Calculating the Uncertainty in the Result 185 10.6.3 The Three Levels of Uncertainty: Zeroth-, First-, and Nth-Order 185 10.6.3.1 Zeroth-Order Replication 186 10.6.3.2 First-Order Replication 187 10.6.3.3 Nth-Order Replication 188 10.6.4 Fractional-Order Replication for Special Cases 188 10.6.5 Summary of Single-Sample Uncertainty Levels 189 10.6.5.1 Zeroth-Order 189 10.6.5.2 First-Order 190 10.6.5.3 Nth-Order 190 References 190 Further Reading 191 Homework 191 11 Using Uncertainty Analysis in Planning and Execution 197 11.1 Using Uncertainty Analysis in Planning 197 11.1.1 The Physical Situation and Energy Analysis 198 11.1.2 The Steady?State Method 199 11.1.3 The Transient Method 200 11.1.4 Reflecting on Assumptions Made During DRE Derivations 201 11.2 Perform Uncertainty Analysis on the DREs 202 11.2.1 Uncertainty Analysis: General Form 202 11.2.2 Uncertainty Analysis of the Steady?State Method 203 11.2.3 Uncertainty Analysis - Transient Method 204 11.2.4 Compare the Results of Uncertainty Analysis of the Methods 205 11.2.5 What Does the Calculated Uncertainty Interval Mean? 206 11.2.6 Cross?Checking the Experiment 207 11.2.7 Conclusions 207 11.3 Using Uncertainty Analysis in Selecting Instruments 208 11.4 Using Uncertainty Analysis in Debugging an Experiment 209 11.4.1 Handling Overall Scatter 209 11.4.2 Sources of Scatter 210 11.4.3 Advancing Toward Calibration 211 11.4.4 Selecting Thresholds 212 11.4.5 Iterating Analysis 212 11.4.6 Rechecking Situational Uncertainty 212 11.5 Reporting the Uncertainties in an Experiment 213 11.5.1 Progress in Uncertainty Reporting 214 11.6 Multiple?Sample Uncertainty Analysis 214 11.6.1 Revisiting Single?Sample and Multiple?Sample Uncertainty Analysis 214 11.6.2 Examples of Multiple?Sample Uncertainty Analysis 215 11.6.3 Fixed Error and Random Error 216 11.7 Coordinate with Uncertainty Analysis Standards 216 11.7.1 Describing Fixed and Random Errors in a Measurement 217 11.7.2 The Bias Limit 217 11.7.2.1 Fossilization 218 11.7.2.2 Bias Limits 218 11.7.3 The Precision Index 219 11.7.4 The Number of Degrees of Freedom 220 11.8 Describing the Overall Uncertainty in a Single Measurement 220 11.8.1 Adjusting for a Single Measurement 220 11.8.2 Describing the Overall Uncertainty in a Result 221 11.8.3 Adding the Overall Uncertainty to Predictive Models 222 11.9 Additional Statistical Tools and Elements 222 11.9.1 Pooled Variance 222 11.9.1.1 Student's t?Distribution - Pooled Examples 223 11.9.2 Estimating the Standard Deviation of a Population from the Standard Deviation of a Small Sample: The Chi?Squared 2 Distribution 223 References 225 Homework 226 12 Debugging an Experiment, Shakedown, and Validation 231 12.1 Introduction 231 12.2 Classes of Error 231 12.3 Using Time-Series Analysis in Debugging 232 12.4 Examples 232 12.4.1 Gas Temperature Measurement 232 12.4.2 Calibration of a Strain Gauge 233 12.4.3 Lessons Learned from Examples 234 12.5 Process Unsteadiness 234 12.6 The Effect of Time-Constant Mismatching 235 12.7 Using Uncertainty Analysis in Debugging an Experiment 236 12.7.1 Calibration and Repeatability 236 12.7.2 Stability and Baselining 238 12.8 Debugging the Experiment via the Data Interpretation Program 239 12.8.1 Debug the Experiment via the DIP 239 12.8.2 Debug the Interface of the DIP 239 12.8.3 Debug Routines in the DIP 240 12.9 Situational Uncertainty 241 13 Trimming Uncertainty 243 13.1 Focusing on the Goal 243 13.2 A Motivating Question for Industrial Production 243 13.2.1 Agreed Motivating Questions for Industrial Example 244 13.2.2 Quick Answers to Motivating Questions 244 13.2.3 Challenge: Precheck Analysis and Answers 245 13.3 Plackett-Burman 12-Run Results and Motivating Question #3 245 13.4 PB 12-Run Results and Motivating Question #1 247 13.4.1 Building a Predictive Model Equation from R-Language Linear Model 248 13.4.2 Parsing the Dual Predictive Model Equation 249 13.4.3 Uncertainty of the Intercept in the Dual Predictive Model Equation 250 13.4.4 Mapping an Answer to Motivating Question #1 251 13.4.5 Tentative Answers to Motivating Question #1 252 13.5 Uncertainty Analysis of Dual Predictive Model and Motivating Question #2 252 13.5.1 Uncertainty of the Constant in the Dual Predictive Model Equation 252 13.5.2 Uncertainty of Other Factors in the Dual Predictive Model Equation 253 13.5.3 Include All Coefficient Uncertainties in the Dual Predictive Model Equation 254 13.5.4 Overall Uncertainty from All Factors in the Predictive Model Equation 254 13.5.5 Improved Tentative Answers to Motivating Questions, Including Uncertainties 256 13.5.6 Search for Improved Predictive Models 256 13.6 The PB 12-Run Results and Individual Machine Models 256 13.6.1 Individual Machine Predictive Model Equations 258 13.6.2 Uncertainty of the Intercept in the Individual Predictive Model Equations 258 13.6.3 Uncertainty of the Constant in the Individual Predictive Model Equations 259 13.6.4 Uncertainty of Other Factors in the Individual Predictive Model Equation 259 13.6.4.1 Uncertainties of Machine 1 259 13.6.4.2 Uncertainties of Machine 2 260 13.6.4.3 Including Instrument and Measurement Uncertainties 260 13.6.5 Include All Coefficient Uncertainties in the Individual Predictive Model Equations 260 13.6.6 Overall Uncertainty from All Factors in the Individual Predictive Model Equations 261 13.6.7 Quick Overview of Individual Machine Performance Over the Operating Map 262 13.7 Final Answers to All Motivating Questions for the PB Example Experiment 263 13.7.1 Answers to Motivating Question #1 263 13.7.2 Answers to Motivating Question #2 263 13.7.3 Answers to Motivating Question #3 (Expanded from Section 13.3) 263 13.7.4 Answers to Motivating Question #4 264 13.7.5 Other Recommendations (to Our Client) 264 13.8 Conclusions 265 Homework 266 14 Documenting the Experiment: Report Writing 269 14.1 The Logbook 269 14.2 Report Writing 269 14.2.1 Organization of the Reports 270 14.2.2 Who Reads What? 270 14.2.3 Picking a Viewpoint 271 14.2.4 What Goes Where? 271 14.2.4.1 What Goes in the Abstract? 272 14.2.4.2 What Goes in the Foreword? 272 14.2.4.3 What Goes in the Objective? 273 14.2.4.4 What Goes in the Results and Conclusions? 273 14.2.4.5 What Goes in the Discussion? 274 14.2.4.6 References 274 14.2.4.7 Figures 275 14.2.4.8 Tables 276 14.2.4.9 Appendices 276 14.2.5 The Mechanics of Report Writing 276 14.2.6 Clear Language Versus "JARGON" 277 Panel 14.1 The Turbo-Encabulator 278 14.2.7 "Gobbledygook": Structural Jargon 279 Panel 14.2 U.S. Code, Title 18, No. 793 279 14.2.8 Quantitative Writing 281 14.2.8.1 Substantive Versus Descriptive Writing 281 Panel 14.3 The Descriptive Bank Statement 281 14.2.8.2 Zero-Information Statements 281 14.2.8.3 Change 282 14.3 International Organization for Standardization, ISO 9000 and other Standards 282 14.4 Never Forget. Always Remember 282 Appendix A: Distributing Variation and Pooled Variance 283 A.1 Inescapable Distributions 283 A.1.1 The Normal Distribution for Samples of Infinite Size 283 A.1.2 Adjust Normal Distributions with Few Data: The Student's t-Distribution 283 A.2 Other Common Distributions 286 A.3 Pooled Variance (Advanced Topic) 286 Appendix B: Illustrative Tables for Statistical Design 289 B.1 Useful Tables for Statistical Design of Experiments 289 B.1.1 Ready-made Ordering for Randomized Trials 289 B.1.2 Exhausting Sets of Two-Level Factorial Designs ( Five Factors) 289 B.2 The Plackett-Burman (PB) Screening Designs 289 Appendix C: Hand Analysis of a Two-Level Factorial Design 293 C.1 The General Two-Level Factorial Design 293 C.2 Estimating the Significance of the Apparent Factor Effects 298 C.3 Hand Analysis of a Plackett-Burman (PB) 12-Run Design 299 C.4 Illustrative Practice Example for the PB 12-Run Pattern 302 C.4.1 Assignment: Find Factor Effects and the Linear Coefficients Absent Noise 302 C.4.2 Assignment: Find Factor Effects and the Linear Coefficients with Noise 303 C.5 Answer Key: Compare Your Hand Calculations 303 C.5.1 Expected Results Absent Noise (compare C.4.1) 303