Statistical learning from a regression perspective

1 Statistical Learning as a Regression Problem 1.1 Getting Started 1.2 Setting the Regression Context 1.3 Revisiting the Ubiquitous Linear Regression Model 1.3.1 Problems in Practice 1.4 Working with Statistical Models that are Wrong 1.4.1 An Alternative Approach to Regression 1.4.2 More on Statistical Inference with Wrong Models 1.4.3 Introduction to Sandwich Standard Errors 1.4.4 Introduction to Conformal Inference 1.4.5 Introduction to the Nonparametric Bootstrap 1.4.6 Wrong Regression Models with Binary Response Variables 1.5 The Transition to Statistical Learning 1.5.1 Models Versus Algorithms 1.6 Some Initial Concepts 1.6.1 Overall Goals of Statistical Learning 1.6.2 Forecasting with Supervised Statistical Learning 1.6.3 Overfitting 1.6.4 Data Snooping 1.6.5 Some Constructive Responses to Overfitting and Data Snooping 1.6.6 Loss Functions and Related Concepts 1.6.7 The Bias-Variance Tradeoff 1.6.8 Linear Estimators 1.6.9 Degrees of Freedom 1.6.10 Basis Functions 1.6.11 The Curse of Dimensionality 1.7 Statistical Learning in Context Endnotes References 2 Splines, Smoothers, and Kernels 2.1 Introduction 2.2 Regression Splines 2.2.1 Piecewise Linear Population Approximations 2.2.2 Polynomial Regression Splines 2.2.3 Natural Cubic Splines 2.2.4 B-Splines 2.3 Penalized Smoothing 2.3.1 Shrinkage and Regularization 2.4 Penalized Regression Splines 2.4.1 An Application 2.5 Smoothing Splines 2.5.1 A Smoothing Splines Illustration 2.6 Locally Weighted Regression as a Smoother 2.6.1 Nearest Nei or Methods 2.6.2 Locally Weighted Regression 2.7 Smoothers for Multiple Predictors 2.7.1 Smoothing in Two Dimensions 2.7.2 The Generalized Additive Model 2.8 Smoothers with Categorical Variables 2.8.1 An Illustration Using the Generalized Additive Model with a Binary Outcome 2.9 An Illustration of Statistical Inference After Model Selection 2.9.1 Level I Versus Level II Summary 2.10 Kernelized Regression 2.10.1 Radial Basis Kernel 2.10.2 ANOVA Radial Basis Kernel 2.10.3 A Kernel Regression Application 2.11 Summary and Conclusions Endnotes References 3 Classification and Regression Trees (CART) 3.1 Introduction 3.2 An Introduction to Recursive Partitioning in CART 3.3 The Basic Ideas in More Depth 3.3.1 Tree Diagrams for Showing What the Greedy Algorithm Determined 3.3.2 An Initial Application 3.3.3 Classification and Forecasting with CART 3.3.4 Confusion Tables 3.3.5 CART as an Adaptive Nearest Nei or Method 3.4 The Formalities of Splitting a Node 3.5 An Illustrative Prison Inmate Risk Assessment Using CART ... 3.6 Classification Errors and Costs 3.6.1 Default Costs in CART 3.6.2 Prior Probabilities and Relative Misclassification Costs 3.7 Varying the Prior and the Complexity Parameter 3.8 An Example with Three Response Categories 3.9 Regression Trees 3.9.1 A CART Application for the Correlates of a Student's GPA in High School 3.10 Pruning 3.11 Missing Data 3.11.1 Missing Data with CART 3.12 More on CART Instability 3.13 Summary of Statistical Inference with CART 3.13.1 Summary of Statistical Inference for CART Forecasts 3.14 Overall Summary and Conclusions Exercises Endnotes References 4 Bagging 4.1 Introduction 4.2 The Bagging Algorithm 4.3 Some Bagging Details 4.3.1 Revisiting the CART Instability Problem 4.3.2 Resampling Methods for Bagging 4.3.3 Votes Over Trees and Probabilities 4.3.4 Forecasting and Imputation 4.3.5 Bagging Estimation and Statistical Inference 4.3.6 Margins for Classification 4.3.7 Using Out-of-Bag Observations as Test Data 4.3.8 Baggi