Fitting

Models are actually families of models, with every parameter combination specifying a different model.

To fit a model means to identify from the family of models the specific model that fits the data best.

Possible the most important concept in statistics and machine learning.

The loss function defines some summary of the errors committed by the model.

$$\Large Loss = f(Error)$$

Two purposes

In regression, the criterion $Y$ is modeled as the sum of features $X_1, X_2, ...$ times weights $\beta_1, \beta_2, ...$ plus $\beta_0$ the so-called the intercept.

$$\large \hat{Y} = \beta_{0} + \beta_{1} \times X_1 + \beta_{2} \times X2 + ...$$

The weight $\beta_{i}$ indiciates the amount of change in $\hat{Y}$ for a change of 1 in $X_{i}$.

Ceteris paribus, the more extreme $\beta_{i}$, the more important $X_{i}$ for the prediction of $Y$ (Note: the scale of $X_{i}$ matters too!).

If $\beta_{i} = 0$, then $X_{i}$ does not help predicting $Y$

Mean Squared Error (MSE)

Average squared distance between predictions and true values?
$$MSE = \frac{1}{n}\sum_{i \in 1,...,n}(Y_{i} - \hat{Y}_{i})^{2}$$
Mean Absolute Error (MAE)

Average absolute distance between predictions and true values?
$$MAE = \frac{1}{n}\sum_{i \in 1,...,n} \lvert Y_{i} - \hat{Y}_{i} \rvert$$

There are two fundamentally different ways to find the set of parameters that minimizes loss.

Analytically

Fitting

There are two fundamentally different ways to find the set of parameters that minimizes loss.

Analytically

There are two types of supervised learning problems that can often be approached using the same model.

Regression problems involve the prediction of a quantitative feature.

E.g., predicting the cholesterol level as a function of age.

Classification problems involve the prediction of a categorical feature.

E.g., predicting the type of chest pain as a function of age.

In logistic regression, the class criterion $Y \in (0,1)$ is modeled also as the sum of feature times weights, but with the prediction being transformed using a logistic link function:

$$\large \hat{Y} = Logistic(\beta_{0} + \beta_{1} \times X_1 + ...)$$

The logistic function maps predictions to the range of 0 and 1, the two class values.

$$Logistic(x) = \frac{1}{1+exp(-x)}$$

In logistic regression, the class criterion $Y \in (0,1)$ is modeled also as the sum of feature times weights, but with the prediction being transformed using a logistic link function:

$$\large \hat{Y} = Logistic(\beta_{0} + \beta_{1} \times X_1 + ...)$$

The logistic function maps predictions to the range of 0 and 1, the two class values.

$$Logistic(x) = \frac{1}{1+exp(-x)}$$

Logloss is used to fit the parameters, alternative distance measures are MSE and MAE.

$$\small LogLoss = -\frac{1}{n}\sum_{i}^{n}(log(\hat{y})y+log(1-\hat{y})(1-y))$$ $$\small MSE = \frac{1}{n}\sum_{i}^{n}(y-\hat{y})^2, \: MAE = \frac{1}{n}\sum_{i}^{n} \lvert y-\hat{y} \rvert$$

Does the predicted class match the actual class. Often preferred for ease of interpretation.

$$\small Loss_{01}=\frac{1}{n}\sum_i^n I(y \neq \lfloor \hat{y} \rceil)$$

The confusion matrix tabulates prediction matches and mismatches as a function of the true class.

The confusion matrix permits specification of a number of helpful performance metrics.

Confusion matrix

The confusion matrix tabulates prediction matches and mismatches as a function of the true class.

The confusion matrix permits specification of a number of helpful performance metrics.

Confusion matrix

Model performance must be evaluated as true prediction on an unseen data set.

The unseen data set can be naturally occurring, e.g., using 2019 stock prizes to evaluate a model fit using 2018 stock prizes.

More commonly unseen data is created by splitting the available data into a training set and a test set.

Overfitting

Occurs when a model fits data too closely and therefore fails to reliably predict future observations.

In other words, overfitting occurs when a model 'mistakes' random noise for a predictable signal.

More complex models are more prone to overfitting.

Penalizes regression loss for having large $\beta$ values using the lambda λ tuning parameter and one of several penalty functions.

$$Regularized \;loss = \sum_i^n (y_i-\hat{y}_i)^2+\lambda \sum_j^p f(\beta_j))$$

Regularized regression

Despite superficial similarities, Lasso and Ridge show very different behavior.

Ridge

By penalizing the most extreme βs most strongly, Ridge leads to (relatively) more uniform βs.

Lasso

By penalizing all βs equally, irrespective of magnitude, Lasso drives some βs to 0 resulting effectively in automatic feature selection.

CART is short for Classification and Regression Trees, which are often just called Decision trees.

In decision trees, the criterion is modeled as a sequence of logical TRUE or FALSE questions.

Classification trees (and regression trees) are created using a relatively simple three-step algorithm.

Algorithm

1 - Split nodes to maximize purity gain (e.g., Gini gain).

2 - Repeat until pre-defined threshold (e.g., minsplit) splits are no longer possible.

3 - Prune tree to reasonable size.

Classification trees attempt to minize node impurity using, e.g., the Gini coefficient.

$$\large Gini(S) = 1 - \sum_j^kp_j^2$$

Nodes are split using the variable and split value that maximizes Gini gain.

$$Gini \; gain = Gini(S) - Gini(A,S)$$

with

$$Gini(A, S) = \sum \frac{n_i}{n}Gini(S_i)$$

Classification trees are pruned back such that every split has a purity gain of at least cp, with cp typically set to .01.

Minimize:

$$\large \begin{split} Loss = & Impurity\,+\\ &cp*(n\:terminal\:nodes)\\ \end{split}$$

Classification trees are pruned back such that every split has a purity gain of at least cp, with cp typically set to .01.

Minimize:

$$\large \begin{split} Loss = & Impurity\,+\\ &cp*(n\:terminal\:nodes)\\ \end{split}$$

Trees can also be used to perform regression tasks. Instead of impurity, regression trees attempt to minimize within-node variance (or maximize node homogeneity):

$$\large SSE = \sum_{i \in S_1}(y_i - \bar{y}_1)^2+\sum_{i \in S_2}(y_i - \bar{y}_2)^2$$ Algorithm

1 - Split nodes to maximize homogeneity gain.

2 - Repeat until pre-defined threshold (e.g., minsplit) splits are no longe possible.

3 - Prune tree to reasonable size.

Random Forest

In Random Forest, the criterion is modeled as the aggregate prediction of a large number of decision trees each based on different features.

Algorithm

1 - Repeat n times

      1 - Resample data

      2 - Grow non-pruned decision tree

            Each split consider only m features

2 - Average fitted values

Random forests make use of important machine learning elements, resampling and averaging that together are also referred to as bagging.

Random forests make use of important machine learning elements, resampling and averaging that together are also referred to as bagging.

All machine learning models are equipped with tuning parameters that control model complexity.

These tuning parameters can be identified using a validation set created from the traning data.

Logic

1 - Create separate test set.

2 - Fit model using various tuning parameters.

3 - Select tuning leading to best prediction on      validation set.

4 - Refit model to entire training set (training +      validation).

Resampling methods automatize and generalize model tuning.

Resampling methods automatize and generalize model tuning.

Resampling methods automatize and generalize model tuning.

k-fold cross validation for Ridge and Lasso

Goal

Use 10-fold cross-validation to identify optimal regularization parameters for a regression model.

Consider

$\alpha \in 0, .5, 1$ and $\lambda \in 1, 2, ..., 100$