Regression Algorithms

Ordinary Least Squares

OLS is the method with which linear regression is performed. The square of the difference from the mean is taken for every data point, and the summed loss function is to be minimized.

\[l = \sum_{i=1}^n (y_i - \bar{y})^2\]


Polynomial regression is a modification of linear regression where the existing features are mapped to a polynomial form. The problem is still a linear regression problem, but the input vector is now mapped to a higher dimensional vector which serves as a pseudo-input vector of sorts.

\[\textbf{x} = (x_0, x_1) \rightarrow \textbf{x'} = (x_0, x^2_0, x_1, x^2_1, x_0x_1)\]


Lasso Regression tries to reduce the ordinary least squares error similar to vanilla regression, but adds an extra term. The sum of the \(L_1\) norm for every data point multiplied by a hyperparameter \(\alpha\) is used. This reduces model complexity and prevents overfitting.

\[l = \sum_{i=1}^n (y_i - \tilde{y})^2 + \alpha \sum_{j=1}^p |w_j|\]


Ridge regression is similar to lasso regression, but the regularization term uses the \(L_2\) norm instead.

\[l = \sum_{i=1}^n (y_i - \tilde{y})^2 + \alpha \sum_{j=1}^p w^2_j\]


Stepwise regression or spline regression helps us fit a piece wise function to the data. It is usually used with linear models, but it can be generalized to higher degrees as well. The regression equation takes the form of

\[y = ax + b(x-\bar{x})H_{\alpha}+c\]

where \(H_{\alpha}\) is the shifted Heaviside step function, having its discontinuity at \(\alpha\).