Polynomial Regression

Polynomial Regression is an extension of Linear Regression that describes the connection between input features and target variables as an n-th degree polynomial. Unlike linear regression, which fits a straight line, polynomial regression can fit a curve to the data.

1. What is Polynomial Regression?

In Polynomial Regression, the relationship between the input variable $x$ and the output variable $y$ is modeled as an $n$-th degree polynomial. The general form of a polynomial regression model is:

$$\hat{y} = \theta_0 + \theta_1 x + \theta_2 x^2 + \theta_3 x^3 + \dots + \theta_n x^n$$

Where:

• $\hat{y}$ is the predicted value.
• $x$ is the input feature.
• $\theta_0, \theta_1, \theta_2, \dots, \theta_n$ are the parameters (coefficients) of the model.
• $n$ is the degree of the polynomial.

2. Why Use Polynomial Regression?

Polynomial Regression is used when the data shows a nonlinear relationship between the input features and the target variable. If a linear model fails to capture the pattern in the data, a polynomial model can provide a better fit by capturing the curvature in the data.

• Nonlinear Relationships: Polynomial regression is useful when the relationship between the input features and the target variable is nonlinear.
• Flexibility: The degree of the polynomial can be changed to regulate the model's flexibility. Higher degrees can fit more complex patterns, but they also raise the possibility of overfitting.
• Better Fit: In cases where linear regression fails to capture the underlying trend in the data, polynomial regression can provide a better fit by modeling the curvature of the data.

3. Working Process of Polynomial Regression

Step 1: Data Transformation

To use polynomial regression, the input feature x is converted into polynomial features. For example, if $x = [1, 2, 3]$, and we want to fit a quadratic (2nd degree) polynomial, the transformed features would be:

$$\text{Original } X: [1, 2, 3] \quad \text{Transformed } X: \left[\begin{array}{ccc} 1 & 1^2 & 1^3 \\ 2 & 2^2 & 2^3 \\ 3 & 3^2 & 3^3 \\ \end{array}\right] = \left[\begin{array}{ccc} 1 & 1 & 1 \\ 2 & 4 & 8 \\ 3 & 9 & 27 \\ \end{array}\right]$$

Step 2: Fit the Polynomial Model

After transforming the input features, you fit a linear regression model to the transformed features. The model will learn the parameters $\theta_0, \theta_1, \theta_2, \dots$ which minimize the error between the predicted and actual values.

Step 3: Make Predictions

Using the fitted model, you can make predictions for new data by applying the learned parameters to the polynomial features of the new input data.

Implementation of Polynomial Regression

import numpy as np

import matplotlib.pyplot as plt

from sklearn.preprocessing import PolynomialFeatures

from sklearn.linear_model import LinearRegression

from sklearn.pipeline import make_pipeline

# Sample data

X = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9]).reshape(-1, 1)

y = np.array([1, 4, 9, 16, 25, 36, 49, 64, 81])

# Transforming the data to include polynomial features

degree = 2  # You can change this to higher degrees to fit more complex curves

polynomial_features = PolynomialFeatures(degree=degree, include_bias=False)

# Create a pipeline that first transforms the data to polynomial features and then fits a linear model

model = make_pipeline(polynomial_features, LinearRegression())

# Fit the model

model.fit(X, y)

# Predicting

X_new = np.linspace(0, 10, 100).reshape(100, 1)

y_new = model.predict(X_new)

# Plotting the results

plt.figure(figsize=(8, 6))

plt.scatter(X, y, color='blue', label='Data Points')

plt.plot(X_new, y_new, color='red', linewidth=2, label='Polynomial Regression Line')

plt.xlabel('X', fontsize=14)

plt.ylabel('y', fontsize=14)

plt.title(f'Polynomial Regression (degree={degree})', fontsize=16)

plt.legend()

plt.grid(True)

plt.show()

Advantages:

1. Flexibility in Modeling Nonlinear Relationships:

   • Polynomial regression can model complex, nonlinear relationships between the input features and the target variable. This makes it a versatile tool for capturing trends that linear regression cannot.

2. Easy to Implement:

   • Polynomial regression is relatively straightforward to implement, especially using tools like scikit-learn. It builds on the principles of linear regression, making it accessible for those familiar with basic regression techniques.

3. Interpretability:

   • Although more complex than linear regression, polynomial regression still maintains a degree of interpretability, particularly for low-degree polynomials. You can understand the impact of each term in the polynomial equation.

4. Good Fit for Small Datasets:

   • Polynomial regression can be effective for small datasets where the relationship between variables is inherently nonlinear. It can provide a better fit than linear regression when the dataset is small and well-behaved.

Disadvantages:

1. Overfitting:

   • A major risk of polynomial regression is overfitting, especially when the degree of the polynomial is high. The model may fit the training data very well but fail to generalize to new, unseen data, leading to poor performance on test datasets.

2. Extrapolation Issues:

   • Polynomial regression models can behave unpredictably when making predictions outside the range of the training data. The curve can become extremely steep or oscillatory, leading to unrealistic predictions.

3. Complexity with High-Degree Polynomials:

   • As the degree of the polynomial increases, the model becomes increasingly complex, and it becomes harder to interpret the relationship between the features and the target variable. High-degree polynomials also require more computational resources.

4. Sensitive to Outliers:

   • Polynomial regression is sensitive to outliers. Since the model tries to minimize the error for all points, an outlier can significantly skew the polynomial curve, leading to a poor fit for the majority of the data.

5. Multicollinearity:

   • When using polynomial regression with multiple features, there can be a high degree of multicollinearity (correlation between the polynomial terms). This can make the model unstable and difficult to interpret.