Linear & Multilinear Regression

Linear regression is one of the most basic and commonly used statistical techniques in machine learning. The link between a dependent variable (target) and one or more independent variables (features) is modeled using this technique. The objective is to find the optimal linear equation that fits the data and can be used to forecast the dependent variable using the values of the independent variables.

Simple Linear Regression 

The one independent variable (feature) and the single dependent variable (target) in basic linear regression. A straight line is used to depict their relationship.

It involves one independent variable: $y={\beta }_0+{\beta }_1\times x+\mathrm{ϵ }$

• $y$ is the dependent variable (target).
• $x$ is the independent variable (feature).
• ${\beta }_0$ is the y-intercept (constant term).
• ${\beta }_1$ is the slope of the line (coefficient for the feature).
• $ϵ$ is the error term (residual).

Multiple Linear Regression

In multiple linear regression, there are numerous independent variables (features). A hyperplane in higher dimensions represents the relationship between the dependent variable and the independent variables.

It involves multiple independent variables: $y = \beta_0 + \beta_1 \times x_1 + \beta_2 \times x_2 + \dots + \beta_n \times x_n + \epsilon$

Working Process of Linear Regression

1. Hypothesis Representation:

   • The relationship between the dependent and independent variables is represented as a linear equation. The objective is to find the best-fitting line (in simple linear regression) or hyperplane (in multiple linear regression).

2. Cost Function (Mean Squared Error):

   • The cost function computes the difference between the expected and actual values. The most frequent cost function for linear regression is Mean Squared Error. (MSE): $J(\beta_0, \beta_1) = \frac{1}{2m} \sum_{i=1}^{m} \left( \hat{y}^{(i)} - y^{(i)} \right)^2$where $\hat{y}^{(i)}$ is the predicted value, $y^{(i)}$ is the actual value, and $\mathrm{m is\; the\; number\; of\; training\; examples.}$

3. Optimization (Gradient Descent):

   • The goal is to minimize the cost function by adjusting the parameters $\beta_0$ and $\beta_1$. Gradient Descent is a common optimization technique used to find the minimum of the cost function: ${\beta }_j:={\beta }_j-\alpha \frac{\mathrm{\partial }}{\mathrm{\partial }{\beta }_j}J({\beta }_0,{\beta }_1)\; where$$\alpha$ is the learning rate, and $\frac{\partial}{\partial \beta_j} J(\beta_0, \beta_1)$ is the partial derivative of the cost function for $\beta_j$.

4. Model Evaluation:

• The performance of the linear regression model can be evaluated using metrics such as R-squared ($R^2$), Mean Absolute Error (MAE), or Root Mean Squared Error (RMSE).

Implementation of Simple Linear Graph:

import numpy as np

import matplotlib.pyplot as plt

from sklearn.linear_model import LinearRegression

# Sample data

X = np.array([[1], [2], [3], [4], [5]])

y = np.array([1, 3, 2, 3, 5])

# Creating and training the model

model = LinearRegression()

model.fit(X, y)

# Predicting

y_pred = model.predict(X)

# Plotting the data points and the regression line

plt.figure(figsize=(8, 6))

plt.scatter(X, y, color='blue', label='Data Points', s=100)  # Larger dots for clarity

plt.plot(X, y_pred, color='red', linewidth=2, label='Regression Line')  # Thicker line for better visibility

plt.xlabel('X', fontsize=14)

plt.ylabel('y', fontsize=14)

plt.title('Simple Linear Regression', fontsize=16)

plt.legend()

plt.grid(True)  # Adding grid for better reference

plt.show()

Implementation of Multi-Linear Regression:

import numpy as np

import matplotlib.pyplot as plt

from mpl_toolkits.mplot3d import Axes3D

from sklearn.linear_model import LinearRegression

# Sample data

X = np.array([[1, 2], [2, 3], [3, 4], [4, 5], [5, 6]])

y = np.array([2, 3, 4, 5, 6])

# Creating and training the model

model = LinearRegression()

model.fit(X, y)

# Predicting

y_pred = model.predict(X)

# Plotting the data points and the regression plane

fig = plt.figure()

ax = fig.add_subplot(111, projection='3d')

ax.scatter(X[:, 0], X[:, 1], y, color='blue', label='Data Points')

# Creating the plane

x_surf, y_surf = np.meshgrid(np.linspace(X[:, 0].min(), X[:, 0].max(), 100),

                             np.linspace(X[:, 1].min(), X[:, 1].max(), 100))

z_surf = model.predict(np.c_[x_surf.ravel(), y_surf.ravel()]).reshape(x_surf.shape)

# Plotting the plane

ax.plot_surface(x_surf, y_surf, z_surf, color='red', alpha=0.5)

ax.set_xlabel('X1')

ax.set_ylabel('X2')

ax.set_zlabel('y')

ax.set_title('Multiple Linear Regression')

plt.show()