Linear Regression with OLS and SGD in Python

Linear regression is one of the most widely used tools in data analysis. Many people run it in Excel, but Python can read the same spreadsheet, fit the model, compute useful metrics and plot the result in a few lines.

Ways to fit a linear model

We compare two methods from scikit-learn:

• Ordinary least squares (OLS), which finds the line minimising the sum of squared residuals. For a straight line this has an exact closed-form solution.
• Stochastic gradient descent (SGD), via SGDRegressor, which minimises a (regularised) squared-error loss iteratively.

How good is the fit?

• \(R^2\) score: the proportion of the variance in the dependent variable explained by the model. (sklearn.metrics.r2_score)
• Mean absolute error (MAE): the average absolute difference between predicted and actual values. (sklearn.metrics.mean_absolute_error)
• Mean squared error (MSE): the average squared difference, which is the loss that SGDRegressor minimises by default. (sklearn.metrics.mean_squared_error)
• Root mean squared error (RMSE): the square root of the MSE, in the same units as the dependent variable.
• Residual plots: for a good fit, the residuals scatter randomly around zero with no visible pattern.

Below we use \(R^2\), MSE and MAE.

World population data

We fit a linear trend to the world population by year. The data are stored in an Excel file and read into Python with pandas; the year is the independent variable and the world population the dependent variable.

OLS

We use \(80\%\) of the data to train the model, that is, to find \(m\) and \(c\) in \(y=mx+c\) that minimise the sum of squared residuals. The remaining \(20\%\) is held back to test it.

SGD

For SGD we first standardise the independent variable (the years), so that the optimisation is well conditioned. SGDRegressor then moves step by step towards the minimum of the loss, for at most \(1000\) iterations.

Results

Method	MSE	\(R^2\)	MAE
OLS (on the test set)	\(5.03 \times 10^{14}\)	\(0.99977\)	\(1.77 \times 10^{7}\)
SGD (on all data)	\(4.54 \times 10^{14}\)	\(0.99973\)	\(1.70 \times 10^{7}\)

Both models fit the data extremely well, and the two lines are almost indistinguishable (red: OLS; orange: SGD):

Note that the two rows are not directly comparable: in the code below the OLS model is scored on the held-out test set, while the SGD model is trained and scored on all the data. To compare them fairly, train both on the same training set and evaluate both on the same test set. For a one-variable linear model, OLS is also the natural choice, since its exact solution leaves nothing for SGD to improve on; SGD comes into its own for very large datasets and models.

Code

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, SGDRegressor
from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error
from sklearn.preprocessing import StandardScaler

# Read the data from the Excel file
df = pd.read_excel('world_population_data.xlsx')

# Convert the data to column vectors
years = df['Year'].to_numpy()[:, np.newaxis]
populations = df['World Population'].to_numpy()[:, np.newaxis]

# Split the data into training and testing sets
years_train, years_test, populations_train, populations_test = train_test_split(
    years, populations, test_size=0.2, random_state=42
)

# Ordinary least squares
regr = LinearRegression()
regr.fit(years_train, populations_train)
populations_pred_lr = regr.predict(years_test)

mse_lr = mean_squared_error(populations_test, populations_pred_lr)
r2_lr = r2_score(populations_test, populations_pred_lr)
mae_lr = mean_absolute_error(populations_test, populations_pred_lr)

# Standardise the years for SGDRegressor
scaler = StandardScaler()
years_scaled = scaler.fit_transform(years)

# Stochastic gradient descent
sgd_reg = SGDRegressor(max_iter=1000, tol=1e-3, penalty='l2', alpha=0.0001, random_state=42)
sgd_reg.fit(years_scaled, populations.ravel())
populations_pred_sgd = sgd_reg.predict(years_scaled)

# Metrics for SGD (computed on all the data, since this model was not split)
mse_sgd = mean_squared_error(populations, populations_pred_sgd)
r2_sgd = r2_score(populations, populations_pred_sgd)
mae_sgd = mean_absolute_error(populations, populations_pred_sgd)

print(f"Linear Regression - MSE: {mse_lr}, R²: {r2_lr}, MAE: {mae_lr}\n")
print(f"SGD Regressor - MSE: {mse_sgd}, R²: {r2_sgd}, MAE: {mae_sgd}")

# Plot the data and both regression lines
plt.figure(figsize=(12, 8))
plt.scatter(years_train, populations_train, color='blue', label='Training data')
plt.scatter(years_test, populations_test, color='green', label='Testing data')
plt.plot(years_test, populations_pred_lr, color='red', linewidth=2, label='Linear Regression line')
plt.plot(years, populations_pred_sgd, color='orange', linewidth=2, label='SGD Regression line')
plt.xlabel('Year')
plt.ylabel('World Population')
plt.title('Year vs World Population')
plt.legend()
plt.grid(True)
plt.show()

Changes on 25 October 2026

What was wrong. Nothing in the content.

What changed. The introduction and explanations are shorter and clearer. Previously titled “Linear Regression using OLS and SGD Methods in Python”.