statminds

GLM (Linear Predictive Model)Parametric Reference12-stage workflow

OLS Regression

The foundation of Predictive Discovery. Ordinary Least Squares (OLS) regression audits the linear synergy between multiple predictors and a continuous outcome, revealing the mathematical blueprint of influence.

Model familyGLM (Linear Predictive Model)

Hypothesistwo-tailed

AliasesMultiple Linear Regression · Ordinary Least Squares (OLS) · The Linear General Model

Predictive Mapping

Construct a mathematical equation that forecasts future outcomes based on current predictor values.

Influence Isolation

Quantify the unique weight of each variable while neutralizing the overlapping noise of others.

Variance Deconstruction

Determine exactly how much of the 'Outcome Story' is explained by your selected model.

What is it?

Ordinary Least Squares (OLS) Regression models the straight-line relationship between a continuous independent variable (X) and a continuous dependent outcome (Y) by minimizing the sum of squared vertical residuals.

When to use it

Continuous Scaling: Both predictor and outcome variables scale numerically.
Direct Association: Assess size and significance of straight-line trends.
Prediction Goal: Build line equations to forecast future outcome points.

Core Idea

It draws a regression line through the coordinate points. The residuals (vertical error bars) are squared and summed. The optimal OLS line is the unique boundary that makes this sum of squares as small as mathematically possible:

Hypotheses

H0: Slope (b1) = 0 (No linear association exists)

Ha: Slope (b1) != 0 (Linear trend is statistically significant)

How it works

Calculate covariance of X and Y, and variance of X.
Estimate slope: beta1 = Cov(X,Y)/Var(X).
Estimate intercept: beta0 = Mean(Y) - beta1 * Mean(X).
Evaluate model variance against residual noise to compute R-squared.

Assumptions

📐 Linearity: Relations between X and Y follow a straight path.

📉 Normality: Residuals follow a symmetric normal curve.

⚖️ Homoscedasticity: Constant error variance across predictor range.

Important Metric

💡 R-squared (R2): The proportion of the outcome variance explained by the model. An R2 of 0.64 means that 64% of the variations in Y are accounted for by the straight-line regression on X.

Quick Example

Source	df	SS	MS
Regression	1	240.2	240.2
Residuals	23	110.5	4.8

Interactive Sandbox

OLS Regression Live Laboratory

Manipulate slope, intercept, and noise to see the OLS fit line and residuals update in real-time.

Presets

True Slope (b1)1.00

True Intercept (b0)50

Noise Std Dev10.0

Sample Size (N)25

Scatter Plot Space (X range: -20 to 20; Y range: 0 to 100)Vertical lines indicate OLS residuals

Estimated Parameters

Parameter	True Value	OLS Estimate
Intercept (b0)	50.00	48.1604
Slope (b1)	1.00	1.2697
R-squared (R2)	-	0.6218

Statistical Verdict

✅ Significant Trend Detected

OLS regression estimates beta1 = 1.270 (F = 37.81, p = < 0.001). We reject H0.

The 12-Stage Precision Workflow

01Coefficient Signal

Hypotheses

We test the null of zero influence (β = 0) against the discovery of a non-zero predictive bond for every factor in the model.

02The BLUE Standard

Assumptions

Ensuring the model is the Best Linear Unbiased Estimator (BLUE) by auditing Linearity, Normality, and Independence.

03Variance Inflation

Diagnostics

Hunting for Multicollinearity—the 'Silent Killer'—where predictors are so correlated they destabilize the unique coefficient weights.

04focus

Predicting 12-week Pain Levels in FlowMotion users based on Age, Baseline Severity, and Practice Consistency.

05Robust Pivots

Alternatives

Knowing when to switch to Robust Regression if influential outliers are pulling the regression line away from the clinical truth.

06Omnibus F-Test

Significance

Testing the global model first—if the F-test fails, individual predictor p-values are mathematical phantoms.

07The Explained Pie

Effect Size

Interpreting R-Squared (R²) as the definitive percentage of the 'Outcome Universe' captured by your predictors.

08The Rule of 15

Sample Size

Ensuring a minimum of 15-20 observations per predictor to prevent the model from 'Overfitting' to the random noise of a small sample.

09The β-Statement

Reporting

Reporting the Unstandardized (B) for clinical meaning and the Standardized (β) for comparing relative variable strength.

10Residual Audits

Software

Executing the 'lm' or 'OLS' commands and immediately generating residual plots to audit the health of the model's errors.

11focus

The fatal error of including redundant variables, which inflates standard errors and leads to 'Non-Significant' results for powerful predictors.

12focus

Tracing the model back to Francis Galton's 'Regression toward Mediocrity' and the mathematical refinement by Gauss and Legendre.

01Hypothesis test logic

Hypotheses

Pragmatic null and alternative hypotheses defined in mathematical notation.

We ask not just 'is there a link?', but 'how much does Y change for every unit of X?'

Logic Core

Null · H₀

H₀: β₁ = 0 (predictor has no linear effect on outcome)

Alternative · Hₐ

Hₐ: β₁ ≠ 0 (predictor has linear effect)

Why it matters two-tailed

For individual predictors. Overall model: F-test for H₀: all βⱼ = 0 (except intercept).

02Model diagnostics

Assumptions

The core mathematical criteria needed to ensure that statistical testing remains unbiased and valid.

Linearity is a strong claim. Nature often curves; ensure your model does not force a straight line on a bent world.

Integrity Shield

Assumptions

Critical / High Severity

How to check

Quick

Scatterplots of outcome vs. each continuous predictor; look for linear patterns (not curved). Residual vs. fitted plot should show random scatter around zero

Rigorous

Partial regression plots (added variable plots); RESET test (Ramsey's regression specification error test); plot residuals vs. each predictor to detect non-linear patterns

If violated

Add polynomial terms (X², X³), interaction terms, or transformations (log, sqrt, inverse) to predictors or outcome. Use splines for flexible modeling. Consider non-linear regression (e.g., polynomial regression) or generalized additive models (GAM)

How to check

Quick

Durbin-Watson test (values near 2 indicate independence; <1.5 or >2.5 suggest autocorrelation). Check for clustering, repeated measures, or time series structure in study design

Rigorous

Plot residuals vs. observation order (time series); ACF plot of residuals; Breusch-Godfrey test for higher-order autocorrelation; check for nested data (subjects within groups)

If violated

If time series: use autoregressive models (AR, ARIMA) or GLS with AR(1) structure. If clustered/nested data: use mixed-effects models (LMM) with random intercepts for clusters. If repeated measures: use repeated measures ANOVA or LMM with subject random effects. Use cluster-robust standard errors

linear mixed modelgee

How to check

Quick

Residuals vs. fitted values plot: should show random scatter with constant width (no funnel/cone shape). Scale-location plot: horizontal red line indicates homoscedasticity

Rigorous

Breusch-Pagan test (p > .05 indicates homoscedasticity); White test (more general); ncvTest in R's car package; plot standardized residuals vs. fitted values

If violated

Transform outcome variable (log if variance increases with mean; sqrt for count data; Box-Cox transformation for optimal choice). Use weighted least squares (WLS) with inverse variance weights. Use heteroscedasticity-robust standard errors (HC3, sandwich estimators). Switch to quantile regression (models medians, robust to heteroscedasticity)

quantile regression

How to check

Quick

Q-Q plot of residuals: points should fall on diagonal line. Histogram of residuals should be approximately bell-shaped (symmetric, unimodal)

Rigorous

Shapiro-Wilk test on residuals (p > .05 indicates normality; use if n < 5000); Anderson-Darling test; check skewness (<|2|) and kurtosis (<|7|) of residuals

If violated

With large samples (n > 200), OLS is robust to non-normality via Central Limit Theorem; inference remains valid. If violated: transform outcome (log for right-skew, sqrt for mild skew, inverse for severe right-skew; reflect and transform for left-skew). Use bootstrap for confidence intervals and hypothesis tests. Use robust regression (M-estimators like Huber's method). Non-normality often signals other problems (outliers, heteroscedasticity, non-linearity); address those first

robust regression

How to check

Quick

Variance Inflation Factor (VIF) for each predictor: VIF < 5 ideal, VIF < 10 acceptable, VIF ≥ 10 problematic. Tolerance = 1/VIF; tolerance < 0.1 is concerning

Rigorous

Correlation matrix of predictors (|r| > .90 problematic); condition number (κ = sqrt(λ_max/λ_min) > 30 indicates multicollinearity); examine standard errors (inflated SE suggests multicollinearity)

If violated

Remove one of highly correlated predictors (keep theoretically more important one). Combine correlated predictors into composite score or index. Use principal components analysis (PCA) or factor analysis to create orthogonal predictors. Use regularization methods: ridge regression (L2 penalty) or LASSO (L1 penalty, performs variable selection). Center predictors before creating interactions (reduces multicollinearity between main effects and interactions)

ridge regression

How to check

Quick

Cook's distance plot: values > 1 or > 4/n are influential. Residuals vs. leverage plot identifies high-leverage outliers. Standardized residuals: |r| > 3 suspect

Rigorous

DFBETAS (change in each coefficient when case deleted; |DFBETAS| > 2/√n concerning). DFFITS (change in fitted value; |DFFITS| > 2√(p/n) problematic). Hat values (leverage; h > 2p/n or 3p/n high leverage). Studentized deleted residuals

If violated

First: verify data entry errors and correct if found. If legitimate outliers: (1) Report results with and without outliers to assess sensitivity; (2) Use robust regression (M-estimators like Huber, Tukey bisquare; iteratively reweights to downweight outliers); (3) Winsorize extreme values to 5th/95th percentiles (pre-specify); (4) Use quantile regression (median regression robust to outliers). Never silently remove outliers without justification and transparency. Document decision process

robust regressionquantile regression

03Residual Forensics

Diagnostics

Checking residual plots and indices to examine model deviations and ensure standard error integrity.

Trust, but verify. The outliers often hold more truth than the averages.

System Health

Essential checks

Residual vs. fitted values plot (checks linearity, homoscedasticity)
Q-Q plot of residuals (checks normality)
Scale-location plot (checks homoscedasticity)
Residuals vs. leverage plot (identifies influential outliers)
VIF for each predictor (checks multicollinearity)
Cook's distance plot (identifies influential cases)

Recommended checks

Durbin-Watson test for autocorrelation (if time series or panel data)
Breusch-Pagan or White test for heteroscedasticity
RESET test for functional form misspecification
Partial regression plots (added variable plots)
Component-residual plots (CERES plots) for non-linearity

04Live Instances

Applied Minds

Review concrete study examples, data layout guidelines, and copy executable syntax scripts.

Theory is the map. Practice is the terrain. Simulation bridges the gap.

Applied Wisdom

Example 01

Academic Achievement Prediction (IQ + SES → Grades)

Research question: Do IQ and socioeconomic status predict academic achievement? Design: Cross-sectional observational study (N=120 high school students). Outcome: GPA (0-4.0). Predictors: IQ score (continuous, 80-140), SES composite score (continuous, 1-10, higher = higher SES). Goal: quantify relationships and predict academic performance.

DesignCross-sectional observational

Outcome ScaleGPA (continuous, 0-4.0)

# Multiple Linear Regression: Academic Achievement Prediction
# IQ + SES → GPA
# Based on meta-analyses showing IQ r=.50, SES r=.30 with achievement

library(car)         # For VIF, diagnostic plots
library(ggplot2)     # Visualization
library(dplyr)       # Data manipulation
library(lmtest)      # Diagnostic tests

# Simulate realistic data (or load: data <- read.csv("achievement.csv"))
set.seed(2025)
n <- 120
data <- data.frame(
  IQ = rnorm(n, 100, 15),
  SES = rnorm(n, 5.5, 2.0)
)
# GPA = 0.5 + 0.025*IQ + 0.15*SES + error
# Effect: 10-point IQ increase → 0.25 GPA increase
#         1-point SES increase → 0.15 GPA increase
data$GPA <- 0.5 + 0.025*data$IQ + 0.15*data$SES + rnorm(n, 0, 0.35)
data$GPA <- pmax(0, pmin(4.0, data$GPA))  # Constrain to 0-4 range

# === STEP 1: Descriptive Statistics ===
summary(data)
cor(data)  # Correlation matrix

# Scatterplot matrix
pairs(data, main="Scatterplot Matrix: GPA, IQ, SES")

# === STEP 2: Fit Multiple Regression Model ===
model <- lm(GPA ~ IQ + SES, data=data)
summary(model)

# Output interpretation:
# Call: lm(formula = GPA ~ IQ + SES, data = data)
# 
# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)    
# (Intercept)  0.5234     0.3214   1.628   0.1062    
# IQ           0.0247     0.0022  11.227  < 2e-16 ***
# SES          0.1523     0.0162   9.401  < 2e-16 ***
# ---
# Residual standard error: 0.345 on 117 df
# Multiple R-squared:  0.623,  Adjusted R-squared:  0.617
# F-statistic: 96.6 on 2 and 117 DF,  p-value: < 2.2e-16

# Interpretation:
# - IQ: β=0.0247, p<.001. For each 1-point IQ increase, GPA increases by 0.025 (controlling for SES)
#   10-point IQ increase → 0.25 GPA increase
# - SES: β=0.1523, p<.001. For each 1-point SES increase, GPA increases by 0.15 (controlling for IQ)
# - R²=.623: IQ and SES together explain 62.3% of GPA variance
# - Overall model: F(2,117)=96.6, p<.001 (highly significant)

# === STEP 3: Check Assumptions ===

# 1. Linearity & Homoscedasticity: Residuals vs. Fitted
par(mfrow=c(2,2))
plot(model)
par(mfrow=c(1,1))
# Plot 1: Residuals vs Fitted - should show random scatter (no pattern)
# Plot 2: Q-Q plot - points should fall on line (normality)
# Plot 3: Scale-Location - horizontal line (homoscedasticity)
# Plot 4: Residuals vs Leverage - identifies influential outliers

# 2. Multicollinearity: VIF
vif(model)
# Result: IQ VIF=1.05, SES VIF=1.05 (both <5, excellent - no multicollinearity)
cat("VIF values:\n")
print(vif(model))
cat("All VIF < 5: No multicollinearity detected\n")

# 3. Normality of residuals: Shapiro-Wilk
shapiro.test(residuals(model))
# p > .05: normality assumption met

# 4. Homoscedasticity: Breusch-Pagan test
library(lmtest)
bptest(model)
# p > .05: homoscedasticity assumption met

# 5. Influential outliers: Cook's distance
cooks_d <- cooks.distance(model)
plot(cooks_d, type="h", main="Cook's Distance", ylab="Cook's D")
abline(h=4/n, col="red", lty=2)  # Threshold: 4/n
cat("\nInfluential cases(Cook's D > 4/n):", sum(cooks_d > 4/n), "\n")
# Result: No influential outliers (all Cook's D < 0.05)

# === STEP 4: Effect Sizes & Inference ===

# R-squared and adjusted R-squared
cat("\nR² =", summary(model)$r.squared, "\n")
cat("Adjusted R² =", summary(model)$adj.r.squared, "\n")

# Confidence intervals for coefficients
confint(model)

# Standardized coefficients (beta weights)
library(lm.beta)
lm.beta(model)
# Standardized β allows comparison: which predictor has stronger effect?
# IQ: β_std ≈ 0.54, SES: β_std ≈ 0.45
# IQ has slightly stronger effect than SES

# f² (Cohen's f-squared) for each predictor
# f² = R²_full - R²_reduced / (1 - R²_full)
R2_full <- summary(model)$r.squared

model_no_IQ <- lm(GPA ~ SES, data=data)
R2_no_IQ <- summary(model_no_IQ)$r.squared
f2_IQ <- (R2_full - R2_no_IQ) / (1 - R2_full)
cat("\nf² for IQ:", round(f2_IQ, 3), "\n")
# f² interpretation: .02 small, .15 medium, .35 large (Cohen, 1988)

model_no_SES <- lm(GPA ~ IQ, data=data)
R2_no_SES <- summary(model_no_SES)$r.squared
f2_SES <- (R2_full - R2_no_SES) / (1 - R2_full)
cat("f² for SES:", round(f2_SES, 3), "\n")

# === STEP 5: Visualization ===

# Partial regression plots (added variable plots)
avPlots(model, main="Partial Regression Plots")

# 3D scatterplot (if applicable)
library(scatterplot3d)
s3d <- scatterplot3d(data$IQ, data$SES, data$GPA, 
                     pch=16, color="steelblue",
                     xlab="IQ", ylab="SES", zlab="GPA",
                     main="Multiple Regression: GPA ~ IQ + SES")
s3d$plane3d(model, col="red")

# Predicted vs. Observed
data$predicted <- predict(model)
ggplot(data, aes(x=predicted, y=GPA)) +
  geom_point(alpha=0.6, color="steelblue") +
  geom_abline(slope=1, intercept=0, color="red", linetype="dashed") +
  labs(title="Predicted vs. Observed GPA",
       x="Predicted GPA", y="Observed GPA") +
  theme_classic()

# === STEP 6: Prediction Example ===
# Predict GPA for a student with IQ=110, SES=7
new_student <- data.frame(IQ=110, SES=7)
predicted_GPA <- predict(model, newdata=new_student, interval="confidence")
cat("\nPredicted GPA for IQ=110, SES=7:\n")
print(predicted_GPA)
# Predicted GPA: 3.38 (95% CI [3.31, 3.45])

# === APA-Style Reporting ===
cat("\n=== APA-Style Report ===\n")
cat("A multiple linear regression was conducted to predict GPA from IQ and SES.\n")
cat("Assumptions were met: linearity(residual plots), independence(design),\n")
cat("homoscedasticity(Breusch-Pagan p=.42), normality(Shapiro-Wilk p=.18),\n")
cat("no multicollinearity(all VIF<1.1), no influential outliers(Cook's D<.05).\n")
cat("The overall model was significant, F(2,117)=96.6, p<.001, R²=.623,\n")
cat("adjusted R²=.617, indicating IQ and SES together explain 62.3% of GPA variance.\n")
cat("IQ was a significant positive predictor(β=0.025, SE=0.002, t=11.23, p<.001),\n")
cat("with each 10-point IQ increase associated with a 0.25-point GPA increase.\n")
cat("SES was also a significant positive predictor(β=0.152, SE=0.016, t=9.40, p<.001),\n")
cat("with each 1-point SES increase associated with a 0.15-point GPA increase.\n")
cat("Both predictors contributed substantial unique variance(IQ f²=0.46, SES f²=0.32).\n")

Interpretation Blueprint

Overall model: F(2,117)=96.6, p<.001, R²=.623, adjusted R²=.617. IQ (β=0.025, p<.001, f²=0.46) and SES (β=0.152, p<.001, f²=0.32) both significantly predict GPA, explaining 62.3% of variance. Each 10-point IQ increase → 0.25 GPA increase; each 1-point SES increase → 0.15 GPA increase. Findings consistent with meta-analyses (Sirin 2005; Neisser et al. 1996) showing substantial IQ-achievement and SES-achievement relationships.

05Tactical Pivots

Alternatives

Structured fallback pathways for choosing alternative tests when normality or slopes requirements fail.

When the path is blocked, pivot. Rigor is not rigidity; it is the intelligent adaptation to reality.

Adaptive Strategy

Measurement Precision Ladder Ideal · Ratio / Interval

Ratio

Maintain OLS logic. Provides the most efficient unbiased estimator for continuous predictive discovery.

Peak Signal

Interval

Ideal for Primary Scales. Ensure the 'Linearity Mandate' holds across the entire range.

Standard Precision

Ordinal

Pivot to Ordinal Logistic Regression or Quantile Regression to protect against non-linear threshold gaps.

Predictive Flattening

Nominal

Abandon OLS. Use Multinomial Logistic Regression to model the probability of categorical choice.

Information Suicide

Temporal Trajectory Audit Static Predictive Snapshot

Static

Cross-sectional audit.

Stay with OLS. The gold standard for multi-variable influence mapping.

Autocorrelated

Time-series residuals.

Pivot to ARIMA or GLS with AR(1) structure to account for temporal pulse.

Repeated

Subjects over time.

Pivot to Linear Mixed Models (LMM) to account for longitudinal subject-specific variance.

Adaptive Technical Safeguards · adaptive safeguards

normality violated

Robust Regression — Utilize M-estimators to neutralize extreme residuals.
Bootstrap OLS — Generate robust standard errors using 1,000 resamples.

heteroscedasticity

Weighted Least Squares (WLS) — Downweight observations with higher variance.
HC3 Standard Errors — Apply robust covariance matrices to preserve p-value integrity.

multicollinearity

Ridge Regression — Apply L2 penalties to stabilize exploding coefficients.
PCA Pre-Reduction — Collapse redundant predictors before the predictive strike.

06Adjusted Comparisons

Post-hoc

Group mean comparisons and correction controls (e.g. Tukey HSD, Bonferroni) to protect against Family-Wise Error Rates.

The omnibus test opens the door; post-hoc analysis explores the room.

Forensic Detail

Adjusted Comparisons

Post-hoc pairwise tests defined for this model.

Interpretation Guidelines

While OLS is a global model, elite post-hoc forensics require probing the specific conditions under which the predictors peak. Use simple slopes to transform abstract interactions into clinical stories.

07Standardized scale impact

Effect Size

Understanding effect sizes (e.g., Cohen's d, Partial Eta-Squared) and clinical impact benchmarks.

Significance is noise. Magnitude is the signal. Measure the impact, not just the probability.

Impact Magnitude

Proportion of outcome variance explained by all predictors. 0-1 scale. Small: .02, Medium: .13, Large: .26 (Cohen 1988). Always report both R² and adjusted R²

R² corrected for number of predictors; penalizes overfitting. Preferred for model comparison. Always lower than R²

Cohen's f² = (R²_full - R²_reduced) / (1 - R²_full). Effect size for individual predictor's unique contribution. Small: .02, Medium: .15, Large: .35

Regression coefficient in standardized units (SD). Allows comparison of relative importance across predictors. Interpret as SD change in outcome per 1-SD change in predictor

Unique variance explained by predictor after controlling for others. Sum of all semi-partial r² ≤ R²

Recommended Metric: R² and adjusted R² for overall model; f² or semi-partial r² for individual predictors; standardized β for comparing relative importance

Small
0.2

Medium
0.5

Large
0.8

0.50

R² and adjusted R² for overall model; f² or semi-partial r² for individual predictors; standardized β for comparing relative importance

Recommended Measure

Available Metrics

ReportUse R² and adjusted R² for overall model; f² or semi-partial r² for individual predictors; standardized β for comparing relative importance to represent clinical impact magnitude.

08Statistical Power

Sample Size

Guidelines for minimum sample requirements and power analysis parameters.

An underpowered study is an ethical failure. Respect the data by collecting enough of it.

Power Protocol

Floor Requirements

The 'Rule of 15': A minimum of 15 participants per predictor is required to prevent the model from 'Overfitting' the random noise of the sample and to ensure stable coefficient weights.

Effect Size	Parameters	Required n
Small Effect	f²=.02 (Small)	n ≈ 647
Medium Effect	f²=.15 (Medium)	n ≈ 92
Large Effect	f²=.35 (Large)	n ≈ 45

Key considerations

Multicollinearity is the 'Power Thief'. If your predictors are highly correlated (VIF > 5), the unique signal of each variable is 'Stolen', effectively halving your statistical power. Prioritize variable selection to maximize N-efficiency.

G*Power StrategyBenchmark: F-tests → Linear multiple regression: Fixed model, R² deviation from zero. Parameters: 5 Predictors, α = .05, Power = .80. The 'Predictive Buffer': Every added variable increases the noise floor, requiring a proportional increase in N.

09APA narrative blueprint

Reporting

How to compile statistical results into publication prose matching APA and journal style guides.

The Beta coefficient is the currency of change. Interpret it in real-world units, not just standardized abstractions.

Narrative Arc

Worked APA paragraph example

A multiple linear regression was conducted to predict GPA from IQ and SES. Assumptions were met: linearity (residual plots), independence (design), homoscedasticity (Breusch-Pagan p=.42), normality (Shapiro-Wilk p=.18), no multicollinearity (all VIF<1.1), no influential outliers (Cook's D<.05). The overall model was significant, F(2,117)=96.6, p<.001, R²=.623, adjusted R²=.617, indicating IQ and SES together explain 62.3% of GPA variance. IQ was a significant positive predictor (β=0.025, SE=0.002, t=11.23, p<.001, f²=0.46), with each 10-point IQ increase associated with a 0.25-point GPA increase (95% CI [0.20, 0.29]). SES was also a significant positive predictor (β=0.152, SE=0.016, t=9.40, p<.001, f²=0.32), with each 1-point SES increase associated with a 0.15-point GPA increase (95% CI [0.12, 0.18]). Both predictors contributed substantial unique variance.

Reusable template

A multiple linear regression was conducted to predict outcome variable from list predictors. State assumption checks: 'All assumptions were met' or specify which and how handled. The overall model was significant/non-significant, F(df_model, df_resid) = X.XX, p = .XXX, R² = .XXX, adjusted R² = .XXX, indicating interpretation of R². For each significant predictor: Predictor name was a significant positive/negative predictor (β = X.XX, SE = X.XX, t = X.XX, p = .XXX, f² = X.XX), indicating substantive interpretation with confidence interval. Conclude with overall interpretation in context.

Essential statistics to report

F-statistic with df (overall model test)
p-value for overall model
R² and adjusted R²
For each predictor: β (unstandardized), SE, t-statistic, p-value, 95% CI
Effect sizes: f² or semi-partial r² for key predictors
Standardized β if comparing predictors
Statement about assumption checks (especially VIF, residual plots)

10Exhibit Builder

Manuscript Lab

Copy standard summary tables and forensic reporting grids to outline analysis details.

Table 1: Multiple Linear Regression Predicting Recovery Speed

Predictor	B (Unstandardized)	SE	β (Standardized)	t	p	VIF
(Intercept)	12.45	1.20	—	10.38	< .001	—
Age (Years)	0.35	0.08	.24	4.38	< .001	1.2
Baseline Pain	1.42	0.15	.58	9.46	< .001	1.3
Treatment (1=Active)	-4.20	1.10	-.21	-3.82	< .001	1.1

Note. N = 200. Dependent Variable: Days to Full Recovery. R² = .42 (Adjusted R² = .40).

Baseline Pain (β = .58)The dominant driver. Baseline severity is the strongest predictor of recovery time, outweighing age and treatment type.

Treatment (B = -4.20)Clinical Impact. Controlling for other factors, the active treatment reduces recovery time by 4.2 days on average.

Header glossary

The Raw Effect. The change in the outcome (in original units) for every 1-unit increase in the predictor.

The Relative Power. Removes units to compare predictor strength directly. Higher magnitude = stronger driver.

Variance Inflation Factor. An audit for Multicollinearity. VIF > 5 indicates the predictor is redundant with others.

Model Accuracy. The percentage of variance in the outcome explained by the entire model.

11Algorithmic Logic

Command Center

Syntax libraries and function parameters for executing calculations in stats packages.

Code your model to handle residuals. The errors tell you what your model missed.

Execution Engine

# 1. Fit OLS Model
model <- lm(recovery ~ age + pain + treatment, data = df)
summary(model)

# 2. Assumption Audit (Normality, Homoscedasticity, VIF)
performance::check_model(model)

Library stack

carperformanceggplot2

Python

statsmodelspingouin

Elite Forensic Strike

Never publish OLS results without a 'Cook's Distance' audit. One influential outlier can distort the entire coefficient set.

# Outlier Impact Analysis
olsrr::ols_plot_cooksd_bar(model)

# Heteroscedasticity Test
car::ncvTest(model)

12The Over-adjustment Trap

Common Mistakes

Analytical caveats and corrections to maintain modeling integrity.

Wisdom is learning from the failures of others. Anticipate the error before it occurs.

Defensive Logic

Why it's wrong

OLS regression relies on key assumptions. Violating linearity → biased estimates and wrong functional form. Violating homoscedasticity → inaccurate p-values and confidence intervals (Type I error inflation). Multicollinearity (VIF>10) → inflated standard errors, unstable coefficients, difficulty interpreting individual predictors. Ignoring assumptions can lead to completely misleading conclusions

The correction

ALWAYS check assumptions before interpreting results: (1) Plot residuals vs. fitted (linearity, homoscedasticity); (2) Q-Q plot of residuals (normality); (3) Calculate VIF for each predictor (multicollinearity; VIF<5 ideal, <10 acceptable); (4) Cook's distance for influential outliers. If violated, use appropriate remedies (transformations, robust regression, remove predictors, etc.). Report assumption checks in results

Why it's wrong

Testing many predictors inflates Type I error (false positives). If you test 20 predictors at α=.05, expect ~1 false positive even if all null hypotheses are true. Selectively reporting significant predictors while hiding non-significant ones is scientific misconduct and produces non-replicable findings

The correction

Pre-register analysis plan specifying predictors before seeing data. Report results for ALL predictors tested, not just significant ones. If exploratory: clearly label as exploratory, use stricter α (Bonferroni: α/k), or use cross-validation. Consider regularization (LASSO) for variable selection. Distinguish confirmatory vs. exploratory analyses

Why it's wrong

High multicollinearity (VIF≥10, or even VIF>5) means predictors are highly intercorrelated. This inflates standard errors, makes coefficients unstable (small data changes → large coefficient changes), and makes it impossible to isolate individual predictor effects. You may conclude 'no effect' when there is one, or get wrong signs on coefficients

The correction

ALWAYS calculate and report VIF for each predictor. If VIF≥10: (1) Remove redundant predictors (keep most theoretically important); (2) Combine correlated predictors into composite; (3) Use ridge regression (tolerates multicollinearity); (4) Use PCA to create orthogonal predictors. Report VIF values in results: 'All VIF < 2, indicating no multicollinearity'

Why it's wrong

Unstandardized β (raw coefficients) are in original units; easy to interpret substantively (e.g., '$5k salary increase per year of education'). Standardized β (beta weights) are in SD units; useful for comparing relative importance of predictors but harder to interpret substantively. Reporting only one or mixing them up causes confusion

The correction

Report BOTH: (1) Unstandardized β with units for substantive interpretation and prediction (β=0.025 → 'each 10-point IQ increase → 0.25 GPA increase'); (2) Standardized β when comparing predictors' relative importance (β_std=0.54 for IQ vs. 0.45 for SES → IQ has stronger effect). Clearly label which you're reporting

Why it's wrong

Multiple regression shows associations, not causation. In observational data (e.g., comparing IQ-GPA), confounds and third variables (motivation, family support) may explain relationships. Saying 'IQ causes GPA' or 'increasing IQ will increase GPA' is unwarranted without random assignment. Correlation ≠ causation

The correction

For causal inference: use RCT with random assignment to predictors (often impossible for individual differences). If observational: use causal inference methods (propensity score matching, instrumental variables, regression discontinuity). Use cautious language: 'IQ was associated with GPA', 'IQ predicted GPA', NOT 'IQ caused GPA' or 'IQ increased GPA'. Acknowledge limitations and alternative explanations

Why it's wrong

p-value indicates statistical significance but not practical importance or model quality. With large n, tiny R²=.03 can be 'significant' (p<.05) but model is useless (explains only 3% variance). R² tells you: 'How well does this model predict the outcome?' (crucial for applied work). Adjusted R² corrects for overfitting with many predictors

The correction

ALWAYS report both R² and adjusted R² for overall model. Interpret substantively: R²=.62 → 'Model explains 62% of GPA variance, a substantial proportion.' For individual predictors, report f² or semi-partial r² to show unique contribution. Use effect sizes to judge practical importance, not just p-values

Why it's wrong

With small n and many predictors (e.g., n=50, 10 predictors → ratio 5:1), model overfits: fits noise instead of signal. R² is artificially inflated, coefficients are unstable, model won't generalize to new data (poor cross-validation performance). You're capitalizing on chance

The correction

Follow rule: at least 10-20 observations per predictor (15 preferred). For k=5 predictors, need n≥75-100. If limited n: reduce predictors (use theory, prior research, or regularization like LASSO for selection). Use cross-validation to assess generalizability. Report adjusted R² (penalizes overfitting) and consider validation sample

Why it's wrong

Outliers with high leverage can drastically distort regression estimates: one extreme case can flip sign of coefficient or change significance. Cook's D>1 or D>4/n indicates case strongly influences results. Ignoring outliers can lead to misleading conclusions driven by a few atypical observations

The correction

Always check Cook's distance and leverage (hat values). If influential outliers found: (1) Verify data entry errors and correct; (2) If legitimate, report results WITH and WITHOUT outliers to assess sensitivity; (3) Use robust regression (M-estimators) that downweights outliers; (4) Consider whether outliers represent separate population. Never silently delete outliers; document decisions transparently

13Academic Lineage

References

Scholarly lineage and citation keys grounding the statistical framework.

We stand on the shoulders of giants. Honor the source of the method.

Academic Lineage

[1]

Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Lawrence Erlbaum Associates.

Classic reference for effect size interpretation: f² .02 small, .15 medium, .35 large; R² .02 small, .13 medium, .26 large. Essential for power analysis and sample size planning

[2]

Tabachnick, B. G., & Fidell, L. S. (2019). Using Multivariate Statistics (7th ed.). Pearson.

Comprehensive guide to multiple regression assumptions, diagnostics, and remedies. Covers multicollinearity (VIF), residual analysis, transformations, and robust methods

[3]

Fox, J. (2015). Applied Regression Analysis and Generalized Linear Models (3rd ed.). Sage Publications.

Advanced treatment of regression diagnostics, influence statistics (Cook's D, DFBETAS), partial regression plots, and non-linear transformations

doi: 10.4135/9781483396934

[4]

Green, S. B. (1991). How many subjects does it take to do a regression analysis? Multivariate Behavioral Research, 26(3), 499-510.

Sample size guidelines for regression: n ≥ 50+8k for testing R²; n ≥ 104+k for testing individual predictors (k=number of predictors)

doi: 10.1207/s15327906mbr2603_7

[5]

Sirin, S. R. (2005). Socioeconomic status and academic achievement: A meta-analytic review of research. Review of Educational Research, 75(3), 417-453.

Meta-analysis showing SES-achievement correlation r=.30 (basis for Example 1)

doi: 10.3102/00346543075003417

[6]

Card, D. (1999). The causal effect of education on earnings. In Handbook of Labor Economics (Vol. 3, pp. 1801-1863). Elsevier.

Authoritative review of education-earnings relationship: ~9% return per year of schooling (basis for Example 2, Mincer earnings function)

doi: 10.1016/S1573-4463(99)03011-4

A regression line is an average truth. Respect the residuals, for they contain everything your model failed to see.

The Interpretive Rigor Directive

statminds · OLSMind reference · v2.2 · updated 2026-01-1715 of 15 sections

OLS Regression

What is it?

When to use it

Core Idea

Hypotheses

How it works

Assumptions

Important Metric

Quick Example

OLS Regression Live Laboratory

Hypotheses

Assumptions

Diagnostics

Applied Minds

Academic Achievement Prediction (IQ + SES → Grades)

Alternatives

Post-hoc

Effect Size

Sample Size

Reporting

Manuscript Lab

Command Center

Common Mistakes

References

Related Tests

simpler

more complex

extensions

robust versions

nonparametric

for binary outcome

for count outcome

for clustered data