Statistical Analysis

Statistical Analysis

Overview

Statistical analysis is a systematic process for testing hypotheses and quantifying relationships. Conduct hypothesis tests (t-test, ANOVA, chi-square), regression, correlation, and Bayesian analyses with assumption checks and APA reporting. Apply this skill for academic research.

When to Use This Skill

This skill should be used when:

• Conducting statistical hypothesis tests (t-tests, ANOVA, chi-square)
• Performing regression or correlation analyses
• Running Bayesian statistical analyses
• Checking statistical assumptions and diagnostics
• Calculating effect sizes and conducting power analyses
• Reporting statistical results in APA format
• Analyzing experimental or observational data for research

Core Capabilities

1. Test Selection and Planning

• Choose appropriate statistical tests based on research questions and data characteristics
• Conduct a priori power analyses to determine required sample sizes
• Plan analysis strategies including multiple comparison corrections

2. Assumption Checking

• Automatically verify all relevant assumptions before running tests
• Provide diagnostic visualizations (Q-Q plots, residual plots, box plots)
• Recommend remedial actions when assumptions are violated

3. Statistical Testing

• Hypothesis testing: t-tests, ANOVA, chi-square, non-parametric alternatives
• Regression: linear, multiple, logistic, with diagnostics
• Correlations: Pearson, Spearman, with confidence intervals
• Bayesian alternatives: Bayesian t-tests, ANOVA, regression with Bayes Factors

4. Effect Sizes and Interpretation

• Calculate and interpret appropriate effect sizes for all analyses
• Provide confidence intervals for effect estimates
• Distinguish statistical from practical significance

5. Professional Reporting

• Generate APA-style statistical reports
• Create publication-ready figures and tables
• Provide complete interpretation with all required statistics

Workflow Decision Tree

Use this decision tree to determine your analysis path:

START
│
├─ Need to SELECT a statistical test?
│  └─ YES → See "Test Selection Guide"
│  └─ NO → Continue
│
├─ Ready to check ASSUMPTIONS?
│  └─ YES → See "Assumption Checking"
│  └─ NO → Continue
│
├─ Ready to run ANALYSIS?
│  └─ YES → See "Running Statistical Tests"
│  └─ NO → Continue
│
└─ Need to REPORT results?
   └─ YES → See "Reporting Results"

Test Selection Guide

Quick Reference: Choosing the Right Test

Use references/test_selection_guide.md for comprehensive guidance. Quick reference:

Comparing Two Groups:

• Independent, continuous, normal → Independent t-test
• Independent, continuous, non-normal → Mann-Whitney U test
• Paired, continuous, normal → Paired t-test
• Paired, continuous, non-normal → Wilcoxon signed-rank test
• Binary outcome → Chi-square or Fisher's exact test

Comparing 3+ Groups:

• Independent, continuous, normal → One-way ANOVA
• Independent, continuous, non-normal → Kruskal-Wallis test
• Paired, continuous, normal → Repeated measures ANOVA
• Paired, continuous, non-normal → Friedman test

Relationships:

• Two continuous variables → Pearson (normal) or Spearman correlation (non-normal)
• Continuous outcome with predictor(s) → Linear regression
• Binary outcome with predictor(s) → Logistic regression

Bayesian Alternatives: All tests have Bayesian versions that provide:

• Direct probability statements about hypotheses
• Bayes Factors quantifying evidence
• Ability to support null hypothesis
• See references/bayesian_statistics.md

Assumption Checking

Systematic Assumption Verification

ALWAYS check assumptions before interpreting test results.

Use the provided scripts/assumption_checks.py module for automated checking:

from scripts.assumption_checks import comprehensive_assumption_check

# Comprehensive check with visualizations
results = comprehensive_assumption_check(
    data=df,
    value_col='score',
    group_col='group',  # Optional: for group comparisons
    alpha=0.05
)

This performs:

1. Outlier detection (IQR and z-score methods)
2. Normality testing (Shapiro-Wilk test + Q-Q plots)
3. Homogeneity of variance (Levene's test + box plots)
4. Interpretation and recommendations

Individual Assumption Checks

For targeted checks, use individual functions:

from scripts.assumption_checks import (
    check_normality,
    check_normality_per_group,
    check_homogeneity_of_variance,
    check_linearity,
    detect_outliers
)

# Example: Check normality with visualization
result = check_normality(
    data=df['score'],
    name='Test Score',
    alpha=0.05,
    plot=True
)
print(result['interpretation'])
print(result['recommendation'])

What to Do When Assumptions Are Violated

Normality violated:

• Mild violation + n > 30 per group → Proceed with parametric test (robust)
• Moderate violation → Use non-parametric alternative
• Severe violation → Transform data or use non-parametric test

Homogeneity of variance violated:

• For t-test → Use Welch's t-test
• For ANOVA → Use Welch's ANOVA or Brown-Forsythe ANOVA
• For regression → Use robust standard errors or weighted least squares

Linearity violated (regression):

• Add polynomial terms
• Transform variables
• Use non-linear models or GAM

See references/assumptions_and_diagnostics.md for comprehensive guidance.

Running Statistical Tests

Python Libraries

Primary libraries for statistical analysis:

• scipy.stats: Core statistical tests
• statsmodels: Advanced regression and diagnostics
• pingouin: User-friendly statistical testing with effect sizes
• pymc: Bayesian statistical modeling
• arviz: Bayesian visualization and diagnostics

Example Analyses

T-Test with Complete Reporting

import pingouin as pg
import numpy as np

# Run independent t-test
result = pg.ttest(group_a, group_b, correction='auto')

# Extract results
t_stat = result['T'].values[0]
df = result['dof'].values[0]
p_value = result['p-val'].values[0]
cohens_d = result['cohen-d'].values[0]
ci_lower = result['CI95%'].values[0][0]
ci_upper = result['CI95%'].values[0][1]

# Report
print(f"t({df:.0f}) = {t_stat:.2f}, p = {p_value:.3f}")
print(f"Cohen's d = {cohens_d:.2f}, 95% CI [{ci_lower:.2f}, {ci_upper:.2f}]")

ANOVA with Post-Hoc Tests

import pingouin as pg

# One-way ANOVA
aov = pg.anova(dv='score', between='group', data=df, detailed=True)
print(aov)

# If significant, conduct post-hoc tests
if aov['p-unc'].values[0] < 0.05:
    posthoc = pg.pairwise_tukey(dv='score', between='group', data=df)
    print(posthoc)

# Effect size
eta_squared = aov['np2'].values[0]  # Partial eta-squared
print(f"Partial η² = {eta_squared:.3f}")

Linear Regression with Diagnostics

import statsmodels.api as sm
from statsmodels.stats.outliers_influence import variance_inflation_factor

# Fit model
X = sm.add_constant(X_predictors)  # Add intercept
model = sm.OLS(y, X).fit()

# Summary
print(model.summary())

# Check multicollinearity (VIF)
vif_data = pd.DataFrame()
vif_data["Variable"] = X.columns
vif_data["VIF"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]
print(vif_data)

# Check assumptions
residuals = model.resid
fitted = model.fittedvalues

# Residual plots
import matplotlib.pyplot as plt
fig, axes = plt.subplots(2, 2, figsize=(12, 10))

# Residuals vs fitted
axes[0, 0].scatter(fitted, residuals, alpha=0.6)
axes[0, 0].axhline(y=0, color='r', linestyle='--')
axes[0, 0].set_xlabel('Fitted values')
axes[0, 0].set_ylabel('Residuals')
axes[0, 0].set_title('Residuals vs Fitted')

# Q-Q plot
from scipy import stats
stats.probplot(residuals, dist="norm", plot=axes[0, 1])
axes[0, 1].set_title('Normal Q-Q')

# Scale-Location
axes[1, 0].scatter(fitted, np.sqrt(np.abs(residuals / residuals.std())), alpha=0.6)
axes[1, 0].set_xlabel('Fitted values')