Xgboost Lightgbm

XGBoost & LightGBM - Gradient Boosting for Tabular Data

XGBoost (eXtreme Gradient Boosting) and LightGBM (Light Gradient Boosting Machine) are the de facto standard libraries for machine learning on tabular/structured data. They consistently win Kaggle competitions and are widely used in industry for their speed, accuracy, and robustness.

When to Use

• Classification or regression on tabular data (CSVs, databases, spreadsheets).
• Kaggle competitions or data science competitions on structured data.
• Feature importance analysis and feature selection.
• Handling missing values automatically (no need to impute).
• Working with imbalanced datasets (built-in class weighting).
• Need for fast training on large datasets (millions of rows).
• Hyperparameter tuning with cross-validation.
• Ranking tasks (learning-to-rank algorithms).
• When you need interpretable feature importances.
• Production ML systems requiring fast inference on tabular data.

Reference Documentation

XGBoost Official: https://xgboost.readthedocs.io/ XGBoost GitHub: https://github.com/dmlc/xgboost LightGBM Official: https://lightgbm.readthedocs.io/ LightGBM GitHub: https://github.com/microsoft/LightGBM Search patterns: xgboost.XGBClassifier, lightgbm.LGBMRegressor, xgboost.train, lightgbm.cv

Core Principles

Gradient Boosting Trees

Both libraries build an ensemble of decision trees sequentially, where each new tree corrects errors from previous trees. This creates highly accurate models that capture complex non-linear patterns.

Speed vs Accuracy Trade-offs

XGBoost: Slower but often slightly more accurate. Better for smaller datasets (<100k rows). LightGBM: Faster, especially on large datasets (millions of rows). Uses histogram-based learning.

Regularization

Both include L1/L2 regularization (alpha, lambda parameters) to prevent overfitting. This is crucial when you have many features.

Handling Categorical Features

LightGBM has native categorical feature support. XGBoost requires encoding (label encoding or one-hot).

Quick Reference

Installation

# Install both
pip install xgboost lightgbm

# For GPU support
pip install xgboost[gpu] lightgbm[gpu]

Standard Imports

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, mean_squared_error

# XGBoost
import xgboost as xgb
from xgboost import XGBClassifier, XGBRegressor

# LightGBM
import lightgbm as lgb
from lightgbm import LGBMClassifier, LGBMRegressor

Basic Pattern - Classification with XGBoost

from xgboost import XGBClassifier
from sklearn.model_selection import train_test_split

# 1. Prepare data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# 2. Create and train model
model = XGBClassifier(
    n_estimators=100,
    learning_rate=0.1,
    max_depth=6,
    random_state=42
)
model.fit(X_train, y_train)

# 3. Predict and evaluate
y_pred = model.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
print(f"Accuracy: {accuracy:.4f}")

Basic Pattern - Regression with LightGBM

from lightgbm import LGBMRegressor

# 1. Create model
model = LGBMRegressor(
    n_estimators=100,
    learning_rate=0.1,
    num_leaves=31,
    random_state=42
)

# 2. Train
model.fit(X_train, y_train)

# 3. Predict
y_pred = model.predict(X_test)
rmse = mean_squared_error(y_test, y_pred, squared=False)
print(f"RMSE: {rmse:.4f}")

Critical Rules

✅ DO

• Use Early Stopping - Always use early stopping with a validation set to prevent overfitting and save training time.
• Start with Defaults - Both libraries have excellent default parameters. Start there before tuning.
• Monitor Training - Use eval_set parameter to track validation metrics during training.
• Handle Imbalance - For imbalanced classes, use scale_pos_weight (XGBoost) or class_weight (LightGBM).
• Feature Engineering - Create interaction features, polynomial features, aggregations - boosting excels with rich feature sets.
• Use Native API for Advanced Control - For complex tasks, use xgb.train() or lgb.train() instead of sklearn wrappers.
• Save Models Properly - Use .save_model() and .load_model() methods, not pickle (more robust).
• Check Feature Importance - Always examine feature importances to understand your model and detect data leakage.

❌ DON'T

• Don't Forget to Normalize Target - For regression, if target has wide range, consider log-transform or standardization.
• Don't Ignore Tree Depth - max_depth (XGBoost) or num_leaves (LightGBM) are critical. Too deep = overfit.
• Don't Use Default Learning Rate for Large Datasets - Reduce learning_rate to 0.01-0.05 for datasets >1M rows.
• Don't Mix Up Parameters - XGBoost uses max_depth, LightGBM uses num_leaves. They're different!
• Don't One-Hot Encode for LightGBM - Use categorical_feature parameter instead for better performance.
• Don't Skip Cross-Validation - Always CV before trusting a single train/test split.

Anti-Patterns (NEVER)

# ❌ BAD: Training without validation set or early stopping
model = XGBClassifier(n_estimators=1000)
model.fit(X_train, y_train)  # Will likely overfit

# ✅ GOOD: Use early stopping with validation
model = XGBClassifier(n_estimators=1000, early_stopping_rounds=10)
model.fit(
    X_train, y_train,
    eval_set=[(X_val, y_val)],
    verbose=False
)

# ❌ BAD: One-hot encoding categorical features for LightGBM
X_encoded = pd.get_dummies(X)  # Creates many sparse columns
model = LGBMClassifier()
model.fit(X_encoded, y)

# ✅ GOOD: Use categorical_feature parameter
model = LGBMClassifier()
model.fit(
    X, y,
    categorical_feature=['category_col1', 'category_col2']
)

# ❌ BAD: Ignoring class imbalance
model = XGBClassifier()
model.fit(X_train, y_train)  # Majority class dominates

# ✅ GOOD: Handle imbalance with scale_pos_weight
from sklearn.utils.class_weight import compute_sample_weight
scale_pos_weight = (y_train == 0).sum() / (y_train == 1).sum()
model = XGBClassifier(scale_pos_weight=scale_pos_weight)
model.fit(X_train, y_train)

XGBoost Fundamentals

Scikit-learn Style API

from xgboost import XGBClassifier
import numpy as np

# Binary classification
model = XGBClassifier(
    n_estimators=100,        # Number of trees
    max_depth=6,             # Maximum tree depth
    learning_rate=0.1,       # Step size shrinkage (eta)
    subsample=0.8,           # Row sampling ratio
    colsample_bytree=0.8,    # Column sampling ratio
    random_state=42
)

# Train with early stopping
model.fit(
    X_train, y_train,
    eval_set=[(X_val, y_val)],
    early_stopping_rounds=10,
    verbose=True
)

# Get best iteration
print(f"Best iteration: {model.best_iteration}")
print(f"Best score: {model.best_score}")

Native XGBoost API (More Control)

import xgboost as xgb

# 1. Create DMatrix (XGBoost's internal data structure)
dtrain = xgb.DMatrix(X_train, label=y_train)
dval = xgb.DMatrix(X_val, label=y_val)

# 2. Set parameters
params = {
    'objective': 'binary:logistic',  # or 'reg:squarederror' for regression
    'max_depth': 6,
    'eta': 0.1,                      # learning_rate
    'subsample': 0.8,
    'colsample_bytree': 0.8,
    'eval_metric': 'auc',
    'seed': 42
}

# 3. Train with cross-validation monitoring
evals = [(dtrain, 'train'), (dval, 'val')]
model = xgb.train(
    params,
    dtrain,
    num_boost_round=1000,
    evals=evals,
    early_stopping_rounds=10,
    verbose_eval=50
)

# 4. Predict
dtest = xgb.DMatrix(X_test)
y_pred_proba = model.predict(dtest)
y_pred = (y_pred_proba > 0.5).astype(int)

Cross-Validation

import xgboost as xgb

# Prepare data
dtrain = xgb.DMatrix(X_train, label=y_train)

# Parameters
params = {
    'objective': 'binary:logistic',