Feature Engineering — The Skill That Wins Tabular Competitions

"70% of top performers in tabular competitions attributed their success to feature engineering rather than algorithm selection."

Feature engineering is the single highest-leverage skill in tabular ML competitions. This guide covers every major technique used in winning solutions.

---

PART 1 — GROUPBY AGGREGATIONS (The #1 Technique)

The most powerful class of tabular features. Compute statistics grouped by categorical columns.

Basic Groupby Features

import pandas as pd

# For each categorical column, compute aggregates of numerical columns
cat_cols = ['customer_id', 'product_category', 'region']
num_cols = ['amount', 'quantity', 'price']

for cat in cat_cols:
    for num in num_cols:
        grp = df.groupby(cat)[num]
        df[f'{cat}_{num}_mean']   = df[cat].map(grp.mean())
        df[f'{cat}_{num}_std']    = df[cat].map(grp.std())
        df[f'{cat}_{num}_min']    = df[cat].map(grp.min())
        df[f'{cat}_{num}_max']    = df[cat].map(grp.max())
        df[f'{cat}_{num}_count']  = df[cat].map(grp.count())
        df[f'{cat}_{num}_nuniq']  = df[cat].map(grp.nunique())
        df[f'{cat}_{num}_skew']   = df[cat].map(grp.skew())
        # Percentiles
        df[f'{cat}_{num}_q10']    = df[cat].map(grp.quantile(0.10))
        df[f'{cat}_{num}_q90']    = df[cat].map(grp.quantile(0.90))

Ratio Features from Groupby

# Value vs. group statistics
df['amount_vs_cat_mean'] = df['amount'] / (df['cat_amount_mean'] + 1e-6)
df['amount_vs_cat_std']  = df['amount'] / (df['cat_amount_std'] + 1e-6)
df['count_vs_nuniq']     = df['cat_amount_count'] / (df['cat_amount_nuniq'] + 1)

Two-Column Interaction Groupby

# Combine two categorical columns into one interaction key
df['cat1_cat2'] = df['cat1'].astype(str) + '_' + df['cat2'].astype(str)
# Then compute groupby aggregates on this combined key

---

PART 2 — CATEGORICAL ENCODING

Target Encoding (Inside CV Folds — No Leakage)

from sklearn.model_selection import KFold
import numpy as np

def target_encode_cv(train, test, col, target, n_folds=5, alpha=10):
    """Target encoding with regularization and proper fold handling."""
    global_mean = train[target].mean()
    oof_encoded = np.zeros(len(train))
    
    kf = KFold(n_splits=n_folds, shuffle=True, random_state=42)
    for tr_idx, val_idx in kf.split(train):
        tr, val = train.iloc[tr_idx], train.iloc[val_idx]
        stats = tr.groupby(col)[target].agg(['mean', 'count'])
        # Regularization: blend toward global mean for rare categories
        stats['encoded'] = (stats['mean'] * stats['count'] + global_mean * alpha) / \
                           (stats['count'] + alpha)
        oof_encoded[val_idx] = val[col].map(stats['encoded']).fillna(global_mean)
    
    # For test set: use all training data
    stats_full = train.groupby(col)[target].agg(['mean', 'count'])
    stats_full['encoded'] = (stats_full['mean'] * stats_full['count'] + global_mean * alpha) / \
                            (stats_full['count'] + alpha)
    test_encoded = test[col].map(stats_full['encoded']).fillna(global_mean)
    
    return oof_encoded, test_encoded

Frequency Encoding

# Replace category with its frequency in training data
freq = df[col].value_counts(normalize=True)
df[f'{col}_freq'] = df[col].map(freq)

When to Use Each Encoding

Method	Use When	Risk
Target encoding	High cardinality, regression/classification	Data leakage if done outside CV
Frequency encoding	High cardinality, no target access	No leakage risk
One-hot encoding (OHE)	Low cardinality (< 20 categories)	Dimensionality explosion
Label encoding	Ordinal categories OR tree-based models	Not meaningful for linear models
CatBoost native	Any categoricals with CatBoost	None

---

PART 3 — MISSING DATA AS SIGNAL

Never discard missing values — missingness is often predictive.

# Step 1: Create binary missingness indicator columns
for col in df.columns:
    if df[col].isnull().any():
        df[f'{col}_was_missing'] = df[col].isnull().astype(int)

# Step 2: Encode co-occurrence patterns of missing values
missing_cols = [c for c in df.columns if df[c].isnull().any()]
df['missing_pattern'] = 0
for i, col in enumerate(missing_cols):
    df['missing_pattern'] += df[col].isnull().astype(int) * (2 ** i)

# Step 3: Fill numerical with mean (for tree models)
for col in num_cols:
    df[col] = df[col].fillna(df[col].mean())

# Step 4: Fill categorical with new category
for col in cat_cols:
    df[col] = df[col].fillna('MISSING')

---

PART 4 — TIME SERIES AND SEQUENTIAL FEATURES

Critical rule: all features must be computed using only past data relative to each row. No future leakage.

df = df.sort_values('date')

# Lag features
for lag in [1, 2, 3, 7, 14, 30]:
    df[f'target_lag_{lag}'] = df.groupby('entity_id')['target'].shift(lag)

# Rolling statistics (must exclude current row)
for window in [7, 14, 30]:
    rolling = df.groupby('entity_id')['value'].shift(1).rolling(window)
    df[f'roll_mean_{window}']  = rolling.mean().reset_index(0, drop=True)
    df[f'roll_std_{window}']   = rolling.std().reset_index(0, drop=True)
    df[f'roll_min_{window}']   = rolling.min().reset_index(0, drop=True)
    df[f'roll_max_{window}']   = rolling.max().reset_index(0, drop=True)

# Date/time decomposition
df['day_of_week']  = df['date'].dt.dayofweek
df['month']        = df['date'].dt.month
df['quarter']      = df['date'].dt.quarter
df['is_weekend']   = (df['date'].dt.dayofweek >= 5).astype(int)
df['day_of_year']  = df['date'].dt.dayofyear

---

PART 5 — NUMERICAL BINNING AND DIGIT EXTRACTION

# Round at multiple precision levels
df['price_rounded_10']  = (df['price'] / 10).round() * 10
df['price_rounded_100'] = (df['price'] / 100).round() * 100

# Extract individual digits (captures price encoding tricks like 9.99)
df['price_cents'] = df['price'] % 1.0  # fractional part
df['price_tens_digit'] = (df['price'] // 10) % 10

# Histogram binning
df['price_bin'] = pd.qcut(df['price'], q=10, labels=False, duplicates='drop')

---

PART 6 — POLYNOMIAL INTERACTION FEATURES

from itertools import combinations

# All pairwise products of numerical features
num_cols = ['feature_a', 'feature_b', 'feature_c']
for c1, c2 in combinations(num_cols, 2):
    df[f'{c1}_x_{c2}'] = df[c1] * df[c2]
    df[f'{c1}_div_{c2}'] = df[c1] / (df[c2] + 1e-6)
    df[f'{c1}_plus_{c2}'] = df[c1] + df[c2]
    df[f'{c1}_minus_{c2}'] = df[c1] - df[c2]

---

PART 7 — AUTOMATED FEATURE GENERATION AT SCALE

For competitions where you have compute, automate across all combinations:

# Generate thousands of groupby features automatically
import pandas as pd
from itertools import product

def auto_groupby_features(df, cat_cols, num_cols, aggs=['mean','std','min','max','count']):
    new_features = {}
    for cat, num in product(cat_cols, num_cols):
        grp = df.groupby(cat)[num]
        for agg in aggs:
            fname = f'{cat}_{num}_{agg}'
            new_features[fname] = df[cat].map(getattr(grp, agg)())
    return pd.DataFrame(new_features)

# On GPU with cuDF (RAPIDS) — same API, ~100x faster
import cudf
df_gpu = cudf.from_pandas(df)
# Same code works with cudf.DataFrame

---

PART 8 — FEATURE SELECTION

More features ≠ better score. Prune aggressively.

import lightgbm as lgb
import shap
import numpy as np

# Method 1: LightGBM built-in importance
model = lgb.LGBMClassifier()
model.fit(X_train, y_train)
importance = pd.DataFrame({
    'feature': X_train.columns,
    'importance': model.feature_importances_
}).sort_values('importance', ascending=False)

# Drop features with zero importance
zero_importance = importance[importance['importance'] == 0]['feature'].tolist()
X_train = X_train.drop(columns=zero_importance)

# Method 2: SHAP values (more reliable than built-in importance)
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_train)
shap_importance = np.abs(shap_values).mean(0)

# Method 3: Permutation importance (model-agnostic)
from sklearn.inspection import permutation_importance
result = permutation_importance(model, X_val, y_val, n_repeats=10, random_state=42)

---

PART 9 — REFERENCE DATA INTEGRATION

When Kaggle Playground datasets originate from public real datasets:

1. Search Kaggle discussions for the original source dataset
2. Download the real data
3. Join on overlapping IDs or matching columns
4. Features from the real data that are absent from the Playground version = immediate edge

---

Key Rules

1. Every feature must be validated — add it, run CV, keep only if score improves
2. Target encoding must always happen inside CV folds — leakage here is invisible and catastrophic
3. Time series features must respect temporal order — validate your CV split first
4. More raw combinations → more signal found — automate generation, then select
5. Missingness is a feature — always create _was_missing indicators
6. SHAP over importance — built-in importance can be misleading; use SHAP for final selection