Kaggle Finance Competitions — Deep Winning Guide

Jane Street, Optiver, and Two Sigma are the most prestigious finance-themed Kaggle competitions. Winning or placing highly in these is a top-tier signal for quant finance roles and trading coaching authority.

---

COMPETITION 1 — Jane Street Real-Time Market Data Forecasting (2024–2025, $120K)

URL: kaggle.com/competitions/jane-street-real-time-market-data-forecasting

Competition Facts

• Launched: October 14, 2024 | Deadline: January 13, 2025
• ~3,700 teams, $120,000 prize
• Most popular Kaggle competition of 2025 by team count
• ~2.5M data points, 500 training days, ~1 year test period
• Critical constraint: 16ms inference limit per iteration — eliminates heavy transformers

The Core Challenge

"The relationship of the features with the response is constantly changing." This is financial non-stationarity at its most extreme. Models that fit the past do not fit the future.

Winning Approach

Architecture (neural networks won, not GBDTs — rare for tabular):

• Supervised Autoencoder + MLP ensemble
• Layer structure: BatchNorm → Linear → LeakyReLU → Dropout
• 3 layers deep
• Multi-task training: simultaneously predict multiple responders
• Autoencoder learns denoised, distribution-robust representations

import torch
import torch.nn as nn

class JaneStreetNet(nn.Module):
    def __init__(self, input_dim=130, hidden_dim=256, output_dim=5, dropout=0.3):
        super().__init__()
        
        # Encoder (learns robust representation)
        self.encoder = nn.Sequential(
            nn.BatchNorm1d(input_dim),
            nn.Linear(input_dim, hidden_dim),
            nn.LeakyReLU(0.01),
            nn.Dropout(dropout),
            nn.Linear(hidden_dim, hidden_dim // 2),
            nn.LeakyReLU(0.01),
            nn.Dropout(dropout),
        )
        
        # Decoder (reconstruction for autoencoder pretraining)
        self.decoder = nn.Sequential(
            nn.Linear(hidden_dim // 2, hidden_dim),
            nn.LeakyReLU(0.01),
            nn.Linear(hidden_dim, input_dim)
        )
        
        # Predictor (multi-task: predict all responders)
        self.predictor = nn.Linear(hidden_dim // 2, output_dim)
    
    def forward(self, x):
        encoded = self.encoder(x)
        prediction = self.predictor(encoded)
        reconstruction = self.decoder(encoded)
        return prediction, reconstruction

# Training: joint loss = prediction_loss + reconstruction_loss * lambda
def combined_loss(pred, target, recon, original, lambda_recon=0.1):
    pred_loss  = nn.MSELoss()(pred, target)
    recon_loss = nn.MSELoss()(recon, original)
    return pred_loss + lambda_recon * recon_loss

Feature Quirks (Important Preprocessing)

Features 3, 4, 6, 19, 20, 22, 38 have histogram spikes → treat as categorical or clip them. Features 71, 85, 87, 92, 97, 105, 127, 129 have extreme long tails → clip at 5 std.

def preprocess_jane_street(X):
    spike_features = [3, 4, 6, 19, 20, 22, 38]
    tail_features = [71, 85, 87, 92, 97, 105, 127, 129]
    
    X = X.copy()
    
    # Handle spike features — round to nearest 0.1 to reveal categorical structure
    for f in spike_features:
        col = f'feature_{f:02d}'
        if col in X.columns:
            X[col] = X[col].round(1)
    
    # Handle fat-tail features — clip at 5 standard deviations
    for f in tail_features:
        col = f'feature_{f:02d}'
        if col in X.columns:
            std = X[col].std()
            mean = X[col].mean()
            X[col] = X[col].clip(mean - 5*std, mean + 5*std)
    
    return X

Validation Strategy

• Use Purged CV with large embargo (high autocorrelation in financial data)
• Time-based split: train on earliest data, validate on latest
• Monitor CV score stability across time — if variance is high, model is overfit to regime

---

COMPETITION 2 — Optiver Realized Volatility Prediction (2021)

URL: kaggle.com/competitions/optiver-realized-volatility-prediction

The Legendary 1st Place "Hack"

The winner noticed that tick sizes change over time in specific ways. By analyzing the price tick size distribution, they reverse-engineered the chronological order of time_ids — information that wasn't explicitly provided. This gave them a time-ordered dataset while other participants treated time_id as unordered.

Lesson: Always look for hidden ordering/leakage in competition data before modeling.

Core Features Every Top Solution Used

import pandas as pd
import numpy as np

def compute_realized_vol_features(book_df, trade_df):
    """
    book_df: bid_price1/2, ask_price1/2, bid_size1/2, ask_size1/2
    trade_df: price, size, order_count
    """
    features = {}
    
    # 1. Weighted Average Price (WAP) — THE fundamental microstructure signal
    def wap(bp, ap, bs, as_):
        return (bp * as_ + ap * bs) / (bs + as_)
    
    features['wap1'] = wap(book_df['bid_price1'], book_df['ask_price1'],
                           book_df['bid_size1'],  book_df['ask_size1'])
    features['wap2'] = wap(book_df['bid_price2'], book_df['ask_price2'],
                           book_df['bid_size2'],  book_df['ask_size2'])
    
    # 2. Log returns on WAP
    features['log_return1']    = np.log(features['wap1']).diff()
    features['log_return2']    = np.log(features['wap2']).diff()
    features['log_return_ask'] = np.log(book_df['ask_price1']).diff()
    features['log_return_bid'] = np.log(book_df['bid_price1']).diff()
    
    # 3. Realized volatility (target-equivalent for features at sub-windows)
    def realized_vol(log_returns):
        return np.sqrt(np.sum(log_returns**2))
    
    features['realized_vol_wap1'] = realized_vol(features['log_return1'].dropna())
    
    # 4. Best documented feature: "Weighted realized volatility"
    # = realized_vol × log(bid/ask size ratio during window)
    size_ratio = np.log(book_df['bid_size1'] / (book_df['ask_size1'] + 1e-9))
    features['weighted_realized_vol'] = realized_vol(features['log_return1'].dropna()) * \
                                        np.abs(size_ratio.mean())
    
    # 5. Order book pressure
    features['bid_ask_spread1'] = (book_df['ask_price1'] - book_df['bid_price1']) / \
                                   features['wap1']
    
    # 6. Order flow imbalance
    features['order_imbalance'] = (book_df['bid_size1'] - book_df['ask_size1']) / \
                                   (book_df['bid_size1'] + book_df['ask_size1'])
    
    # 7. Trade statistics
    features['trade_size_mean'] = trade_df['size'].mean()
    features['trade_size_std']  = trade_df['size'].std()
    features['n_trades']        = len(trade_df)
    
    return pd.Series({k: v if np.isscalar(v) else v.mean() for k, v in features.items()})

Cross-Asset Features (Used by Top Solutions)

def cross_stock_features(all_stocks_rv: pd.DataFrame) -> pd.DataFrame:
    """
    Compute sector-level volatility correlations.
    all_stocks_rv: realized volatility per stock per time_id
    """
    # Sector-level realized vol (average across sector peers)
    sector_avg_rv = all_stocks_rv.groupby('sector').transform('mean')
    
    # Each stock's RV relative to its sector
    all_stocks_rv['rv_vs_sector'] = all_stocks_rv['rv'] / (sector_avg_rv['rv'] + 1e-9)
    
    # Market-wide average RV (a macro signal)
    all_stocks_rv['market_avg_rv'] = all_stocks_rv.groupby('time_id')['rv'].transform('mean')
    
    return all_stocks_rv

---

COMPETITION 3 — Optiver Trading at the Close (2023)

URL: kaggle.com/competitions/optiver-trading-at-the-close

Task

Predict NASDAQ closing auction price movements for ~200 stocks, using the last 10 minutes of each trading session. Evaluate against a synthetic index.

Target: (StockWAP_{t+60} / StockWAP_t - IndexWAP_{t+60} / IndexWAP_t) × 10000

1st Place Feature Engineering (Complete)

import numba
import numpy as np
import pandas as pd
from itertools import combinations

def build_features(df):
    
    # V1 — Direct features from data
    prices   = ['reference_price', 'far_price', 'near_price', 'ask_price', 'bid_price', 'wap']
    sizes    = ['matched_size', 'bid_size', 'ask_size', 'imbalance_size']
    
    # V2 — Pairwise price features: (x - y) / (x + y)  [Numba-accelerated]
    @numba.njit
    def pairwise_features(arr):
        n_cols = arr.shape[1]
        result = np.empty((arr.shape[0], n_cols * (n_cols - 1) // 2))
        idx = 0
        for i in range(n_cols):
            for j in range(i + 1, n_cols):
                result[:, idx] = (arr[:, i] - arr[:, j]) / (arr[:, i] + arr[:, j] + 1e-9)
                idx += 1
        return result
    
    price_arr = df[prices].values
    pairwise  = pairwise_features(price_arr)
    for i, (c1, c2) in enumerate(combinations(prices, 2)):
        df[f'pair_{c1}_{c2}'] = pairwise[:, i]
    
    # V3 — Triplet features: (max - mid) / (mid - min)
    def triplet_features(row, cols):
        vals = sorted([row[c] for c in cols])
        if vals[2] - vals[0] < 1e-9: return 0
        return (vals[2] - vals[1]) / (vals[1] - vals[0] + 1e-9)
    
    # V4 — Micro-price
    df['micro_price'] = (df['bid_price'] * df['ask_size'] + df['ask_price'] * df['bid_size']) / \
                        (df['bid_size'] + df['ask_size'])
    
    # V5 — Market urgency and depth pressure
    df['market_urgency'] = df['imbalance_size'] * df['reference_price']
    df['depth_pressure']  = (df['ask_size'] - df['bid_size']) / (df['ask_size'] + df['bid_size'])
    
    # V6 — Statistical aggregations
    price_arr = df[prices].values
    df['price_mean']  = price_arr.mean(axis=1)
    df['price_std']   = price_arr.std(axis=1)
    df['price_skew']  = pd.DataFrame(price_arr).skew(axis=1).values
    df['price_kurt']  = pd.DataFrame(price_arr).kurtosis(axis=1).values
    
    # V7 — Per-stock historical statistics (computed globally, joined in)
    # median_size, std_price — aggregated over entire training history per stock_id
    
    # V8 — Temporal features
    df['day_of_week']  = df['date_id'] % 5   # 0-4
    df['day_of_month'] = df['date_id'] % 20  # 0-19
    
    # V9 — Lagged features
    for lag in [1, 2, 3, 5, 10]:
        df[f'wap_lag_{lag}']    = df.groupby('stock_id')['wap'].shift(lag)
        df[f'wap_pct_{lag}']    = df['wap'] / (df[f'wap_lag_{lag}'] + 1e-9) - 1
        df[f'target_lag_{lag}'] = df.groupby('stock_id')['target'].shift(lag)
    
    return df

# Model config (1st place):
# LightGBM: n_estimators=6300, subsample=0.7, colsample_bytree=0.7
# 5-fold purged CV with 2-date gap between train and validation

---

COMPETITION 4 — Two Sigma Financial Modeling (2016–2017)

5th Place Strategy (Team "Bestfitting")

Feature engineering:

# Original features + transformations
def build_two_sigma_features(df):
    base_features = [c for c in df.columns if c.startswith('technical_')]
    
    for col in base_features:
        df[f'{col}_abs']  = np.abs(df[col])
        df[f'{col}_log']  = np.log(np.abs(df[col]) + 1e-9)
        df[f'{col}_std']  = df.groupby('id')[col].transform(lambda x: x.rolling(20).std())
        for lag in [1, 5, 10, 20]:
            df[f'{col}_lag{lag}'] = df.groupby('id')[col].shift(lag)
    
    return df

# Whole-market macro features (key insight from 5th place):
def add_market_features(df):
    df['market_return']  = df.groupby('timestamp')['y'].transform('mean')
    df['market_vol']     = df.groupby('timestamp')['y'].transform('std')
    df['market_trend']   = df['market_return'].rolling(5).mean()
    df['volatility_regime'] = (df['market_vol'] > df['market_vol'].rolling(60).quantile(0.75)).astype(int)
    return df

Two-level stacking:

• Level 1: multiple weak models trained on features
• Level 2: another model trained on Level 1 predictions as features, plus original features
• Focus: model stability across regimes, not peak performance on single regime

---

UNIVERSAL FINANCE COMPETITION CHECKLIST

[ ] Use time-based CV split — NEVER random k-fold
[ ] Apply purged CV with embargo if labels have forward-looking windows
[ ] Compute WAP at Level 1 AND Level 2 of order book
[ ] Add log returns on WAP (not raw price changes)
[ ] Include order flow imbalance feature
[ ] Include bid-ask spread feature
[ ] Add lagged features: 1, 2, 3, 5, 10 periods minimum
[ ] Add rolling statistics: mean, std over 5/10/20/60 windows
[ ] Compute pairwise ratios: (x-y)/(x+y) for all price pairs
[ ] Encode time-of-day (seconds, minute bucket, day-of-week)
[ ] Train LightGBM with early stopping, NOT fixed n_estimators
[ ] Use neural networks if inference time is generous (> 1 second)
[ ] Ensemble at minimum 3 models trained on different seeds
[ ] Submit purged OOF predictions as meta-features for stacking
[ ] Check for hidden temporal structure (tick size, ID ordering)

---

KEY GITHUB REPOSITORIES

Resource	URL
Optiver Trading at Close — 1st place code	kaggle.com/competitions/optiver-trading-at-the-close/discussion
Optiver Realized Volatility — solutions	kaggle.com/competitions/optiver-realized-volatility-prediction/discussion
Jane Street 2024 — top solutions	kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion
Two Sigma 5th place interview	medium.com/kaggle-blog (search "two sigma bestfitting")
mlfinlab (Purged CV)	github.com/hudson-and-thames/mlfinlab