Financial ML Competition Guide — Transformers, NLP, Purged CV
This guide covers the three areas that separate top-10% from top-1% in finance-focused ML competitions: transformer architectures for time series, NLP signal extr…
Financial ML Competition Guide — Transformers, NLP, Purged CV
This guide covers the three areas that separate top-10% from top-1% in finance-focused ML competitions: transformer architectures for time series, NLP signal extraction for trading, and proper cross-validation that doesn't cheat.
PART 1 — TRANSFORMER ARCHITECTURES FOR FINANCIAL TIME SERIES
Architecture Rankings (2024–2025)
| Model | Best For | Key Innovation | When to Use |
|---|---|---|---|
| iTransformer | Multivariate long-horizon forecasting | Attention across variates (stocks), not time steps | Best default for multi-stock prediction |
| PatchTST | Univariate/multivariate, limited history | Segments series into patches like ViT | When history is short or computation is limited |
| Non-stationary Transformer | Non-stationary financial data | Series Stationarization + De-stationary Attention | Explicitly handles regime changes |
| Temporal Fusion Transformer (TFT) | Known future inputs (calendar, events) | Gated attention + variable selection | When you have scheduled future covariates |
| Autoformer/Informer | Legacy | — | Only for ensembles; newer architectures outperform |
Key 2024 insight: iTransformer (ICLR 2024 Spotlight) inverts the typical transformer — each time series (stock) becomes a token, not each time step. This captures cross-variate dependencies better for financial forecasting.
iTransformer — Implementation
# pip install iTransformer (or use the official repo)
# github.com/thuml/iTransformer
class ITransformer(nn.Module):
"""
Key insight: transpose input so attention runs across variates, not time.
Input: [batch, time_steps, n_stocks]
After transpose: [batch, n_stocks, time_steps] ← each stock is a token
"""
def __init__(self, seq_len, pred_len, n_variates, d_model=512, n_heads=8, n_layers=3):
super().__init__()
self.embedding = nn.Linear(seq_len, d_model) # embed each variate's time series
self.encoder = nn.TransformerEncoder(
nn.TransformerEncoderLayer(d_model, n_heads, dim_feedforward=2048, batch_first=True),
num_layers=n_layers
)
self.projection = nn.Linear(d_model, pred_len)
def forward(self, x):
# x: [batch, time, variates]
x = x.permute(0, 2, 1) # [batch, variates, time] — variates as tokens
x = self.embedding(x) # [batch, variates, d_model]
x = self.encoder(x) # attention across variates
return self.projection(x) # [batch, variates, pred_len]
PatchTST — Implementation
# "A Time Series is Worth 64 Words" — ICLR 2023
# Reduces complexity from O(T²) to O((T/patch_size)²)
class PatchTST(nn.Module):
def __init__(self, seq_len=512, patch_size=16, pred_len=96, d_model=128, n_heads=16):
super().__init__()
self.patch_size = patch_size
self.n_patches = seq_len // patch_size
self.patch_embedding = nn.Linear(patch_size, d_model)
self.positional_encoding = nn.Parameter(torch.zeros(1, self.n_patches, d_model))
self.transformer = nn.TransformerEncoder(
nn.TransformerEncoderLayer(d_model, n_heads, batch_first=True),
num_layers=3
)
self.head = nn.Linear(d_model * self.n_patches, pred_len)
def forward(self, x):
# x: [batch, seq_len]
patches = x.unfold(-1, self.patch_size, self.patch_size) # [batch, n_patches, patch_size]
tokens = self.patch_embedding(patches) + self.positional_encoding
encoded = self.transformer(tokens) # [batch, n_patches, d_model]
return self.head(encoded.flatten(1)) # [batch, pred_len]
RevIN — Apply to EVERY Financial Transformer
Reversible Instance Normalization (RevIN) handles non-stationarity. Apply as first and last layer:
class RevIN(nn.Module):
def __init__(self, num_features, eps=1e-5, affine=True):
super().__init__()
self.eps = eps
if affine:
self.affine_weight = nn.Parameter(torch.ones(num_features))
self.affine_bias = nn.Parameter(torch.zeros(num_features))
def forward(self, x, mode):
if mode == 'norm':
self.mean = x.mean(dim=1, keepdim=True).detach()
self.std = x.std(dim=1, keepdim=True).detach() + self.eps
x = (x - self.mean) / self.std
if hasattr(self, 'affine_weight'):
x = x * self.affine_weight + self.affine_bias
elif mode == 'denorm':
if hasattr(self, 'affine_weight'):
x = (x - self.affine_bias) / (self.affine_weight + self.eps)
x = x * self.std + self.mean
return x
# Usage in forward pass:
# x_norm = revin(x, 'norm')
# pred_norm = transformer(x_norm)
# pred = revin(pred_norm, 'denorm')
Preprocessing Financial Data for Transformers
import numpy as np
import pandas as pd
def preprocess_financial_for_transformer(prices: pd.DataFrame, window: int = 60):
"""
Standard preprocessing pipeline for feeding price data into transformers.
"""
# Step 1: Log returns (more stationary than raw prices)
log_returns = np.log(prices / prices.shift(1)).dropna()
# Step 2: Rolling z-score normalization (not global — prevents look-ahead)
def rolling_zscore(series, window=60):
mean = series.rolling(window).mean()
std = series.rolling(window).std() + 1e-9
return (series - mean) / std
normalized = log_returns.apply(lambda col: rolling_zscore(col, window))
# Step 3: Winsorize at 3 std
normalized = normalized.clip(-3, 3)
# Step 4: Forward-fill then zero-fill missing values
normalized = normalized.ffill().fillna(0)
return normalized
# Decomposition for STL (optional — separate trend from cycle)
from statsmodels.tsa.seasonal import STL
def decompose_series(series, period=252):
stl = STL(series.dropna(), period=period, robust=True)
result = stl.fit()
return result.trend, result.seasonal, result.resid
Competition-Specific Architecture Choices
| Competition | What Won | Why |
|---|---|---|
| Jane Street 2024 (~$120K) | MLP + autoencoder ensembles | 16ms inference limit eliminates heavy transformers |
| Optiver Realized Volatility | LightGBM + hand-crafted features | Order book structure → feature engineering wins |
| VN1 Forecasting 2024 | TFT (top non-ensemble) | Known calendar covariates → TFT advantage |
| General multivariate forecasting | iTransformer or PatchTST | Strong benchmark results vs legacy models |
Rule for competitions with inference time limits (< 100ms): Use LightGBM or MLP, not transformers.
PART 2 — NLP IN FINANCE / TRADING
Model Selection for Financial NLP (2025)
| Model | Best For | Key Stats |
|---|---|---|
| Large LLMs (LLaMA-2, GPT-4, OPT) | Alpha generation from text | OPT: 74.4% accuracy, Sharpe 3.05 in L/S portfolio |
| FinBERT | Structured sentiment (headlines, short text) | Fine-tuned on FPB + FiQA + TRC2; best for narrow classification |
| Longformer | Long documents (earnings calls, 10-K, 10-Q) | Handles 4,096 tokens; outperforms BERT on full transcripts |
| FinGPT | Open-source LLM for finance | Fine-tuned LLaMA — lower cost than GPT-4 for production |
Key finding: LLMs outperform FinBERT significantly in volatile markets. FinBERT fades under volatility; LLMs stay robust.
GPT-4 via MarketSenseAI: 125.9% cumulative return on S&P 100 (2023–2024) vs 73.5% index return.
Fine-Tuning FinBERT
from transformers import BertTokenizer, BertForSequenceClassification
import torch
model_name = "ProsusAI/finbert"
tokenizer = BertTokenizer.from_pretrained(model_name)
model = BertForSequenceClassification.from_pretrained(model_name)
# Labels: 0=negative, 1=neutral, 2=positive
def get_sentiment(text):
inputs = tokenizer(text, return_tensors="pt", max_length=512, truncation=True, padding=True)
with torch.no_grad():
outputs = model(**inputs)
probs = torch.softmax(outputs.logits, dim=-1)
return {
'negative': probs[0][0].item(),
'neutral': probs[0][1].item(),
'positive': probs[0][2].item(),
'sentiment_score': probs[0][2].item() - probs[0][0].item() # signed score
}
Longformer for Earnings Calls
from transformers import LongformerTokenizer, LongformerForSequenceClassification
# For full earnings call transcripts (often > 4,000 words)
tokenizer = LongformerTokenizer.from_pretrained('allenai/longformer-base-4096')
model = LongformerForSequenceClassification.from_pretrained('allenai/longformer-base-4096')
def analyze_earnings_call(transcript: str):
inputs = tokenizer(
transcript,
max_length=4096,
truncation=True,
return_tensors="pt",
padding="max_length"
)
# Longformer uses global attention on CLS token
inputs['global_attention_mask'] = torch.zeros_like(inputs['attention_mask'])
inputs['global_attention_mask'][:, 0] = 1 # CLS global attention
with torch.no_grad():
outputs = model(**inputs)
return torch.softmax(outputs.logits, dim=-1)
Alpha Signals from Different Text Sources
From earnings calls:
BULLISH_PHRASES = ['record results', 'exceeded expectations', 'strong momentum',
'will deliver', 'committed to', 'expect growth']
BEARISH_PHRASES = ['headwinds', 'challenging environment', 'may impact',
'uncertain', 'could affect', 'monitoring closely']
def earnings_call_features(transcript: str) -> dict:
# 1. Overall sentiment score (use FinBERT or LLM)
sentiment = get_sentiment(transcript[:512])
# 2. Guidance language detection
guidance_score = sum(1 for p in BULLISH_PHRASES if p in transcript.lower()) - \
sum(1 for p in BEARISH_PHRASES if p in transcript.lower())
# 3. Q&A section tone (analyst questions vs executive answers)
qa_start = transcript.find('Q&A') or transcript.find('Questions')
if qa_start > 0:
prepared = transcript[:qa_start]
qa = transcript[qa_start:]
qa_sentiment = get_sentiment(qa[:512])['sentiment_score']
else:
qa_sentiment = sentiment['sentiment_score']
# 4. Length features
n_words = len(transcript.split())
n_sentences = transcript.count('. ')
return {
'sentiment_score': sentiment['sentiment_score'],
'guidance_score': guidance_score,
'qa_sentiment': qa_sentiment,
'transcript_length': n_words,
'sentiment_minus_qa': sentiment['sentiment_score'] - qa_sentiment # tension signal
}
From news (event-driven windows):
def news_alpha_features(news_df: pd.DataFrame, price_df: pd.DataFrame) -> pd.DataFrame:
"""
Most signal lives in the first 15 minutes and first day after announcement.
news_df: columns ['date', 'ticker', 'headline', 'source']
"""
features = []
for ticker, group in news_df.groupby('ticker'):
daily = group.groupby('date').agg(
n_articles=('headline', 'count'),
avg_sentiment=('headline', lambda x: np.mean([
get_sentiment(h)['sentiment_score'] for h in x
])),
source_weight=('source', lambda x: np.mean([
2.0 if s in ['Reuters', 'Bloomberg'] else 1.0 for s in x
]))
)
# Buzz deviation from 60-day baseline
daily['buzz_deviation'] = daily['n_articles'] - daily['n_articles'].rolling(60).mean()
# Sentiment momentum (5-day)
daily['sentiment_momentum'] = daily['avg_sentiment'].diff(5)
features.append(daily.assign(ticker=ticker))
return pd.concat(features)
From SEC filings (10-K year-over-year changes):
def sec_filing_features(current_10k: str, previous_10k: str) -> dict:
# Risk factor section length change (longer = more uncertainty)
risk_start = current_10k.find('RISK FACTORS')
risk_end = current_10k.find('UNRESOLVED STAFF COMMENTS')
current_risk_len = len(current_10k[risk_start:risk_end].split()) if risk_start > 0 else 0
risk_start_prev = previous_10k.find('RISK FACTORS')
risk_end_prev = previous_10k.find('UNRESOLVED STAFF COMMENTS')
prev_risk_len = len(previous_10k[risk_start_prev:risk_end_prev].split()) if risk_start_prev > 0 else 0
return {
'risk_section_change': current_risk_len - prev_risk_len, # longer = more risk
'risk_section_pct_change': (current_risk_len - prev_risk_len) / (prev_risk_len + 1)
}
Combining NLP + Price/Volume (The Winning Formula)
NLP signals work best as filters and adjusters on top of price-based signals, not as standalone predictors:
def combined_signal(price_signal, nlp_signal, nlp_threshold=0.3):
"""
price_signal: standard price/volume alpha (LightGBM prediction or GBDT score)
nlp_signal: sentiment score [-1, +1]
Use NLP to:
1. Filter out trades where NLP strongly disagrees with price signal
2. Scale up positions where NLP strongly agrees
"""
# Filter: suppress trade if NLP disagrees
agreement_mask = np.sign(price_signal) == np.sign(nlp_signal)
filtered_signal = price_signal * agreement_mask
# Scale: amplify where both agree strongly
nlp_confidence = np.abs(nlp_signal)
scaling = 1.0 + nlp_confidence * (np.abs(nlp_signal) > nlp_threshold)
return filtered_signal * scaling
PART 3 — PURGED CROSS-VALIDATION
The most important concept for financial ML that beginners get wrong.
Why Standard CV Fails for Financial Data
Standard k-fold assumes IID data. Financial data violates this: a 5-day return label at day T uses prices at T+1 through T+5. If T+2 is in your training set and T is in your test set, your model has seen the future. Your CV score is a lie.
Purged CV — Implementation
import numpy as np
import pandas as pd
from sklearn.model_selection import KFold
class PurgedKFold:
"""
Purged K-Fold: removes training samples whose labels overlap with test labels.
Parameters:
-----------
n_splits : int
pct_embargo : float
Fraction of dataset to embargo after each test set (prevents autocorrelation leakage)
"""
def __init__(self, n_splits=5, pct_embargo=0.01):
self.n_splits = n_splits
self.pct_embargo = pct_embargo
def split(self, X, y=None, pred_times=None, eval_times=None):
"""
pred_times : pd.Series — when each prediction is made
eval_times : pd.Series — when each label is realized (pred_time + horizon)
"""
indices = np.arange(len(X))
embargo_size = int(len(X) * self.pct_embargo)
test_ranges = [(i[0], i[-1] + 1) for i in np.array_split(indices, self.n_splits)]
for start, end in test_ranges:
test_indices = indices[start:end]
if pred_times is not None:
# Purge: remove training samples whose eval_times overlap with test pred_times
test_pred_start = pred_times.iloc[start]
test_eval_end = eval_times.iloc[end - 1] if end <= len(eval_times) else eval_times.iloc[-1]
# Keep training indices that don't overlap with test period
train_mask = (eval_times < test_pred_start) | (pred_times > test_eval_end)
else:
# Simple purge: remove border observations
train_mask = (indices < start - embargo_size) | (indices >= end + embargo_size)
train_indices = indices[train_mask]
yield train_indices, test_indices
Using the mlfinlab library (easiest):
# pip install mlfinlab
from mlfinlab.cross_validation import PurgedKFold
pkf = PurgedKFold(
n_splits=5,
pct_embargo=0.01 # embargo 1% of dataset after each test period
)
for train_idx, test_idx in pkf.split(X, y, pred_times=t1.index, eval_times=t1):
X_tr, X_val = X.iloc[train_idx], X.iloc[test_idx]
y_tr, y_val = y.iloc[train_idx], y.iloc[test_idx]
model.fit(X_tr, y_tr)
predictions[test_idx] = model.predict(X_val)
Simple Walk-Forward CV (Minimum Viable)
If you don't have mlfinlab, at minimum use time-based splits:
from sklearn.model_selection import TimeSeriesSplit
# Standard walk-forward
tscv = TimeSeriesSplit(n_splits=5, gap=20) # gap=20 days embargo
for train_idx, test_idx in tscv.split(X):
...
# Competition-style: fixed validation window (most recent data)
# Train on everything before cutoff date; validate on last N days
cutoff = int(len(X) * 0.8)
gap = 20 # days between train end and validation start
X_tr, X_val = X.iloc[:cutoff - gap], X.iloc[cutoff:]
y_tr, y_val = y.iloc[:cutoff - gap], y.iloc[cutoff:]
Combinatorial Purged CV (CPCV) — For Multiple Backtest Paths
Used to estimate Probability of Backtest Overfitting (PBO):
# pip install skfolio
from skfolio.model_selection import CombinatorialPurgedCV
cpcv = CombinatorialPurgedCV(
n_splits=10, # divide into 10 groups
n_test_splits=2, # 2 groups as test each time
purged_size=10, # observations to purge between train/test
embargo_size=5 # observations to embargo after each test
)
sharpe_scores = []
for train_idx, test_idx in cpcv.split(X):
model.fit(X.iloc[train_idx], y.iloc[train_idx])
preds = model.predict(X.iloc[test_idx])
returns = compute_strategy_returns(preds, prices.iloc[test_idx])
sharpe_scores.append(compute_sharpe(returns))
# Estimate PBO: what fraction of backtest paths had negative Sharpe?
pbo = np.mean(np.array(sharpe_scores) < 0)
print(f"Probability of Backtest Overfitting: {pbo:.1%}")
# If PBO > 50%, your strategy is likely overfit
Embargo Size Rules
| Situation | Embargo Size |
|---|---|
| Daily returns, 1-day prediction | 5 days |
| Daily returns, 5-day prediction | 10–20 days |
| Intraday (minutes), 1-hour prediction | 60 bars |
| Using 60-day rolling features | At least 60 observations |
| Optiver Trading at Close (competition) | 2 date_ids (used by 1st place) |
Part 4 — RESOURCES
| Resource | URL |
|---|---|
| iTransformer (official) | github.com/thuml/iTransformer |
| PatchTST (official) | github.com/PatchTST/PatchTST |
| RevIN paper | openreview.net/pdf?id=cGDAkQo1C0p |
| Non-stationary Transformer | arxiv.org/abs/2205.14415 |
| Financial time series transformer survey | github.com/UVA-MLSys/Financial-Time-Series |
| FinBERT | github.com/ProsusAI/finBERT |
| mlfinlab (Purged CV) | github.com/hudson-and-thames/mlfinlab |
| skfolio (CPCV) | skfolio.org/generated/skfolio.model_selection.CombinatorialPurgedCV |
| López de Prado — Advances in Financial ML | Available on Amazon; Chapter 7 (Purged CV) + Chapter 12 (CPCV) |
| QuantInsti CPCV tutorial | blog.quantinsti.com/cross-validation-embargo-purging-combinatorial |