How to Win Machine Learning Competitions

Step 1 — Pick the Right Competition

• Choose competitions that match your skill level or slightly above
• Prefer competitions with active discussion forums (more learning)
• Social impact competitions (DrivenData, Zindi) have fewer participants = better odds

Step 2 — Understand the Problem Deeply (Day 1–2)

• Read EVERYTHING: problem statement, data description, evaluation metric
• Know your metric cold: AUC? RMSE? MAP? F1? Each requires different optimization
• Read ALL public notebooks on the discussion tab before writing a single line

Step 3 — Exploratory Data Analysis (EDA)

• Distribution of target variable (class imbalance?)
• Missing value patterns (MCAR? MAR? MNAR?)
• Feature correlations with target
• Time-based leakage checks
• Use: pandas-profiling or ydata-profiling for fast EDA

Step 4 — Baseline Model FAST

• Get a simple baseline running in < 1 day
• LightGBM/XGBoost for tabular, BERT for NLP, ResNet for vision
• Validate locally with proper cross-validation (stratified k-fold for classification)
• Submit — know your baseline public LB score

Step 5 — Feature Engineering (Biggest Alpha)

• This is where most competitions are won
• Aggregate features, lag features, interaction features
• Domain knowledge matters: think like the problem owner
• Never add features blindly — validate each with CV score

Step 6 — Model Selection & Tuning

• Try: LightGBM, XGBoost, CatBoost, TabNet (tabular); Transformers (NLP); EfficientNet/ViT (vision)
• Use Optuna for hyperparameter tuning
• Don't over-tune early — it's a trap

Step 7 — Ensembling (Top 10% Secret)

• Simple averaging of diverse models often beats any single model
• Stacking: use out-of-fold predictions as meta-features
• Blending: linear combination of predictions
• Diverse models > same model tuned differently
• Grandmaster-level: 72-model 3-level stack (XGBoost + LightGBM + CatBoost + NN + TabPFN + KNN + SVR + Ridge + RF) — documented win in 2025
• Hill climbing ensembles: Start with best single model, add one at a time, keep only if CV improves. GPU-accelerated with CuPy to test thousands of weight combos

Step 8 — Validation Strategy (Most Important)

• Your local CV must match the public LB — if not, investigate
• "Trust your CV" is the cardinal rule — multiple 2025 Grandmaster wins came from ignoring the public LB entirely
• Find a K with the lowest gap between CV and public LB score
• Time-series data: always use time-based splits (no random shuffle)
• GroupKFold when data has patient IDs, user IDs, or any natural grouping
• Simulate public/private split: use multiple CV folds to estimate potential shakeup risk

Step 9 — Final Submission Strategy

• Submit 2 final submissions: (1) best CV score, (2) best LB score
• Don't chase the public LB in the final days — private LB shakeups are real
• Retrain final model on 100% of training data after CV-based tuning is done
• Keep a log of ALL submissions with scores and wandb run links

Grandmaster-Level Techniques (2025–2026)

Pseudo-Labeling

• Generate predictions on unlabeled test data, fold back into training as "labels"
• Use soft labels (probabilities), not hard predictions
• Run 2–5 rounds — multi-round consistently outperforms single-pass
• In k-fold: compute k separate pseudo-label sets to prevent leakage
• Can jump leaderboard ranking dramatically (documented: rank 49 → rank 1)

Synthetic Data

• Use GPT-4/Claude to generate synthetic training examples, then train on real + synthetic
• Most effective when original training data is small
• Required in top solutions for ARC Prize 2025, AIMO, and NLP competitions

AutoML — AutoGluon

• Won medals in 15 of 18 tabular competitions in 2024, including 7 gold
• Can beat manually tuned GBDT ensembles with less effort
• Use as a strong baseline before building custom pipelines

TabPFN v2 (2025 — Game Changer for Small Tabular Datasets)

TabPFN is a pre-trained transformer that does in-context learning on tabular data. No training required — it's inference-only.

# pip install tabpfn
from tabpfn import TabPFNClassifier
import numpy as np

clf = TabPFNClassifier(device='cuda')  # GPU recommended
clf.fit(X_train, y_train)   # "fits" instantly — no actual training
predictions = clf.predict(X_test)
prediction_probas = clf.predict_proba(X_test)

When to use TabPFN v2:

• Training set < 10,000 rows: TabPFN v2 often beats LightGBM
• Training set < 1,000 rows: TabPFN v2 is almost always best
• Use as one model in your ensemble — its predictions are uncorrelated with GBDT predictions
• Do NOT use for > 10K rows — performance degrades vs GBDTs at scale

Test-Time Training (TTT) for NLP/Vision

Fine-tune the model on each test example at inference time using self-supervised objectives.

# TTT Pattern for NLP
def test_time_train(model, test_sample, n_steps=10, lr=1e-5):
    """Fine-tune on test sample using masked language modeling objective."""
    optimizer = torch.optim.AdamW(model.parameters(), lr=lr)
    model.train()
    
    for _ in range(n_steps):
        # Create masked version of test sample
        masked_input, labels = mask_tokens(test_sample)
        loss = model(**masked_input, labels=labels).loss
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    
    model.eval()
    return model

TTT is especially powerful for ARC Prize (see ARC_PRIZE_GUIDE.md) and domain-shift NLP tasks.

Hill Climbing Ensemble Weights (GPU-Accelerated)

Instead of simple averaging or Ridge meta-learner, search for optimal weights:

import numpy as np

def hill_climb_ensemble(oof_preds, y_true, metric_fn, n_iter=1000, lr=0.01):
    """
    Start with equal weights. Each iteration: perturb one weight, 
    keep the change if CV score improves.
    """
    n_models = len(oof_preds)
    weights = np.ones(n_models) / n_models
    best_score = metric_fn(y_true, np.average(oof_preds, axis=0, weights=weights))
    
    for _ in range(n_iter):
        i = np.random.randint(n_models)
        delta = np.random.uniform(-lr, lr)
        new_weights = weights.copy()
        new_weights[i] = max(0, new_weights[i] + delta)
        new_weights /= new_weights.sum()  # normalize
        
        score = metric_fn(y_true, np.average(oof_preds, axis=0, weights=new_weights))
        if score > best_score:
            weights = new_weights
            best_score = score
    
    return weights, best_score

# Usage: oof_preds is list of OOF prediction arrays, one per model
weights, cv_score = hill_climb_ensemble(oof_preds, y_train, metric_fn=roc_auc_score)

Common Mistakes to Avoid

• Data leakage (using future data in features)
• Overfitting to public LB (trust CV instead)
• Not reading the evaluation metric carefully
• Ignoring the discussion tab
• Starting with deep learning when gradient boosting would win
• Not logging experiments — you WILL forget what worked

---

2025–2026 Competition Landscape Facts

• 390+ competitions, $16M+ total prize pool — largest ever
• Over 50% of winners were solo competitors (third consecutive year)
• Over 50% of winners were first-time winners — the field is accessible
• Qwen models (Qwen2.5, Qwen3) won all three major NLP grand prizes
• Transformer-based vision models surpassed CNNs among winning solutions for the first time
• H100 now the most common GPU among winners (replaced A100)
• 9 winning solutions ran entirely on free Kaggle Notebooks

Tools Used in 2025 Winning Solutions

Tool	Role	2025 Wins
pandas	Tabular data processing	61
NumPy	Numerical computation	62
XGBoost	Gradient boosting	14
LightGBM	Gradient boosting	14
PyTorch	Deep learning	44
wandb	Experiment tracking	11
CatBoost	Gradient boosting	8
AutoGluon	AutoML tabular	7 gold medals in tabular
Optuna	Hyperparameter tuning	Standard
RAPIDS cuDF/cuML	GPU-accelerated data + training	Multiple
Qwen2.5/Qwen3	NLP foundation model	All 3 NLP grand prizes
Unsloth	LLM fine-tuning (14B in 16GB GPU)	3 winning solutions
vLLM	LLM batch inference	4 winning solutions
TabPFN v2	Tabular foundation model	First competition win (2025)
TabICL v2	Tabular foundation model (2026)	10x faster than TabPFN v2

---

Deep-Dive Guides in This Folder

File	What's Inside
FEATURE_ENGINEERING_GUIDE.md	Groupby aggregations, target encoding, missing data, time series features, polynomial interactions — with full code
DEEP_LEARNING_COMPETITION_GUIDE.md	Vision backbones (2025 ranking), augmentation strategy, NLP models, AWP, LoRA, pseudo-labeling
STAYING_AHEAD_META_GUIDE.md	Paper tracking, solution readup framework, reusable codebase template, grandmaster mindset

---

Resources

• "How to Win a Data Science Competition" — Coursera (top-rated MOOC)
• Kaggle winning solutions: farid.one/kaggle-solutions/
• "Approaching Almost Any ML Problem" — book by Abhishek Thakur (free PDF)
• NVIDIA Grandmaster Playbook: developer.nvidia.com/blog (search "kaggle grandmaster")
• MLContests 2025 State of ML Competitions: mlcontests.com/state-of-machine-learning-competitions-2025/
• Winning Tips from Kazanova (#3 on Kaggle): hackerearth.com/practice/machine-learning/advanced-techniques/winning-tips-machine-learning-competitions-kazanova-current-kaggle-3/tutorial/