Deep Learning Competition Guide — Vision & NLP

---

PART 1 — COMPUTER VISION COMPETITIONS

Backbone Selection (2025 State of the Art)

Backbone	Best For	Notes
DINOv2 / ViT	Natural images, fine-grained classification	Vision Transformers surpassed CNNs for first time in 2025 competitions
Swin Transformer	Hierarchical vision tasks	Strong for detection/segmentation with PVT-style features
ConvNeXt	Best CNN for natural images	Outperforms EfficientNet on most natural image benchmarks
EfficientNet	Remote sensing, medical imaging, plant datasets	Most versatile CNN across diverse domains
RegNet	Competitive across diverse image types	Good alternative to EfficientNet
YOLOv8/v11	Object detection	Standard for detection tasks; ResNet/EfficientNet declining here
U-Net family	Segmentation	Still dominant; pair with ConvNeXt or EfficientNet encoder

Rule: For low-data fine-tuning → use pure CNNs (ConvNeXt, EfficientNet). Transformers need more data.

Library: Use timm for all pretrained backbone access.

import timm
model = timm.create_model('convnext_base', pretrained=True, num_classes=NUM_CLASSES)
# List all available models
timm.list_models('efficientnet*')

---

Augmentation Strategy

Standard augmentations (always include):

import albumentations as A
from albumentations.pytorch import ToTensorV2

train_transform = A.Compose([
    A.RandomResizedCrop(height=224, width=224, scale=(0.8, 1.0)),
    A.HorizontalFlip(p=0.5),
    A.VerticalFlip(p=0.5),
    A.RandomRotate90(p=0.5),
    A.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2, hue=0.1, p=0.5),
    A.GaussianBlur(p=0.2),
    A.GaussNoise(p=0.2),
    A.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
    ToTensorV2(),
])

Advanced augmentations:

# CutMix — best for classification and localization tasks
def cutmix(data, target, alpha=1.0):
    indices = torch.randperm(data.size(0))
    lam = np.random.beta(alpha, alpha)
    
    # Generate random box
    W, H = data.size(3), data.size(2)
    cut_w = int(W * np.sqrt(1 - lam))
    cut_h = int(H * np.sqrt(1 - lam))
    cx = np.random.randint(W)
    cy = np.random.randint(H)
    bbx1 = np.clip(cx - cut_w // 2, 0, W)
    bby1 = np.clip(cy - cut_h // 2, 0, H)
    bbx2 = np.clip(cx + cut_w // 2, 0, W)
    bby2 = np.clip(cy + cut_h // 2, 0, H)
    
    data[:, :, bby1:bby2, bbx1:bbx2] = data[indices, :, bby1:bby2, bbx1:bbx2]
    lam = 1 - ((bbx2 - bbx1) * (bby2 - bby1) / (W * H))
    target_a, target_b = target, target[indices]
    return data, target_a, target_b, lam

# Mixup — best for classification, avoid for detection/segmentation
def mixup(data, target, alpha=0.4):
    lam = np.random.beta(alpha, alpha)
    indices = torch.randperm(data.size(0))
    mixed_data = lam * data + (1 - lam) * data[indices]
    return mixed_data, target, target[indices], lam

When to use each:

Augmentation	Classification	Detection	Segmentation
CutMix	✓ Best	✓ Good	✓ Good
Mixup	✓ Good	✗ Avoid	✗ Avoid
Cutout/Erasing	✓ Good	✓ Good	✓ Good
Standard flips/rotations	✓ Always	✓ Always	✓ Always

---

Test-Time Augmentation (TTA)

def predict_with_tta(model, image, n_augments=5):
    """Generate multiple predictions, then average."""
    model.eval()
    predictions = []
    
    # Original
    with torch.no_grad():
        predictions.append(torch.softmax(model(image), dim=1))
    
    # Flipped versions
    predictions.append(torch.softmax(model(torch.flip(image, [3])), dim=1))  # H flip
    predictions.append(torch.softmax(model(torch.flip(image, [2])), dim=1))  # V flip
    
    # Averaged
    return torch.stack(predictions).mean(0)

TTA typically improves scores by 0.5–2% with zero training cost.

---

Small Dataset Strategies

• Use 7–10 CV folds (more stable estimates)
• Prefer CNNs over transformers (less data-hungry)
• Apply stronger augmentation pipelines
• Freeze early backbone layers
• Use pretrained features from ImageNet / CLIP / DINOv2
• Knowledge distillation: train from a larger pretrained teacher

Large Dataset Strategies

• Use 3–5 CV folds to save compute
• Fine-tune full transformer backbone
• Generate synthetic data with diffusion models
• Use semi-supervised approaches with pseudo-labeling

---

PART 2 — NLP COMPETITIONS

Model Selection (2025–2026)

Use Case	Best Model	Notes
Classification / regression (≤512 tokens)	DeBERTa-v3-large	Still dominant encoder for structured NLP tasks; 300M params
General NLP reasoning	Qwen2.5-7B / Qwen3-8B	Qwen won ALL THREE major NLP Kaggle grand prizes in 2025
Long-context reasoning	Qwen2.5-72B / Gemma2-27B	For tasks needing large context or complex chain-of-thought
Math / reasoning competitions	Qwen2.5-Math-72B	Specialized for AIMO-style mathematical reasoning
Instruction following (smaller)	Llama3-8B / Mistral-7B	Good but Qwen now preferred for competitions
Most winning models (2025–2026)	7–9B Qwen range	Decoder models dominate; encoder-only (BERT) nearly gone from top solutions

The structural shift: Encoder-only models (BERT, DeBERTa) are nearly gone from winning NLP solutions. Decoder-only models with LoRA fine-tuning + 4-bit inference are now the standard.

For NLP competitions: Qwen2.5-7B → fine-tune with Unsloth + LoRA → infer with vLLM. This is the 2025 gold-medal stack.

---

Fine-Tuning Tricks

Partial fine-tuning (most efficient):

# Fine-tune only last 6 transformer blocks
model = AutoModel.from_pretrained('microsoft/deberta-v3-large')
for name, param in model.named_parameters():
    param.requires_grad = False  # Freeze all first

# Unfreeze last 6 blocks
for i in range(18, 24):  # DeBERTa has 24 layers
    for param in model.encoder.layer[i].parameters():
        param.requires_grad = True
# Always unfreeze the pooler
for param in model.pooler.parameters():
    param.requires_grad = True

Adversarial Weight Perturbation (AWP) — adds ~0.001–0.01 CV improvement:

class AWP:
    def __init__(self, model, optimizer, adv_param="weight", adv_lr=0.1, adv_eps=1e-4):
        self.model = model
        self.optimizer = optimizer
        self.adv_param = adv_param
        self.adv_lr = adv_lr
        self.adv_eps = adv_eps
        self.backup = {}
        self.backup_eps = {}

    def attack_backward(self, inputs, labels):
        self._save()
        self._attack_step()
        with torch.cuda.amp.autocast():
            adv_loss = self.model(**inputs).loss
        self.optimizer.zero_grad()
        adv_loss.backward()
        self._restore()
        return adv_loss

    def _save(self):
        for name, param in self.model.named_parameters():
            if param.requires_grad and self.adv_param in name:
                self.backup[name] = param.data.clone()

    def _attack_step(self):
        e = 1e-6
        for name, param in self.model.named_parameters():
            if param.requires_grad and self.adv_param in name:
                norm = torch.norm(param.grad) + e
                r_at = self.adv_lr * param.grad / norm
                param.data.add_(r_at)
                param.data = self._project(name, param.data)

    def _restore(self):
        for name, param in self.model.named_parameters():
            if name in self.backup:
                param.data = self.backup[name]

LoRA / QLoRA for 7B+ models:

from peft import LoraConfig, get_peft_model, TaskType

lora_config = LoraConfig(
    task_type=TaskType.SEQ_CLS,
    r=16,               # Rank — higher = more parameters
    lora_alpha=32,      # Scaling factor
    lora_dropout=0.1,
    target_modules=["q_proj", "v_proj"],  # Which layers to apply LoRA
)

model = get_peft_model(base_model, lora_config)
model.print_trainable_parameters()
# Output: trainable params: 4,194,304 || all params: 6,742,609,920 || trainable%: 0.06%

---

Pooling Strategies

class MeanPooling(nn.Module):
    def forward(self, last_hidden_state, attention_mask):
        mask = attention_mask.unsqueeze(-1).expand(last_hidden_state.size()).float()
        return torch.sum(last_hidden_state * mask, 1) / torch.clamp(mask.sum(1), min=1e-9)

class AttentionPooling(nn.Module):
    def __init__(self, hidden_size):
        super().__init__()
        self.attention = nn.Linear(hidden_size, 1)
    
    def forward(self, last_hidden_state, attention_mask):
        scores = self.attention(last_hidden_state).squeeze(-1)
        scores = scores.masked_fill(attention_mask == 0, -1e9)
        weights = torch.softmax(scores, dim=1).unsqueeze(-1)
        return (last_hidden_state * weights).sum(1)

# Try all three and ensemble them:
# 1. CLS token:      output = last_hidden_state[:, 0, :]
# 2. Mean pooling:   output = MeanPooling()(last_hidden_state, attention_mask)
# 3. Attention pool: output = AttentionPooling(hidden_size)(last_hidden_state, attention_mask)

---

Synthetic Data Generation

When training data is small, generate more with LLMs:

# Prompt template for generating synthetic training data
SYNTH_PROMPT = """
Generate {n} examples similar to the following training samples.
Each example should be in the same format.

Examples:
{examples}

Generate {n} new examples:
"""

# After generation: fine-tune DeBERTa on real + synthetic combined
# This technique jumped one team from rank 49 → rank 1 on final leaderboard

---

NLP Pseudo-Labeling (Multi-Round)

def multi_round_pseudo_label(model, unlabeled_loader, train_dataset, n_rounds=3, threshold=0.95):
    """Multi-round pseudo-labeling — used in ~26% of top-3 NLP solutions."""
    for round_num in range(n_rounds):
        # Generate predictions on unlabeled data
        pseudo_labels = []
        pseudo_probs = []
        with torch.no_grad():
            for batch in unlabeled_loader:
                logits = model(**batch).logits
                probs = torch.softmax(logits, dim=-1)
                pseudo_labels.append(probs.argmax(dim=-1))
                pseudo_probs.append(probs.max(dim=-1).values)
        
        # Keep only high-confidence pseudo-labels
        mask = torch.cat(pseudo_probs) > threshold
        # Add to training set and retrain...
        print(f"Round {round_num+1}: Added {mask.sum()} pseudo-labeled samples")

---

NLP Ensemble Methods

import numpy as np
from scipy.optimize import minimize

# Simple average (strong baseline)
ensemble_pred = np.mean([pred1, pred2, pred3, pred4], axis=0)

# Optimized weights via Nelder-Mead (can discover negative weights)
def neg_score(weights, preds, labels):
    weights = np.array(weights)
    weights = weights / weights.sum()
    blended = np.average(preds, axis=0, weights=weights)
    return -compute_metric(labels, blended)  # replace with your metric

result = minimize(
    neg_score,
    x0=np.ones(len(preds)) / len(preds),
    args=(np.array(preds), labels),
    method='Nelder-Mead'
)
optimal_weights = result.x / result.x.sum()

---

PART 3 — UNIVERSAL DEEP LEARNING RULES

1. Start simple, scale up. ResNet50/DeBERTa-base before ViT/DeBERTa-large. Know what you're comparing against.
2. AWP almost always helps for NLP tasks. Apply from epoch 2 onward.
3. Ensemble at least 3–5 models trained with different seeds and/or augmentations.
4. TTA is free performance — always apply at inference.
5. Pseudo-labeling in multiple rounds consistently outperforms single-pass.
6. Log everything with wandb. You will forget what worked without it.
7. Train on full data for final submissions — CV folds are for validation only.
8. CutMix outperforms Mixup for most classification and all localization tasks.
9. Linear warmup is non-negotiable for transformer training.
10. Quantize for inference — 4-bit/8-bit quantization for 7B+ models is standard.

---

Key Tools

Tool	Purpose
timm	Pretrained vision backbones
albumentations	Image augmentation pipeline
transformers	Hugging Face NLP models
peft	LoRA / QLoRA for large models
unsloth	Efficient fine-tuning — 14B model in 16GB GPU; 3 gold medals 2025
vLLM	Fast LLM inference with speculative decoding — 4 winning solutions 2025
wandb	Experiment tracking
optuna	Hyperparameter optimization
MONAI	Medical imaging DL framework — increasingly standard for medical CV
librosa	Audio feature extraction for BirdCLEF / audio competitions
torchmetrics	100+ PyTorch metrics — replaces manual metric code