# Scaling Parameter-Efficient Fine-Tuning for Brain Encoding

> BrainLoRA adapts 104B-parameter multimodal backbones for brain encoding with 5% of the parameters and 350× faster feature extraction, achieving state-of-the-art accuracy.

- **Source:** [arXiv](https://arxiv.org/abs/2610.26802)
- **Published:** 2026-10-21
- **Permalink:** https://picx.dev/p/yNx3na
- **Whiteboard:** https://picx.dev/p/yNx3na/image

## Summary

## Summary (Overview)

- **Novel PEFT paradigm for brain encoding**: The paper introduces BrainLoRA, a parameter-efficient fine-tuning approach that adapts large pretrained multimodal backbones (up to 104B parameters) for brain encoding, retaining strong pretrained priors while enabling neural adaptation through low-rank adapters.

- **350× computational speedup**: By exploiting causal masking in the backbone, the method replaces overlapping sliding windows with single-pass feature extraction, reducing feature-extraction cost by approximately 350× compared to prior state-of-the-art (MIRAGE).

- **New state-of-the-art performance**: BrainLoRA achieves the best published performance on the Algonauts 2025 Challenge using only ~5% as many trainable parameters as the prior SoTA (61.2M vs. 1.1B), reaching 0.327 ID and 0.228 OOD Pearson correlation.

- **Interpretable adaptations**: Modality attribution via exact Shapley values recovers canonical sensory topography (visual cortex driven by video, auditory cortex by audio), with adaptation increasing audio contribution and cross-modal mixing.

- **Scalable ensemble via multi-LoRA kernel**: A batched training kernel enables joint training of 200+ adapters on a shared backbone, achieving first place on the public Algonauts post-challenge leaderboard.

---

## Introduction and Theoretical Foundation

Brain encoding models predict neural responses to stimuli, providing computational insights into brain information processing. The dominant paradigm is goal-driven: a large model pretrained on behavioral tasks serves as a frozen feature extractor, with only a readout learned from neural data. While this leverages strong pretrained priors, it misses the opportunity to specialize the backbone for brain encoding.

**Key challenge**: Neural data are costly and scarce relative to contemporary model sizes. Prior attempts at adaptation have had mixed success:
- Training smaller models from scratch (St-Yves et al., 2023; Khosla and Wehbe, 2026) has not reached SoTA on whole-brain benchmarks
- Full fine-tuning of pretrained backbones has been reported to underperform freezing (Yang et al., 2023)
- Low-rank adaptation won Algonauts 2023 but only with a relatively small ViT

**Theoretical framework**: The paper formalizes encoding models as:

$$\hat{Y} = g_{\phi}\big(f_{\theta}(X)\big), \quad \min_{\theta \in \Theta, \phi} \mathcal{L}\big(Y, \hat{Y}\big), \tag{1}$$

where $X$ is the stimulus, $Y$ the neural response, $f_{\theta}$ the pretrained backbone, and $g_{\phi}$ the readout that performs temporal alignment.

The optimization space is characterized as:

$$\Theta_{\mathrm{frozen}} = \{\theta_{0}\} \subset \Theta_{\mathrm{PEFT}} = \{\theta_{0} + \Delta(\psi): \psi \in \mathbb{R}^{k}\} \subset \Theta_{\mathrm{full}} = \mathbb{R}^{d}, \qquad k \ll d, \Delta(0) = 0. \tag{2}$$

This formulation suggests **three complementary scaling axes**:
1. **Scale up**: Leverage increasingly capable pretrained backbones ($\theta_0$)
2. **Scale down**: Minimize adaptation parameters (dim $\psi$)
3. **Scale out**: Train K adaptations $\{(\psi_k, \phi_k)\}_{k=1}^{K}$ on shared backbone

---

## Methodology

### Task and Data
- **Benchmark**: CNeuroMod/Algonauts 2025 Challenge (whole-brain fMRI from 4 subjects watching Friends)
- **Stimuli**: Synchronized video, audio, and transcript
- **Evaluation**: Held-out Season 7 (ID) and movies (OOD)
- **Main backbone**: Qwen3-Omni (same as prior SoTA for controlled comparison)

### Single-Pass Feature Extraction
Prior approaches used overlapping sliding windows (2,800 forward passes per clip), causing substantial redundant computation. Since the backbone is causally masked:

> "We encode each 149 s window (100 TRs) with a single forward pass and extract representations for all timesteps simultaneously"

This reduces feature-extraction cost from ~700 to 2 GPU-hours on A800 (≈350× speedup). A 60s context lead recovers most of the small performance drop from missing stimulus context at window starts.

### BrainLoRA Architecture
- **LoRA**: Rank-16 adapters inserted into attention modules through layer 32 (8.91M trainable parameters)
- **Readout**: Linear or cross-attention readouts pooling features into one vector per modality at 2 Hz, aggregated into eight log-spaced temporal-lag blocks aligned with fMRI sampling, mapped to 1,000 brain parcels (52.3M parameters)
- **Closed-form re-fit**: After SGD training, the linear readout W is re-solved using parcel-wise ridge regression (Nunez-Elizalde et al., 2019)

### Multi-LoRA Training Kernel
A specialized batched gather–matrix multiplication fuses routed updates for N adapters:
- At N=32, fused step costs only 1.08× a single-adapter step
- Yields 29.8× speedup over sequential training

---

## Empirical Validation / Results

### Main Results (Table 1)

| Model | Trainable params | val | ID | OOD |
|-------|-----------------|-----|-----|-----|
| **Single model** | | | | |
| TRIBE† (d'Ascoli et al., 2026) | - | - | 0.303 | 0.196 |
| MIRAGE† (Gokce et al., 2026a) | 1.1B | 0.319 | 0.310 | 0.217 |
| Linear + frozen backbone | 52.3M | 0.291 | 0.287 | 0.187 |
| BRAINLoRA (Linear) | 61.2M | 0.326 | 0.322 | 0.222 |
| BRAINLoRA (Cross-attn) | 61.4M | 0.331 | 0.327 | 0.228 |
| **Ensemble** | | | | |
| MIRAGE†, 15 models | 15×1.1B | 0.335 | 0.323 | 0.227 |
| BRAINLoRA, best 15 models | 15×61M | 0.347 | 0.341 | 0.247 |
| BRAINLoRA, 261 models | 261×61M | 0.348 | 0.343 | 0.248 |

*Key finding*: A small adapter (61.2M params, ~5% of MIRAGE) beats a 1.1B-parameter readout on both ID and OOD test sets.

### Adapter Gains Across Backbones
- **LoRA consistently improves** all nine multimodal backbones tested (4.7B–104B parameters)
- Improvements range from **+0.032 to +0.066** on linear readouts
- Cross-attention yields an additional +0.003 gain on average
- Largest gains in parietal and visual areas

### Ablation Results (Table 2)

| Configuration | s06 |
|---------------|-----|
| Frozen backbone | 0.2907 |
| + LoRA r16 (ours) | 0.3263 |
| without closed-form re-fit | 0.3202 |
| Rank 32 | 0.3268 |
| Cross-attention readout | 0.3310 |
| No audio | 0.3026 |

The **LoRA adapter provides the largest improvement** (+0.0356), with closed-form re-fit contributing +0.0061.

### Interpretability Results
- **Canonical topography recovered**: Early visual cortex driven by video (72.5%), auditory cortex by audio (72.9%)
- **Adaptation shifts toward audio**: Audio contribution rises 39%, text falls 17%; 249 parcels switch dominant modality (246 from video→audio)
- **Cross-modal mixing nearly doubles**: From 8.9% (frozen) to 15.3% (adapted) of prediction attributed to information read out at other modalities' token positions

### Ensemble Scaling
- Ensemble accuracy: 0.3483 val, 0.3428 ID, 0.2483 OOD (currently #1 on public leaderboard)
- Scaling law: $m(n) = m_{\infty} - An^{-b}$ with $b \approx 0.94$ ($R^2 = 0.998$), consistent with ensemble theory: residual variance is $\sigma^2[\rho + (1-\rho)/n]$
- Mean residual correlation and mean single-model performance explain 97% of variance in ensemble score jointly (5% and 23% individually)
- **Useful diversity** comes primarily from architecture changes (readout/backbone), not minor training variations (seed, rank, layer choice)

---

## Theoretical and Practical Implications

### Theoretical Implications
1. **PEFT theory for brain encoding**: The paper formally characterizes PEFT as an intermediate regime between frozen and full fine-tuning, showing that low-dimensional adaptation ($k \ll d$) suffices to specialize powerful backbones with scarce neural data.

2. **Full fine-tuning does not outperform LoRA**: At this data scale, full fine-tuning (20.3B params) performs on par with or below LoRA (61.2M) — consistent with prior work showing that overly flexible adaptation overfits limited neural data.

3. **Adaptation encodes generalizable representations**: The adapter transfers to unseen subjects (retaining 97% of improvement) and zero-shot to a different dataset (NNDb, +11.1 points of noise ceiling), suggesting adapted representations capture general stimulus-processing principles rather than subject-specific artifacts.

4. **Ensemble theory validated**: The performance scaling is governed by residual correlation $\rho$ between ensemble members, not just ensemble size — with architecture-level diversity being the primary driver of useful diversity.

### Practical Implications
- **Computational accessibility**: The 350× speedup makes large-backbone adaptation affordable (2 GPU-hours vs. 700), democratizing access to SoTA brain encoding.
- **Efficient ensemble training**: The multi-LoRA kernel enables population-scale adaptation at near single-model cost.
- **Interpretable multimodal analysis**: Shapley-based modality attribution provides a principled alternative to unimodal encoder construction for identifying region-preferred modalities.

---

## Conclusion

This work establishes **parameter-efficient fine-tuning (PEFT) as a powerful and efficient paradigm for brain encoding**, addressing the fundamental challenge of scarce neural data relative to model capacity.

**Three scaling directions for future brain encoding models**:
1. **Scale up** to stronger pretrained priors (up to 104B parameters demonstrated)
2. **Scale down** adaptation to minimal trainable parameters
3. **Scale out** to populations of specialized adapters on shared backbones

The combination of these directions — exploiting a single powerful pretrained model to create a family of specialized encoding models without training each from scratch — points toward a **brain foundation model** that could use a shared adapter trained on neural data spanning tasks, datasets, subjects, and even species.

**Limitations**: Models trained on one benchmark (four subjects, narrative film); transfer tests remain within the domain of narrative film. Future work includes joint training on multiple datasets and modalities.

**Key takeaway**: PEFT transforms the trade-off between strong priors and neural adaptation — small, efficiently trained adapters consistently outperform both frozen backbones and full fine-tuning, while remaining interpretable and scalable to ensemble populations.

---

_Markdown view of https://picx.dev/p/yNx3na, served by PicX — AI-generated visual whiteboard summaries of research papers._