Summary (Overview)

• Novel PEFT paradigm for brain encoding: The paper introduces BrainLoRA, a parameter-efficient fine-tuning approach that adapts large pretrained multimodal backbones (up to 104B parameters) for brain encoding, retaining strong pretrained priors while enabling neural adaptation through low-rank adapters.

• 350× computational speedup: By exploiting causal masking in the backbone, the method replaces overlapping sliding windows with single-pass feature extraction, reducing feature-extraction cost by approximately 350× compared to prior state-of-the-art (MIRAGE).

• New state-of-the-art performance: BrainLoRA achieves the best published performance on the Algonauts 2025 Challenge using only ~5% as many trainable parameters as the prior SoTA (61.2M vs. 1.1B), reaching 0.327 ID and 0.228 OOD Pearson correlation.

• Interpretable adaptations: Modality attribution via exact Shapley values recovers canonical sensory topography (visual cortex driven by video, auditory cortex by audio), with adaptation increasing audio contribution and cross-modal mixing.

• Scalable ensemble via multi-LoRA kernel: A batched training kernel enables joint training of 200+ adapters on a shared backbone, achieving first place on the public Algonauts post-challenge leaderboard.

---

Introduction and Theoretical Foundation

Brain encoding models predict neural responses to stimuli, providing computational insights into brain information processing. The dominant paradigm is goal-driven: a large model pretrained on behavioral tasks serves as a frozen feature extractor, with only a readout learned from neural data. While this leverages strong pretrained priors, it misses the opportunity to specialize the backbone for brain encoding.

Key challenge: Neural data are costly and scarce relative to contemporary model sizes. Prior attempts at adaptation have had mixed success:

• Training smaller models from scratch (St-Yves et al., 2023; Khosla and Wehbe, 2026) has not reached SoTA on whole-brain benchmarks
• Full fine-tuning of pretrained backbones has been reported to underperform freezing (Yang et al., 2023)
• Low-rank adaptation won Algonauts 2023 but only with a relatively small ViT

Theoretical framework: The paper formalizes encoding models as:

$$\hat{Y} = g_{\phi}\big(f_{\theta}(X)\big), \quad \min_{\theta \in \Theta, \phi} \mathcal{L}\big(Y, \hat{Y}\big), \tag{1}$$

where $X$ is the stimulus, $Y$ the neural response, $f_{\theta}$ the pretrained backbone, and $g_{\phi}$ the readout that performs temporal alignment.

The optimization space is characterized as:

$$\Theta_{\mathrm{frozen}} = \{\theta_{0}\} \subset \Theta_{\mathrm{PEFT}} = \{\theta_{0} + \Delta(\psi): \psi \in \mathbb{R}^{k}\} \subset \Theta_{\mathrm{full}} = \mathbb{R}^{d}, \qquad k \ll d, \Delta(0) = 0. \tag{2}$$

This formulation suggests three complementary scaling axes:

1. Scale up: Leverage increasingly capable pretrained backbones ($\theta_0$)
2. Scale down: Minimize adaptation parameters (dim $\psi$)
3. Scale out: Train K adaptations $\{(\psi_k, \phi_k)\}_{k=1}^{K}$ on shared backbone

---

Methodology

Task and Data

• Benchmark: CNeuroMod/Algonauts 2025 Challenge (whole-brain fMRI from 4 subjects watching Friends)
• Stimuli: Synchronized video, audio, and transcript
• Evaluation: Held-out Season 7 (ID) and movies (OOD)
• Main backbone: Qwen3-Omni (same as prior SoTA for controlled comparison)

Single-Pass Feature Extraction

Prior approaches used overlapping sliding windows (2,800 forward passes per clip), causing substantial redundant computation. Since the backbone is causally masked:

"We encode each 149 s window (100 TRs) with a single forward pass and extract representations for all timesteps simultaneously"

This reduces feature-extraction cost from ~700 to 2 GPU-hours on A800 (≈350× speedup). A 60s context lead recovers most of the small performance drop from missing stimulus context at window starts.

BrainLoRA Architecture

• LoRA: Rank-16 adapters inserted into attention modules through layer 32 (8.91M trainable parameters)
• Readout: Linear or cross-attention readouts pooling features into one vector per modality at 2 Hz, aggregated into eight log-spaced temporal-lag blocks aligned with fMRI sampling, mapped to 1,000 brain parcels (52.3M parameters)
• Closed-form re-fit: After SGD training, the linear readout W is re-solved using parcel-wise ridge regression (Nunez-Elizalde et al., 2019)

Multi-LoRA Training Kernel

A specialized batched gather–matrix multiplication fuses routed updates for N adapters:

• At N=32, fused step costs only 1.08× a single-adapter step
• Yields 29.8× speedup over sequential training

---

Empirical Validation / Results

Main Results (Table 1)

Key finding: A small adapter (61.2M params, ~5% of MIRAGE) beats a 1.1B-parameter readout on both ID and OOD test sets.

Adapter Gains Across Backbones

• LoRA consistently improves all nine multimodal backbones tested (4.7B–104B parameters)
• Improvements range from +0.032 to +0.066 on linear readouts
• Cross-attention yields an additional +0.003 gain on average
• Largest gains in parietal and visual areas

Ablation Results (Table 2)

The LoRA adapter provides the largest improvement (+0.0356), with closed-form re-fit contributing +0.0061.

Interpretability Results

• Canonical topography recovered: Early visual cortex driven by video (72.5%), auditory cortex by audio (72.9%)
• Adaptation shifts toward audio: Audio contribution rises 39%, text falls 17%; 249 parcels switch dominant modality (246 from video→audio)
• Cross-modal mixing nearly doubles: From 8.9% (frozen) to 15.3% (adapted) of prediction attributed to information read out at other modalities' token positions

Ensemble Scaling

• Ensemble accuracy: 0.3483 val, 0.3428 ID, 0.2483 OOD (currently #1 on public leaderboard)
• Scaling law: $m(n) = m_{\infty} - An^{-b}$ with $b \approx 0.94$ ($R^2 = 0.998$), consistent with ensemble theory: residual variance is $\sigma^2[\rho + (1-\rho)/n]$
• Mean residual correlation and mean single-model performance explain 97% of variance in ensemble score jointly (5% and 23% individually)
• Useful diversity comes primarily from architecture changes (readout/backbone), not minor training variations (seed, rank, layer choice)

---

Theoretical and Practical Implications

Theoretical Implications

1. PEFT theory for brain encoding: The paper formally characterizes PEFT as an intermediate regime between frozen and full fine-tuning, showing that low-dimensional adaptation ($k \ll d$) suffices to specialize powerful backbones with scarce neural data.

2. Full fine-tuning does not outperform LoRA: At this data scale, full fine-tuning (20.3B params) performs on par with or below LoRA (61.2M) — consistent with prior work showing that overly flexible adaptation overfits limited neural data.

3. Adaptation encodes generalizable representations: The adapter transfers to unseen subjects (retaining 97% of improvement) and zero-shot to a different dataset (NNDb, +11.1 points of noise ceiling), suggesting adapted representations capture general stimulus-processing principles rather than subject-specific artifacts.

4. Ensemble theory validated: The performance scaling is governed by residual correlation $\rho$ between ensemble members, not just ensemble size — with architecture-level diversity being the primary driver of useful diversity.

Practical Implications

• Computational accessibility: The 350× speedup makes large-backbone adaptation affordable (2 GPU-hours vs. 700), democratizing access to SoTA brain encoding.
• Efficient ensemble training: The multi-LoRA kernel enables population-scale adaptation at near single-model cost.
• Interpretable multimodal analysis: Shapley-based modality attribution provides a principled alternative to unimodal encoder construction for identifying region-preferred modalities.

---

Conclusion

This work establishes parameter-efficient fine-tuning (PEFT) as a powerful and efficient paradigm for brain encoding, addressing the fundamental challenge of scarce neural data relative to model capacity.

Three scaling directions for future brain encoding models:

1. Scale up to stronger pretrained priors (up to 104B parameters demonstrated)
2. Scale down adaptation to minimal trainable parameters
3. Scale out to populations of specialized adapters on shared backbones

The combination of these directions — exploiting a single powerful pretrained model to create a family of specialized encoding models without training each from scratch — points toward a brain foundation model that could use a shared adapter trained on neural data spanning tasks, datasets, subjects, and even species.

Limitations: Models trained on one benchmark (four subjects, narrative film); transfer tests remain within the domain of narrative film. Future work includes joint training on multiple datasets and modalities.

Key takeaway: PEFT transforms the trade-off between strong priors and neural adaptation — small, efficiently trained adapters consistently outperform both frozen backbones and full fine-tuning, while remaining interpretable and scalable to ensemble populations.