Summary of: Interpretable MEG Decoding of Perceived Speech: Cortical Sources and the Stimulus Features that Drive Retrieval

Summary (Overview)

• Interpretable architecture with physical priors: The authors build a MEG-to-speech retrieval decoder whose front end is constrained by the physics of MEG measurement (spherical-harmonic spatial attention) and physiology (factorized spatial–temporal branches), replacing the opaque 2D Fourier attention of prior work (Défossez et al.) with a physically grounded parameterization and adding trainable temporal filters matched to neuronal sources.

• Competitive accuracy with 20× fewer parameters: The model achieves 39.75 ± 0.34% Top-1 and 70.40 ± 0.31% Top-10 accuracy among 1005 candidates on MEG-MASC, using only 486,619 trainable parameters (~20× fewer than the baseline brain decoder), while a larger K=270, five-block configuration scores 3.60 percentage points lower in Top-1 despite 14.8× more parameters.

• Cortical source mapping: Learned weights map to source space, recovering generators consistent with the canonical speech-perception network—bilateral superior temporal gyrus, left frontal operculum/inferior frontal cortex, and medial temporal structures—with a striking hemispheric spectral asymmetry: left-lateralized branches carry an additional ~13.3 Hz component not evident on the right, consistent with the asymmetric sampling-in-time account.

• Paired occlusion identifies 15 of 19 contributing stimulus features: The largest effects come from silence (Δr = 75.62), high loudness (60.77), vowels (38.70), and acoustic onsets (36.19); notably, random word lists behave oppositely—narrative MEG substituted into them improves retrieval, suggesting narrative structure supports neural tracking rather than merely providing anomaly responses.

• Asymmetric compression of the target embedding: The 768-dimensional wav2vec feature axis can be reduced to ~12 learned dimensions without loss, but temporal compression (especially global pooling) collapses retrieval to near chance, revealing that the temporal trajectory of speech carries irreplaceable segment-identifying information.

Introduction and Theoretical Foundation

Background and Motivation

Deep networks can retrieve short segments of perceived speech from MEG with high accuracy, but the learned weights "match nothing an electrophysiologist can name: not a cortical location, not a rhythm, not a time course." The authors quote d'Ascoli et al.: "it is unclear whether the decoder relies on the perceptual characteristics of the speech segment or the semantic features of the underlying words."

The paper argues that decoding models should serve as scientific instruments, not just benchmarks. Classical speech-perception studies offer excellent control but suit continuous natural speech poorly. Machine learning offers the reverse approach: set a hard decoding task on natural recordings, then examine the learned decision rule.

MEG Generative Model

The theoretical foundation rests on the biophysical generative model of MEG. The main cortical contributors are intracellular currents in synchronously active pyramidal neurons, approximated by equivalent current dipoles (ECDs) with location $\mathbf{r}_n = [x_n, y_n, z_n]^\top$, orientation $\boldsymbol{\theta}_n$, and time-varying amplitude $s_n(t)$. The measured sensor signal is:

$$\mathbf{x}(t) = \sum_{n=1}^{N} \mathbf{g}_n s_n(t) + \sum_{u=1}^{N_{\mathrm{u}}} \mathbf{q}_u p_u(t) + \mathbf{n}(t), \tag{1}$$

where $\mathbf{g}_n$ is the $M \times 1$ gain vector (topography) of the $n$-th task-related source, $\mathbf{q}_u$ are task-unrelated source topographies (spatially-correlated brain noise), and $\mathbf{n}(t)$ is spatially-white sensor noise. All information about the behavioral variable $z(t)$ is confined to the subspace $S = \mathrm{span}\{\mathbf{g}_1, \ldots, \mathbf{g}_N\}$.

Classical Spatial Filtering

A spatial filter $\mathbf{w}_k^\top$ applied to channel time series estimates source activity:

$$\hat{s}_k(t) = \mathbf{w}_k^\top \mathbf{x}(t) = \sum_{n=1}^{N} \mathbf{w}_k^\top \mathbf{g}_n s_n(t) + \sum_{u=1}^{N_{\mathrm{u}}} \mathbf{w}_k^\top \mathbf{q}_u p_u(t) + \mathbf{w}_k^\top \mathbf{n}(t). \tag{2}$$

The optimal filter must align with the target topography ($\mathbf{w}_k^\top \mathbf{g}_n > 0$ for $k=n$) while tuning away from all others. The optimal filter matrix is:

$$\mathbf{W} = \mathbf{R}_\mathbf{x}^{-1} \mathbf{G} \mathbf{R}_s, \tag{3}$$

where $\mathbf{R}_\mathbf{x}$ is the data covariance and $\mathbf{R}_s$ the source covariance. Three strategies exist: exhaustive search (inverse operators), optimal filtering (task-driven but limited expressivity), and deep learning (powerful but "black box").

Interpretable Factorized Architectures

The key theoretical contribution builds on Petrosyan et al. [4], who showed that in factorized spatial–temporal architectures, both filter types adapt concurrently, so interpretation must account for their mutual dependence. Each branch operates as:

$$r_k(t) = f\big(\mathbf{w}_k^\top \mathbf{x}(t) * h_k(t)\big), \tag{4}$$

where $h_k(t)$ is the T-tap temporal filter. The spatial pattern of the $k$-th source is:

$$\hat{\mathbf{g}}_k \propto \mathbf{R}_{h_k} \mathbf{w}_k, \tag{5}$$

where $\mathbf{R}_{h_k}$ is the covariance of data filtered by the branch's temporal filter. Source localization uses MNE:

$$\mathbf{W}_{\mathrm{MNE}} = \mathbf{G}_{\mathrm{all}}^\top \left(\mathbf{G}_{\mathrm{all}} \mathbf{G}_{\mathrm{all}}^\top + \lambda^2 \mathbf{I}\right)^{-1}, \tag{6}$$

and the temporal pattern is:

$$\hat{\mathbf{p}}_k \propto \mathbf{R}_{a_k} \mathbf{h}_k, \qquad \mathbf{R}_{a_k} = \mathrm{Cov}[\mathbf{a}_k(t)]. \tag{7}$$

The number of branches $K$ is determined as the minimum count before performance plateaus or degrades, serving as the estimate of $N$ in Eq. (1).

Methodology

Architecture

The model (Figure 2) processes 3-second, 208-channel MEG segments through:

1. 3D spherical-harmonic spatial attention: Each of $J = 270$ virtual channels is parameterized by real spherical harmonics with $L = 24$ (degrees 0–23, 576 basis functions per channel):

$$c_{jm} = \sum_{\ell=0}^{L-1} \sum_{q=-\ell}^{\ell} \gamma_j^{q,\ell} Y_\ell^q(\theta_m, \varphi_m), \tag{8}$$

with softmax normalization $\widetilde{\mathbf{c}}_j = \mathrm{softmax}(\mathbf{c}_j)$ applied as $\mathrm{SA}_j(\mathbf{x}(t)) = \widetilde{\mathbf{c}}_j^\top \mathbf{x}(t)$.

2. Unmixing convolution: A shared $1 \times 1$ convolution ($\mathbf{W}_u \in \mathbb{R}^{270 \times 270}$) applies an affine transformation in the virtual-channel space.

3. Subject-specific projection: Projects to $K = 25$ interpretable branches via participant-specific $\mathbf{W}_s \in \mathbb{R}^{K \times 270}$, giving the effective spatial filter:

$$\mathbf{W}^{(s)} = \mathbf{W}_s \mathbf{W}_u \mathbf{C} \in \mathbb{R}^{K \times 208}. \tag{10}$$

4. Depthwise temporal filters: One trainable 15-sample (150 ms) filter per branch, shared across participants: $r_{s,k}(t) = (a_{s,k} * h_k)(t)$.

5. Non-linear decoder: $B = 2$ residual temporal convolutional blocks with dilation factors $d_{b,1} = 2^{(2b \bmod 5)}$, $d_{b,2} = 2^{((2b+1) \bmod 5)}$, followed by a convolutional head projecting to $F = 768$ wav2vec feature channels.

Training and Data

• Dataset: MEG-MASC, 27 participants (22 with two sessions), 49 hour-long recordings. Audio resampled to 16 kHz, MEG from 1000 to 100 Hz. Ocular and cardiac ICA components were removed—critical because eye activity tracks linguistic structure and cardiac dynamics vary with narrative intensity, either could supply stimulus-locked shortcut information.
• Target: wav2vec 2.0 Base embeddings (average of final four hidden layers). MEG segments paired with audio starting 150 ms later (auditory latency).
• Objective: One-directional MEG-to-audio contrastive cross-entropy loss (CLIP-style), with careful handling of duplicate audio targets in minibatches.
• Optimization: AdamW, learning rate $3 \times 10^{-4}$, early stopping with patience 7, validation-based checkpoint selection.

Paired MEG Occlusion Analysis

The core interpretability intervention replaces feature-marked MEG intervals with matched real-MEG donors in two conditions:

$$X^{f \rightarrow 0}: \text{replacement by MEG from a feature-absent interval}, \tag{18}$$ $$X^{f \rightarrow f}: \text{replacement by MEG from a feature-present interval}. \tag{19}$$

The feature-present arm controls for generic replacement damage. Nineteen features were tested: silence, silence onsets/offsets, seven phoneme classes, word onsets, pseudowords, random word lists, surprisal, predictive entropy, word frequency, loudness, and acoustic-onset strength. The rank contrast per feature is:

$$E_{sqf} = \frac{1}{5|\mathcal{I}_f|} \sum_{i \in \mathcal{I}_f} \sum_{j=1}^{5} \left(r_{s q i j}^{f \rightarrow 0} - r_{s q i j}^{f \rightarrow f}\right), \tag{20}$$

with statistical inference via the studentized participant mean:

$$T_f = \frac{\overline{D}_f}{\mathrm{SD}(D_{1f}, \ldots, D_{Sf}) / \sqrt{S}}, \qquad S = 27, \tag{22}$$

tested with 100,000 participant sign-flip permutations and one-sided single-step max-T familywise-error correction:

$$p_f^{\mathrm{FWER}} = \frac{1 + \sum_{b=1}^{N_{\mathrm{perm}}} \mathbf{1}[M^{(b)} \geq T_f]}{N_{\mathrm{perm}} + 1}. \tag{23}$$

A feature was deemed significant only if: (1) mean contrast positive, (2) $p_{\mathrm{FWER}} < 0.05$, and (3) the feature-present control mean rank was below the random expectation of 503.

Empirical Validation / Results

Decoding Accuracy

Note: Direct comparison is confounded by different evaluation protocols—prior work aligned test windows to word onsets; the present work used fixed-stride windows, a harder setting.

The branch-count sweep shows sharp improvement from very small K to a plateau at K ≈ 10–25, with mild decline at larger K—evidence that the task-related subspace is compact.

Segment Duration Effects

Top-1 accuracy increased monotonically with segment duration for the main two-block model: 14.37% at 1.5s → 28.03% at 2.25s → 39.94% at 3s → 52.66% at 4s → 62.20% at 5s. Top-10 reached 86.33% at 5s. The same progression held for zero and five convolutional blocks.

Paired Occlusion Results

15 of 19 features showed significant positive occlusion effects (all with $p_{\mathrm{FWER}} < 10^{-4}$ except pseudowords at $p = 0.022$):

Not significant: predictive entropy (p=0.069), silence onsets (p=0.316), rare words (−2.98), and random word lists (−17.77, negative). The negative word-list effect was reproduced in all six trained models and indicates that narrative structure supports neural tracking—removing it reduces recoverable information even though individual words remain.

Cortical Source Localization

• RAP-MUSIC dipole fitting on the top-ten singular vectors of the cross-subject spatial-pattern matrix localized sources in bilateral auditory cortices, medial temporal lobe, and frontal lobe.
• Clustering of 675 participant–branch items (27 participants × 25 branches) yielded 49 clusters; the 12 largest organized into three groups:
  1. Bilateral superior temporal gyrus/plane (speech envelope tracking)
  2. Posterior/superior toward supramarginal region and central sulcus (dorsal-stream phonological/sensorimotor sites)
  3. Left frontal operculum/inferior frontal cortex plus anterior middle/inferior temporal foci (predictive reconciliation and auditory–conceptual integration)
• Hemispheric spectral asymmetry: Left-lateralized branches show a dominant slow peak below 10 Hz plus a second component at ~13.3 Hz; right-lateralized branches are single-peaked around 6.6 Hz—consistent with the asymmetric sampling-in-time account of auditory lateralization.

Compression Experiments

• Feature-axis compression: Reducing wav2vec features from 768 to 12 learned dimensions (LinearDR) leaves retrieval essentially unchanged; PCA degrades much faster, showing the MEG-relevant subspace is task-selected, not variance-dominated.
• Temporal-axis compression: Severe temporal compression causes clear performance loss; global pooling (mean, max, additive/gated attention) collapses to near chance. The temporal trajectory is irreplaceable.

Ablations

• Removing subject-conditioned spatial mappings: largest drop
• Removing spatial attention entirely: 4–5