# Looped Transformers under the Jacobian Lens: Does the Global Workspace Survive Recurrence?

> Looped transformers retain a global workspace, but recurrence reshapes its interfaces: reads and writes follow iteration boundaries, not absolute depth, yet neither model preserves injected content beyond roughly two recurrences.

- **Source:** [arXiv](https://arxiv.org/abs/2609.01924)
- **Published:** 2026-10-31
- **Permalink:** https://picx.dev/p/MesNk1
- **Whiteboard:** https://picx.dev/p/MesNk1/image

## Summary

# Looped Transformers under the Jacobian Lens: Does the Global Workspace Survive Recurrence?

## Summary (Overview)

- **Core Question**: Does a global workspace—a mid-depth band of verbalisable, causally potent representations—emerge in looped/depth-recurrent transformers that reuse weights across depth, as it does in standard feedforward transformers?

- **Key Finding**: Both looped models examined (Ouro-2.6B and Huginn-0125) exhibit the full workspace signature—verbalisable, causally steerable, selectively engaged content—but recurrence fundamentally reshapes how that workspace can be accessed.

- **Two Distinct Maintenance Regimes**: Ouro re-derives workspace content at each supervised checkpoint (destroy-and-re-derive), while Huginn carries content forward across all sixteen recurrences via continuous recomputation from a re-injected input encoding (carry-forward).

- **Access Constraints**: Linear transport fails at Ouro's loop boundaries and beyond a ~2-recurrence horizon on Huginn; writes and ablations must span every remaining loop on Ouro but remain confined to the transport horizon on Huginn.

- **Verbalisation vs. Steering**: Explicit per-iteration supervision predicts whether newly injected content can be verbalised (Ouro: 31% top-1; Huginn: 0/97), while steering existing content requires only head pressure at depth (Huginn: 98% verbal report).

## Introduction and Theoretical Foundation

### Global Workspace Theory

The global-workspace account of cognition (Baars, 1988; Dehaene and Changeux, 2011; Mashour et al., 2020) holds that flexible behaviour depends on a limited-capacity workspace whose contents are broadcast widely, are reportable, and are open to directed modulation. Gurnee et al. (2026) operationalised this for language models using a **Jacobian lens**: per-layer linear maps fitted to the model's own mean input–output Jacobians, identifying a mid-depth band of residual-stream representations with workspace-like properties.

### The Structural Disanalogy

The original study flagged a critical disanalogy with biological theory: in the brain, workspace dynamics are realised by **recurrent processing**—ignition, sustained reverberation, broadcast over re-entrant loops. A standard transformer has no built-in recurrence within a forward pass. The authors noted that a feedforward network can emulate a recurrent one for as many steps as it has layers, but stated plainly that they did not know whether depth in a transformer plays a role similar enough to recurrence to enable the same workspace functionality.

### The Jacobian Lens

The Jacobian lens at source layer $v$ with target layer $t$ is the mean input–output Jacobian of the intervening computation:

$$
J_v = \mathbb{E}\left[\frac{\partial h_t}{\partial h_v}\right], \tag{1}
$$

estimated over a corpus of forward passes. The lens readout at layer $v$ unembeds the linearly transported state, and the lens direction for a token $u$ is the normalised pullback:

$$d = J_v^{\top} u / \|J_v^{\top} u\|$$

Directions support causal edits: injection $(h \leftarrow h + \alpha d)$, ablation $(h \leftarrow h - (h \cdot d)d)$, and clamp-transfer swaps between two tokens' directions.

### Models Studied

| | Params | Physical layers | Virtual depth | Intermediate supervision |
|---|---|---|---|---|
| Qwen3.6-27B | 27B | 64 (untied) | 64 | none |
| Ouro-2.6B | 2.6B | 48, looped ×4 | 192 | deep supervision at every loop end |
| Huginn-0125 | 3.5B | 2 + 4 + 2, core ×r | 68 (r=16) | none (truncated BPTT, mean 8) |

## Methodology

### Virtual Unrolling Adapter

The lens definition extends to weight-tied models by generalising "layer" to **virtual layer**: the $k$-th firing of a physical block is its own node in the unrolled computation graph. For Ouro, $v = 48 \cdot \text{step} + \text{layer}$; for Huginn, the $i$-th core block's $k$-th firing is $v = 2 + 4k + i$.

The adapter presents the unrolled graph to the original fitting machinery by replacing the model's layer list with virtual-block proxies whose forward hooks dispatch only on their assigned firing—without this, a standard activation recorder silently keeps only the last iteration's state. The remaining gradients are computed automatically through between-loop normalisation and tied weights, so $J_v$ is **exact** for the unrolled graph.

### Fit Protocol

All lenses share one protocol: 1000 WikiText-103 prompts of 128 tokens (identical prompt set for all three models), first 16 positions skipped, parameters frozen, per-source mean Jacobians accumulated. Convergence monitored as relative shift of the running mean (final shift ≤ 0.016 on all Huginn lenses).

### Lens Family, Not a Single Lens

The initial scheme treating the unrolled model as one long transformer failed: Ouro's final-target lens reads punctuation throughout loops 1–3 (transport dies crossing loop boundaries), and Huginn's transport horizon is roughly two recurrences regardless of target. The authors therefore fitted a **family of lenses** whose targets are outputs of individual loops or recurrences:

- **Ouro**: every loop end (v47/95/143/191)
- **Huginn**: recurrence ends r=4/8/12/16, completed with gap-fill targets at r=1–15

All readouts are taken through the lens whose target the source can actually reach, with distance-controlled comparisons at matched offsets from targets.

### Three Silent Failure Modes Documented

1. Standard activation recorders silently keep only the last iteration's state
2. Unevenly spaced lens targets create apparent trends that reflect the instrument
3. Readout claims can be estimator-specific (top-8 readout vs. concept inventory)

## Empirical Validation / Results

### Transport Structure

**Bulk collapse with distance is universal**: Transport weakens with distance in all three architectures. Near its target, per-prompt $\|J\|/\sqrt{d} \approx 1.3–1.5$ on Ouro; across tens of layers, $\|J\|/\sqrt{d} = 0.13–0.3$ beyond one Ouro loop. At distance 39—the standard model's working band-to-target span—Qwen's aggregate transport is just as dead as Ouro's at matched distance (top-1 agreement: 2%).

**The semantic component does not cross Ouro's loop boundary**: Tracking a prompt-absent latent concept (Italy) through transports:

| Transport | Distance | State cosine | Carries Italy? |
|---|---|---|---|
| Qwen band → target | 39 | 0.19 | yes |
| Ouro band → own loop end | 11–23 | 0.3–0.5 | yes |
| Ouro loop-2 → loop-3 band | 32–60 | 0.31–0.45 | **no** |

The workspace component is destroyed roughly 3× faster than the state as a whole. The mechanism: deep supervision trains every loop-end state to be directly decodable (checkpoints), so each loop decodes conclusions into token-aligned states, and the next loop re-encodes them nonlinearly.

**Weight tying induces symmetries**: In the absolute frame, no trace of tying (cosine similarity decays smoothly with virtual distance, no block structure at multiples of 48). In the loop-relative frame, transports repeat: Ouro matched-offset matrices have mean cosine 0.885 (0.932 inside the workspace band), while Huginn shows essentially exact symmetry at cosine **0.9991** (r=8 vs. r=12 targets; 0.43 when offset by one recurrence).

**Factorisation orders the architectures**:

| | Cut | cos(direct, composed) |
|---|---|---|
| Ouro | supervised loop ends | 0.94–1.00 |
| Qwen | band entry, L24 | 0.69–0.95 |
| Huginn | recurrence ends | 0.25–0.84 |

Ouro's near-perfect factorisation shows deep supervision makes loops approximately Markovian; Huginn's poor factorisation reflects preserved prompt-dependent correlations across boundaries.

### What the Iterations Compute

**Ouro**: Iterative refinement, mostly done by loop 3. Loop-3's output matches the final output for 92% of tokens; the model's own early-exit gate assigns exit probability [0.03, 0.14, 0.34, 0.49] across four loops. Task difficulty maps onto convergence: easy facts settle in loop 1, riddles in loop 2, arithmetic in loop 4.

**Huginn**: A sliding window with persistent content. Each lens reads reliably only in the 1–2 recurrences before its own target, but because every window reads the same content, the latent answer computed by recurrence 2–3 persists through all 16 recurrences. Direct unembedding (logit lens) is uniformly weak but informative at every depth (15–25% top-1 hit rate, flat across the stack)—evidence of **implicit deep supervision** from sampled recurrence counts during training.

### Workspace Geometry

Occupancy analysis with the original study's concept-inventory estimator (gradient pursuit reconstruction) recovers the band, loop, and window structure:

- **Qwen**: Latent vocabulary absent below L20, fills the L24–48 band, persists into the top of the stack
- **Ouro**: Four workspace bands, one per loop, with the band containing 17 of each loop's 47 lensed layers
- **Huginn**: Flat plateau across the recurrent stack (31–44 slots per window) with a single jump in the coda (92 slots, 2.4× the plateau mean)

### Causal Suite Results

| Family | Study | Qwen | Ouro | Huginn |
|---|---|---|---|---|
| Verbal report (swap → top-5) | 59–88% | 69% | 80% | **98%** |
| Introspection (injected concept) | reportable | null | **top-1 31%** | null (0/97) |
| Modulation (focus/ctrl/suppress) | ordered | .38/.07/.07 | **.77/.39/.26** | .28/.15/.19 |
| Flexible generalisation (swap) | 40% | **42%** | 38% | 13% |
| Selectivity: language | expl. ≫ auto | +1.00 | +0.25 | +0.38 |
| Ignition (10–90 width) | sharp | 0.10 | **0.55** | none in-window |
| Capacity (words held) | ~25 | 0.7 | **2.4** | 1.8 |
| Dual task (concepts survive) | survive | 6/21 | **17/21** | 7–8/21 |

**Key causal findings**:

1. **Writes must span every remaining iteration on Ouro**: Clamping all remaining loops (2–4) reaches 38% (matching Qwen's 42%), but any single loop alone falls short (21% loop 2, 29% loop 3, 25% loop 4). Per-trial logit-lens trajectories show two distinct failure mechanisms: re-derivation (loop-2-only) vs. displacement against consolidated content (loop-4-only).

2. **Huginn's window equivalence**: Clamping the last window alone gives results identical to clamping all three windows, while the second-to-last window alone gives 3%—the ~2-recurrence transport horizon bounds writes exactly as it bounds reads.

3. **Ablation follows the same interfaces**: Ouro's loop-2-only single-direction ablation is a complete null (100% kept), but a ten-direction subspace ablation at loop 2 alone leaves only 14% of answers correct—re-derivation restores a deleted direction but not the state's wider J-space component.

4. **Verbalising new content requires explicit deep supervision**: Huginn verbalises self-computed content at 98% but never verbalises externally injected content (0/97). Only Ouro, with explicit per-iteration supervision, reports injected concepts (top-1 31%).

5. **The instrument blind spot**: Top-down summoning is ~0 on all three models. The lens is structurally blind to content not driving the current next token—$J_v = \mathbb{E}[\partial h_{\text{target}}/\partial h_v]$ maps idle content to (near) zero regardless of presence.

### Robustness

Re-running the complete pipeline on Ouro-2.6B-Thinking (reasoning SFT) shows every architecture-level signature reproduces within a few points: writes must still span every remaining loop, loop-2-only ablation is again a near-null, introspection is point-identical at 31%.

## Theoretical and Practical Implications

### For Global Workspace Theory

- Depth-as-recurrence **does** support workspace function: both looped models exhibit the full workspace signature, answering the open question from Gurnee et al. (2026)
- Recurrence changes the workspace's **interfaces**: every route by which content enters, survives, or leaves the workspace follows iteration structure rather than absolute depth
- The report experiments separate two abilities the standard transformer did not distinguish: steering existing content (needs only head pressure at depth) vs. verbalising new content (needs explicit per-iteration supervision)

### For Interpretability Practice

- **Cost reduction**: Weight tying means full-depth lens coverage of a recurrent model costs one iteration's fit, not r of them (Huginn lenses at matched offsets agree at cosine 0.9991)
- **Validation requirements**: Readouts beyond a lens's validated horizon do not read the state; unevenly spaced targets create instrument artefacts; readout claims can be estimator-specific
- **Three silent failure modes** in fitting lenses through recurrences, with a documented validation suite

### For Architecture Design

- The checkpoint interface (deep supervision) creates a destroy-and-re-derive regime with Markovian loop structure
- The injection path on Huginn is the only long-range route mean transport preserves—content persistence is continuous recomputation from a re-injected input, not a memory buffer
- Content does not survive even one recurrence beyond its last refresh (attenuation at α ≤ 0.25 destroys output; injection swap flips output in all 30 configurations)

## Conclusion

The paper's central answer: **yes, looped transformers do have a global workspace, but recurrence reshapes its interfaces**. Both looped models exhibit verbalisable, causally steerable, selectively engaged content. The two architectures differ fundamentally in maintenance strategy:

- **Ouro**: Supervised checkpoints decode/re-encode content at each boundary, making loops nearly Markovian and forcing interventions to span every remaining loop
- **Huginn**: Raw latent handoff preserves content across all recurrences, but reads and writes work only within a ~2-recurrence horizon

The two training regimes place maintenance and verbalisation on opposite sides of the boundary: steering existing content needs only head pressure at depth (Huginn: 98% verbal report), while verbalising new content tracks explicit per-iteration supervision exclusively (Ouro: 31% top-1; Huginn: 0/97).

**Limitations**:
1. Mean-Jacobian lenses are linearised summaries with a single-token concept vocabulary—structurally blind to content not driving the current next token
2. One pre-trained checkpoint per family confounds architecture with training recipe and scale
3. The two released checkpoints differ simultaneously in supervision, injection, and training recipe, so controlled training runs varying only the boundary interface are needed to separate causal contributions

**Future directions**: Following checkpoints throughout pretraining to test stability of conclusions and reveal when J-space emerges; systematic sweeps over test-time recurrence count; controlled training runs testing whether continued computation contributes to reasoning strength through built-in self-correction or is overhead a better iteration interface could remove.

---

_Markdown view of https://picx.dev/p/MesNk1, served by PicX — AI-generated visual whiteboard summaries of research papers._