# Extending Reasoning in Typed Decision Models via Adaptive Looping (RELOAD)

> RELOAD retrofits damped loops into pretrained decision models, boosting accuracy up to 23 points on long reasoning chains with zero added parameters via joint training.

- **Source:** [arXiv](https://arxiv.org/abs/2611.07736)
- **Published:** 2026-11-07
- **Permalink:** https://picx.dev/p/UsolQ6
- **Whiteboard:** https://picx.dev/p/UsolQ6/image

## Summary

# Summary of "Extending Reasoning in Typed Decision Models via Adaptive Looping"

## Summary (Overview)

- **Core contribution**: The paper introduces RELOAD (retrofitted loops for adaptive decisions), a method that retrofits a damped loop over a middle block of layers in pretrained typed decision models, enabling them to spend variable test-time compute on hard inputs without generating tokens.
- **Key finding**: Looping helps when trained *together* with the decision model, contrary to prior work that found no gain from loops retrofitted into conventionally pretrained models. Across six models (0.4B–4B parameters), RELOAD improves both accuracy and calibration, with bidirectional encoders gaining up to 23 points on problems requiring longer reasoning chains than seen in training.
- **Efficiency**: A lightweight controller picks the number of passes per input, recovering most of the gain of four passes at about three quarters of the compute.
- **Theoretical insight**: The loop is shown to be an Euler discretization of an ODE, enabling test-time integrator substitution—a midpoint integrator with four passes matches eight Euler passes for Von.
- **Parameter efficiency**: Depth up-scaling (adding separate layers per pass) requires nearly twice the parameters to match RELOAD's performance.

## Introduction and Theoretical Foundation

Typed decision models (e.g., Von, Kev) read a context, question, and candidate answers, outputting a distribution over candidates in a single forward pass. Unlike generative language models that can spend more test-time compute by generating intermediate tokens (chain-of-thought), decision models process every input at the same depth, limiting their ability to handle multi-step reasoning (e.g., composing spatial relations, following proofs).

Prior work on looped decision models (SanSi, LEV) built on language models *pretrained to loop* and reported no gain from retrofitting loops into conventionally pretrained models. RELOAD challenges this by showing that loops help when trained jointly with the decision model.

The theoretical foundation rests on viewing the loop as an ODE solver. The damped update:

$$
h_{k+1} = h_k + \frac{1}{K}(G(h_k) - h_k)
$$

is the explicit Euler method with step size $1/K$ for the ODE:

$$
\frac{dh}{dt} = G(h(t)) - h(t), \quad h(0) = h_0, \quad t \in [0, 1]
$$

This means every budget $K$ discretizes the *same trajectory* and approaches the end state $h(1)$ as $K$ grows—the budget sets the step size, not trajectory length. This yields two predictions: (1) error relative to $h(1)$ falls as $1/K$ for smooth $G$; (2) the $K$ evaluations of $G$ can be spent on a more accurate integrator at test time.

## Methodology

**Model architecture**: The network is split into prefix $P$, looped block $G$ (approximately the middle third of layers), and suffix $S$. With budget $K$:

$$
\begin{array}{c} h_0 = P(e(x)), \\ h_{k+1} = h_k + \frac{1}{K}(G(h_k) - h_k), \\ p(\cdot \mid x) = \operatorname{softmax}(f(S(h_K))) \end{array}
$$

At $K=1$, the update recovers the original network. All passes share weights, so looping adds no parameters.

**Training objective**: One model is trained on all budgets $K \in \{1, 2, 3, 4\}$ simultaneously:

$$
\mathcal{L}(x, y) = \frac{1}{4} \sum_{K=1}^{4} \ell(p_K(\cdot \mid x), y)
$$

where $\ell$ is cross-entropy plus $0.5 \times$ the Brier score. This supervises endpoints of four trajectories with different step sizes.

**Controller**: A request-level controller picks the budget once per request from prefix output features:

$$
z = \mathrm{PCA}_{64}\left(\mathrm{std}\left[\bar{h}_{\mathcal{C}}, \bar{h}_{\mathcal{O}}, \log n, m\right]\right)
$$

One logistic regression per budget estimates correctness probability $\hat{P}_K = \sigma(w_K^\top z + b_K)$, and the controller selects:

$$
K^{\star} = \arg\max_{K} \hat{P}_K - \lambda(K - 1)
$$

**Models tested**: Three bidirectional encoders (Von-395M, Ettin-400M, Ettin-1B) and three causal decoders (Qwen3-0.6B, 1.7B, 4B with Kev's encoding).

## Empirical Validation / Results

### Main Results (Table 1)

| Model | Variant | In-dist. Acc. | Longer Acc. | OOD Acc. | Layers |
|-------|---------|---------------|-------------|----------|--------|
| Von (395M) | Single pass | 92.8 | 70.6 | 50.9 | 28 |
| | RELOAD (K=4) | **93.9** | **85.7** | **54.1** | 55 |
| Ettin-400M | Single pass | 93.0 | 72.3 | 47.9 | 28 |
| | RELOAD (K=4) | **94.0** | **87.0** | **50.5** | 55 |
| Ettin-1B | Single pass | 93.9 | 80.2 | 48.7 | 28 |
| | RELOAD (K=4) | **94.6** | **90.7** | **53.2** | 55 |
| Qwen3-0.6B | Single pass | 86.1 | 48.1 | 52.6 | 28 |
| | RELOAD (K=4) | **88.0** | **55.2** | **53.4** | 58 |
| Qwen3-1.7B | Single pass | 89.3 | 57.7 | 54.6 | 28 |
| | RELOAD (K=4) | **90.9** | **60.2** | **56.5** | 58 |
| Qwen3-4B | Single pass | 90.9 | 62.0 | 60.4 | 36 |
| | RELOAD (K=4) | **93.0** | **72.1** | **62.0** | 72 |

### Control Experiments (Table 2, Von)

| Variant | Longer Acc. | Brier | Params |
|---------|-------------|-------|--------|
| Single pass | 70.6 | 0.404 | 395M |
| Single pass, 12 epochs (matched compute) | 74.9 | 0.455 | 395M |
| Single pass, looped only at inference | 70.7 | 0.399 | 395M |
| Depth up-scaling | 84.3 | 0.227 | 726M |
| RELOAD (K=4) | **85.7** | **0.198** | 395M |

**Key findings**:
- **Reasoning horizon**: Four passes extend the encoders' reasoning horizon by 2–3.5 statements (Von: 6.4 → 9.6); two or three passes give most of this gain.
- **Distractor robustness**: Five distractor statements cost single-pass encoders 16–19 points but RELOAD only 3–4 points.
- **Object tracking**: Four passes raise Von's accuracy from 77% to 98% with six swaps and from 44% to 75% with eight.
- **Integrator substitution**: The midpoint method with four passes matches eight Euler passes for Von on longer reasoning (+1.7 accuracy, −0.022 Brier).

### Design Choice Ablations (Table 3)

| Variant | Gain | Δ Longer |
|---------|------|----------|
| Block loop (RELOAD) | +8.7 | — |
| Split into 4+5 layers | +7.9 | −0.8 |
| Split into 3+3+3 | +5.9 | −4.3* |
| Each layer in place (9 layers) | +1.5 | −7.8* |
| Each layer in place (28 layers) | +1.5 | −7.4* |

## Theoretical and Practical Implications

- **Theoretical**: The ODE view of looping provides a principled framework—budgets are step-size choices along a single trajectory, and better integrators can substitute for more passes. The error follows $\propto 1/K$ with fitted exponent 1.01, confirming Euler convergence. This connects to residual networks as dynamical systems (Haber & Ruthotto, 2018; Chen et al., 2018).

- **Practical**: Looping offers test-time compute for decision models without parameter overhead—crucial for classification, routing, and verification systems where generative approaches are too expensive. The controller achieves most gains at ~75% compute, making adaptive computation practical. RELOAD on Ettin-400M beats single-pass Ettin-1B, showing loops can substitute for model scale.

- **Architectural insight**: The gain requires looping a *block* of layers (enabling feedback from later to earlier layers), not each layer in place. The passes act on statements that must be composed, with Shapley analysis attributing ~2/3 of the gain to chain statements, most of the rest to options, and almost nothing to distractors.

## Conclusion

RELOAD demonstrates that loops retrofitted into conventionally pretrained decision models provide meaningful test-time compute *when trained jointly with the model*. One checkpoint trained on all budgets runs at any number of passes, improving accuracy and calibration without adding parameters—most dramatically on longer reasoning chains for bidirectional encoders (up to 23 points). Matched training compute, more data, and parameter-heavy depth up-scaling cannot close this gap efficiently.

**Future directions**: (1) Objectives that make longer trajectories useful (e.g., longer integration time for harder inputs); (2) features exposing problem structure for more precise controller allocation; (3) testing on longer natural-language reasoning tasks beyond templated stories; (4) understanding why causal decoders benefit less than bidirectional encoders; (5) exploring more sophisticated integrators and training schemes to raise the ceiling on multi-step reasoning gains.

---

_Markdown view of https://picx.dev/p/UsolQ6, served by PicX — AI-generated visual whiteboard summaries of research papers._