Summary (Overview)

• Core finding: Deeper recurrent computation in LLMs can lose correct solutions found at shallower depths, and depth-specific success sets are broadly non-nested, meaning a single stopping depth is fundamentally incomplete.
• New formulation: The paper proposes Depth Portfolio Allocation (DPA) , which reframes recurrent test-time scaling as a budgeted allocation problem across depth-specific candidate sources rather than a single stopping-depth decision.
• Methodology: DPA supports both fixed portfolios and feedback-guided sequential allocation (trained with PPO) under a shared inference budget, treating each depth as both a computation cost and a candidate source.
• Key results: Mixed-depth portfolios consistently expand oracle coverage at matched candidate counts and nominal costs; feedback-guided allocation generally matches or improves uniform allocation and can reduce computation in high-budget settings.
• Evaluation scope: Validated on GSM8K, SVAMP, MultiArith, and MBPP using a frozen Huginn backbone, with gains of up to +9.25 pp coverage over fixed-depth baselines.

---

Introduction and Theoretical Foundation

Background

Test-time scaling (TTS) improves LLM performance by allocating additional inference-time computation without changing model parameters. Existing approaches typically spend compute in token space (longer reasoning traces, multiple candidate samples, verifier-based selection). Recurrent latent reasoning offers an alternative: the model repeatedly applies recurrent computation to hidden representations, allowing answers to be decoded at different recurrent depths.

The Single-Depth Assumption Problem

The common view treats recurrent TTS as deciding when to stop—a single-trajectory formulation that assumes deeper recurrence progressively refines (or at least preserves) solutions from shallower depths. However, recent work shows performance can peak and then degrade with continued recurrence, suggesting deeper computation need not provide monotonic refinement.

Research Question

Does a single stopping depth miss correct solutions available at other depths, and how can a limited budget exploit these complementary sources?

Theoretical Insight

The paper identifies that depth determines not only computational cost but also access to distinct candidate solutions. This reframes inference from a stopping problem to an allocation problem: how to distribute a finite budget across depth-specific candidate sources to maximize final answer accuracy.

---

Methodology

3.1 Preliminary Diagnosis

The authors first establish that deeper recurrence does not preserve correctness:

• Latent updates gradually diminish with depth (normalized update magnitudes decrease, adjacent-state cosine similarities approach 1), yet stable representations do not ensure correct answers persist.
• A concrete example shows the correct answer "4" appears at depths $d=8, 16$ but disappears at $d=24, 32$.

3.2 Success Set Analysis

For each question $q$, depth $d$, and repeat $r$, let $y_{qdr} \in \{0,1\}$ indicate correctness. The success set at depth $d$ is:

$$C_{d} = \{q: \max_{r \in \{1, \dots, 5\}} y_{qdr} = 1\}. \tag{1}$$

Two success sets are mutually non-nested if each contains questions absent from the other. The Jaccard distance measures their difference:

$$J(d_i, d_j) = 1 - \frac{|C_{d_i} \cap C_{d_j}|}{|C_{d_i} \cup C_{d_j}|}.$$

To control for marginal accuracy, they regress pairwise Jaccard distance on depth separation and accuracy differences:

$$\begin{array}{c} J(d_{i}, d_{j}) = \beta_{0} + \beta_{1} |d_{i} - d_{j}| \\ \qquad + \beta_{2} |a_{i} - a_{j}| + \beta_{3} \frac{a_{i} + a_{j}}{2} + \epsilon_{ij}, \end{array} \tag{2}$$

where $a_i$ and $a_j$ are marginal accuracies at the two depths. Bootstrap and fixed-margin randomization tests confirm the complementarity is not explained solely by question difficulty or marginal accuracy.

4.1 Budgeted Allocation Formulation

Given question $q$ and candidate depths $\mathcal{D}$, let $x_{qdr}$ be candidate $r$ at depth $d$, and $\mathbf{n} = (n_d)_{d \in \mathcal{D}}$ denote acquisition counts. With nominal per-candidate cost $c_d = d$:

$$C(\mathbf{n}) = \sum_{d \in \mathcal{D}} d n_d \leq B, \tag{3}$$

where $B$ is the budget cap. The objective is:

$$\begin{array}{c} \alpha^{*} = \arg \max_{\alpha \in \mathcal{F}_{\mathrm{DPA}}} \mathbb{E}[y(q, g(q, R_{\alpha}))] \\ \text{s.t.} C(\mathbf{n}_{\alpha}) \leq B, \end{array} \tag{4}$$

where $R_\alpha$ is the candidate set from allocation rule $\alpha$ and $g(q, R_\alpha)$ selects the final answer.

4.2 Feedback-Guided Sequential Allocation

The policy state is:

$$s_{t} = (\phi(q), \psi(R_{t}), \mathbf{n}_{t}, B - C_{t}), \tag{5}$$

where $\phi(q)$ represents the question and $\psi(R_t)$ summarizes acquired candidates and selector feedback. At each step, the policy chooses:

$$a_{t} \in \mathcal{D} \cup \{\text{stop}\}. \tag{6}$$

If $a_t = d$, one new candidate is acquired:

$$\begin{array}{c} R_{t+1} = R_{t} \cup \{x_{q, d, n_{t,d} + 1}\}, \\ n_{t+1,d} = n_{t,d} + 1, \qquad C_{t+1} = C_{t} + d. \end{array} \tag{7}$$

PPO training: The reward is the change in selector correctness:

$$r_{t} = u_{t+1} - u_{t}, \tag{8}$$

so the undiscounted return telescopes to final selection correctness. The PPO objective uses the probability ratio:

$$\rho_{t}(\theta) = \frac{\pi_{\theta}(a_{t} \mid s_{t})}{\pi_{\theta_{\mathrm{old}}}(a_{t} \mid s_{t})}, \tag{9}$$

with the clipped surrogate objective:

$$\begin{array}{c} L_{\mathrm{PPO}}(\theta) = \mathbb{E}_{t}\big[\min(\rho_{t}\hat{A}_{t}, \\ \operatorname{clip}(\rho_{t}, 1 - \epsilon, 1 + \epsilon)\hat{A}_{t})\big]. \end{array} \tag{10}$$

Implementation details:

• Depth set $\mathcal{D} = \{16, 24, 32\}$, at most 5 candidates per depth, budget caps $B \in \{48, 80, 120, 160, 200, 240, 320\}$
• A five-member Borda selector (based on latent reward models) reranks candidates after each acquisition
• Feasibility masks prevent depth actions that exceed budget or per-depth limits
• Ground-truth labels used only for training rewards; deployment uses candidate feedback only

---

Empirical Validation / Results

RQ1: Does Depth Diversity Expand the Reachable Solution Set?

Table 2A (Fixed candidate count, 4 candidates per configuration):

Table 2B (Fixed nominal cost) highlights:

• At budget 144: depth portfolio achieves 49.55% on GSM8K vs. 40.30% for fixed intermediate depth (+9.25 pp)
• At budget 80: portfolio gains +7.81 pp on SVAMP and +5.58 pp on MBPP
• Gains narrow when shallow sampling approaches coverage saturation (e.g., MultiArith at high budgets)

RQ2: Does Adaptive Allocation Improve Accuracy under the Same Budget?

Table 3 (Representative DPA/PPO results):

Feedback-guided allocation generally improves over Uniform under the same cap, with the largest gains on MultiArith. Single-trajectory baselines (Base/single: 32.60% GSM8K, 51.70% SVAMP, 27.80% MBPP, 75.00% MultiArith) are substantially outperformed, though these represent different inference configurations.

RQ3: When Does Depth Allocation Save Computation?

At $B = 320$, PPO uses less nominal computation than Uniform on GSM8K, MBPP, and MultiArith (e.g., MBPP uses only 238.7 steps vs. 320 cap, i.e., 74.6%). The policy often terminates before exhausting the budget, indicating further acquisition may not help. This is most pronounced on MBPP, where PPO slightly improves accuracy while leaving ~25% of the budget unused.

---

Theoretical and Practical Implications

Theoretical Implications

1. Challenges the monotonic refinement assumption: The paper provides systematic evidence that deeper recurrence does not subsume shallower solutions, undermining the single-trajectory view of recurrent TTS.
2. New conceptual framing: Depth is repositioned as a candidate source (providing access to distinct solutions) rather than merely a stopping position (controlling computation amount).
3. Source complementarity is structural: Fixed-margin randomization tests confirm that depth-dependent success-set differences exceed what question difficulty and marginal accuracy alone would predict, suggesting genuine source-specific information.

Practical Implications

1. Budget allocation as a design principle: Systems should allocate inference budgets across multiple depths rather than committing to a single depth, especially when verifier quality is imperfect.
2. Adaptive computation savings: Feedback-guided allocation can reduce unnecessary computation by stopping early when further acquisition is unlikely to help—valuable for cost-sensitive deployment.
3. Selector-aware allocation: The interaction between candidate source diversity and selector feedback matters; diverse depths provide better material for reranking.

---

Conclusion

Recurrent test-time scaling is not only a question of how deep to run. The paper establishes that:

1. Deeper recurrence can lose correct shallower-depth solutions, and depth-specific success sets are broadly non-nested.
2. Depth Portfolio Allocation (DPA) reframes recurrent TTS as budgeted allocation across complementary depths.
3. Mixed-depth portfolios expand candidate coverage, and feedback-guided allocation (PPO) generally matches or improves uniform allocation while enabling computation savings.

Future directions:

• Extending to additional recurrent architectures, model scales, and task domains
• Richer candidate-generation settings and end-to-end runtime costs (including decoding and selector overhead)
• Broader forms of adaptive recurrent inference beyond the formulations studied

---

Key Mathematical Notation Reference