Full text not available for this paper

Summary (Overview)

  • Key Finding: Completing the full preset latent horizon in latent Chain-of-Thought (CoT) reasoning can actually degrade reasoning performance, as better reasoning outcomes often emerge before the preset horizon is exhausted but fail to be preserved.
  • Three-Stage Evolution: Latent CoT evolves through three distinct functional stages: frontier advancement (new reasoning candidates form), candidate organization (existing candidates are reorganized), and answer revision (later steps mainly revise existing candidates, potentially overturning gold answers).
  • Method: The authors introduce a lightweight estimator that predicts remaining reasoning progress from the observed latent trajectory and adaptively adjusts the latent horizon during inference—without retraining the LLM.
  • Results: Adaptive latent inference improves reasoning accuracy by up to 7.2 percentage points while reducing latent computation by up to 29.0% across CODI, Coconut, and SIM-CoT latent reasoners on GSM8K, SVAMP, and MultiArith benchmarks.
  • Theoretical Foundation: The paper provides formal results (Proposition 4.1, Theorem 4.2) characterizing optimal estimation of remaining progress and monotone allocation of additional computation based on remaining progress.

Introduction and Theoretical Foundation

Background and Motivation

Latent reasoning (Deng et al., 2023; Hao et al., 2024) has emerged as an alternative to explicit Chain-of-Thought (CoT) reasoning (Wei et al., 2022). Unlike explicit CoT, which externalizes intermediate reasoning as discrete language tokens, latent CoT carries intermediate reasoning directly in continuous representations, enabling:

  • Maintenance of multiple reasoning alternatives (Zhu et al., 2026b)
  • Compression of explicit reasoning into continuous states (Cheng & Van Durme, 2024; Shen et al., 2025)

Existing latent reasoners typically execute a fixed number of latent steps (preset latent horizon KK) before generating an explicit answer. The intuitive assumption is that a longer latent horizon should provide more opportunity for reasoning (Goyal et al., 2024).

Key Observation

The paper challenges this intuition by showing that better reasoning outcomes can emerge before the preset horizon is exhausted and fail to be preserved at the full horizon (Figure 1). This suggests that reasoning benefits do not simply accumulate as latent CoT becomes deeper.

Problem Formulation

Latent CoT: Let t∈{1,…,K}t \in \{1, \ldots, K\} index latent computation steps, where KK is the preset latent horizon. The reasoning state after the tt-th latent step is ztz_t. Given input xx and preceding states z<tz_{<t}, the model induces:

zt∼Tθ(⋅∣x,z<t)z_t \sim T_\theta(\cdot | x, z_{<t})

After tt latent steps, the answer distribution is pθ(y∣x,z1:t)p_\theta(y | x, z_{1:t}), with task reward r(x,y)r(x, y) scoring generated answers.

Adaptive Latent Horizon: Let τ∈{1,…,K}\tau \in \{1, \ldots, K\} denote the realized latent horizon, with the answer generated as y∼pθ(⋅∣x,z1:τ)y \sim p_\theta(\cdot | x, z_{1:\tau}). The goal is to determine τ\tau from latent computation available during inference, where continuation at step tt depends only on xx and prefix states z1:tz_{1:t}.


Methodology

1. Identifying Three Functional Stages

Using the ProsQA benchmark (Hao et al., 2024) with explicit reasoning graphs, the authors analyze the symbolic Coconut model (Zhu et al., 2026b) by decoding reasoning candidates at each latent step and tracking probability changes:

StageDescriptionCharacteristics
Frontier AdvancementEarlier evolution forms new task-relevant reasoning statesCandidates at smaller graph depths receive support first; support moves through intermediate reasoning before answers emerge
Candidate OrganizationLater computation reorganizes already-emerged candidatesFewer new candidates enter high-support region; existing candidates continue receiving support
Answer RevisionFinal stage mainly revises existing answer candidatesSupport shifts among existing candidates; can weaken gold candidate support and overturn established answers

2. Empirical Tests of Reasoning Gains

Test 1 (Frontier Stage): Interrupting ongoing latent CoT at the frontier stage by inserting a single explicit realization (decoding current state to its top token and adding it to the causal context) markedly delays subsequent progression toward the gold answer. A matched control with comparable perturbations preserves candidate progression.

Test 2 (Revision Stage): Later computation recovers the gold answer in some cases but also overturns already-established gold results, leaving little net gain. Some later revisions allow gains reached earlier to be lost.

3. Remaining Reasoning Progress Estimation

Current reasoning value after tt latent steps:

Gt=Ey∼pθ(⋅∣x,z1:t)[r(x,y)](1)G_t = \mathbb{E}_{y \sim p_\theta(\cdot|x,z_{1:t})}\left[r(x, y)\right] \tag{1}

Remaining reasoning progress (comparing current frontier to best attainable within full horizon):

Ωt=max⁡1≤j≤KGj−max⁡1≤j≤tGj(2)\Omega_t = \max_{1 \leq j \leq K} G_j - \max_{1 \leq j \leq t} G_j \tag{2}

Training the estimator: A lightweight estimator fϕf_\phi is trained on a causal trajectory representation XtX_t (capturing short-range changes in predictive statistics and hidden states) to predict remaining progress:

Lprog(ϕ)=E(Xt,Ω^t)[(fϕ(Xt)−Ω^t)2](3)\mathcal{L}_{prog}(\phi) = \mathbb{E}_{(X_t, \hat{\Omega}_t)}\left[\left(f_\phi(X_t) - \hat{\Omega}_t\right)^2\right] \tag{3}

Horizon adaptation: Each estimated remaining progress is expressed as a relative position in a frozen reference distribution:

ut=Fref[fϕ(Xt)](4)u_t = F_{ref}[f_\phi(X_t)] \tag{4}

Additional latent computation allocated after step tt:

at=(K−t)⋅g(ut)(5)a_t = (K - t) \cdot g(u_t) \tag{5}

where gg is a bounded, non-decreasing continuation rule.

4. Theoretical Results

Proposition 4.1 (Causal Estimation of Remaining Progress): Under population squared risk R(f)=E[(f(Xt)−Ωt)2]R(f) = \mathbb{E}[(f(X_t) - \Omega_t)^2], the optimal estimator satisfies:

f∗(Xt)=E[Ωt∣Xt](6)f^*(X_t) = \mathbb{E}[\Omega_t | X_t] \tag{6}

Theorem 4.2 (Monotone Allocation by Remaining Progress): Let V(a,ω)V(a, \omega) have increasing differences in (a,ω)(a, \omega). For trajectories with remaining progress ωi\omega_i, consider integer allocations ai∈{0,…,A}a_i \in \{0, \ldots, A\} maximizing ∑iV(ai,ωi)\sum_i V(a_i, \omega_i) subject to shared compute budget ∑iai≤B\sum_i a_i \leq B. Then there exists an optimal allocation a∗a^* such that ωi>ωj\omega_i > \omega_j implies ai∗≥aj∗a^*_i \geq a^*_j.


Empirical Validation / Results

5.1 Adaptive Latent Inference Results

Table 1: Adaptive latent CoT inference (accuracy %)

GSM8KSVAMPMultiArithLatent step
MethodFull-KOursGainFull-KOursGainFull-K
GPT-2
CODI42.143.4+1.341.043.1+2.194.4
SIM-CoT39.241.5+2.340.043.5+3.591.7
Coconut27.127.7+0.631.433.7+2.371.7
SIM-CoT Coconut44.744.7+0.040.642.2+1.692.8
Llama-1B
CODI55.555.7+0.261.162.3+1.297.2
SIM-CoT55.655.9+0.361.162.2+1.197.2

Key findings:

  • Largest gain: MultiArith with GPT-2/Coconut (71.7% → 78.9%)
  • Improvements persist even at high accuracy levels (e.g., Llama-1B/CODI on MultiArith: 97.2% → 98.3%)
  • Significant latent step savings (up to 29.0%)

5.2 Constructive Progress Across Latent CoT

  • Progress completion: Majority of constructive progress is completed by the third latent step across all six reasoners
  • Reasoner differences: CODI variants show earlier completion; Coconut variants retain more progress for later steps
  • Table 3 shows higher constructive progress consistently corresponds to higher reasoning accuracy:
    • GPT-2/SIM-CoT on SVAMP: 13.2% (Lower) vs. 73.4% (Higher)
    • Llama-1B/CODI on GSM8K: 24.9% (Lower) vs. 69.8% (Higher)

5.3 Remaining Progress and Computation Returns

Table 4 shows:

  • Higher remaining progress (Ω3\Omega_3) corresponds to larger accuracy gains from Step 3 to Step 4
  • Example: GPT-2/SIM-CoT on GSM8K: accuracy increases from 18.5% to 57.3% under higher Ω3\Omega_3, vs. remaining at 35.0% under lower Ω3\Omega_3
  • Future gain differences grow over subsequent latent steps (e.g., Llama-1B/CODI: only 1.6% difference at Step 4, expanding to 58.4% by Step 5)
  • Larger decreases in Ωt\Omega_t are associated with larger reductions in subsequent reasoning gain

5.4 Continuation Signals Comparison

Table 5 (matched latent computation): Remaining progress outperforms stability, confidence, and convergence signals in most settings.

Table 6 (transferred adaptive methods):

DatasetBackboneOursALRDEERDynamic K-Head
GSM8KGPT-2/CODI41.138.437.338.4
GSM8KGPT-2/SIM-CoT39.235.134.334.3
SVAMPGPT-2/SIM-CoT40.239.939.639.8
MultiArithGPT-2/SIM-CoT89.488.987.187.2
MultiArithLlama-1B/CODI97.296.696.996.6

Remaining progress yields higher accuracy than all transferred criteria by up to 4.9 percentage points, with the largest gap on GSM8K where continued latent computation still yields gains.


Theoretical and Practical Implications

Theoretical Implications

  1. Non-uniform reasoning benefits: The paper formally demonstrates that latent CoT's reasoning value is non-uniform along the trajectory, contradicting the implicit assumption that deeper latent reasoning always helps.

  2. Causal estimation framework: Proposition 4.1 establishes that remaining progress can be optimally predicted from the latent trajectory through conditional expectation, providing a principled foundation for online adaptation.

  3. Monotone allocation principle: Theorem 4.2 shows that under increasing differences in the value function, trajectories with more remaining progress should receive no less additional computation—a formal justification for progress-based budget allocation.

Practical Implications

  1. Inference efficiency: The method reduces latent computation by up to 29.0% while improving accuracy, offering immediate computational savings for deployed latent reasoners.

  2. No retraining required: The approach works on frozen LLMs, making it a lightweight, plug-and-play solution compatible with existing latent CoT reasoners.

  3. Generalizable framework: The constructive-progress framework applies across diverse latent reasoner architectures (CODI, Coconut, SIM-CoT) and backbones (GPT-2, Llama-1B), suggesting broad applicability.

  4. Beyond fixed-horizon thinking: The work challenges the field to reconsider fixed-horizon inference as a design choice rather than a default, opening new directions for adaptive inference strategies.


Conclusion

Main Takeaways

  1. Completing the full latent horizon can degrade reasoning performance—gold answers reached earlier can be overwritten by later revision stages.

  2. Latent CoT evolves through three distinct functional stages (frontier advancement, candidate organization, answer revision), with reasoning gains concentrated in the frontier stage.

  3. Adaptive latent inference based on estimated remaining progress consistently improves reasoning accuracy (up to +7.2 percentage points) while reducing latent computation (up to 29.0%) across CODI, Coconut, and SIM-CoT variants on GSM8K, SVAMP, and MultiArith.

  4. Higher realized constructive progress corresponds to higher current reasoning accuracy, while more remaining progress corresponds to larger future gains from continued latent CoT.

Future Directions

  • Extending adaptive latent inference to more complex reasoning tasks beyond arithmetic benchmarks
  • Investigating whether the three-stage evolution pattern generalizes to other latent reasoning architectures
  • Exploring training-time adaptations that could make latent CoT more robust to revision-stage degradation
  • Developing richer trajectory representations to improve remaining progress estimation
  • Scaling the approach to larger models and more diverse reasoning domains

The paper concludes that "by rethinking the fixed-horizon of latent CoT, we develop a method that adaptively adjusts the executed horizon during inference," consistently improving reasoning accuracy while reducing latent computation.

Related papers