# Best-of-Depth: Learning Exploratory Dynamics in Looped Transformers > Best-of-Depth training, which minimizes loss across all recurrent depths, maximizes solution coverage in looped transformers, outperforming standard objectives on combinatorial tasks and code generation. - **Source:** [arXiv](https://arxiv.org/abs/2611.04663) - **Published:** 2026-11-07 - **Permalink:** https://picx.dev/p/DaAqhX - **Whiteboard:** https://picx.dev/p/DaAqhX/image ## Summary # Best-of-Depth: Learning Exploratory Dynamics in Looped Transformers ## Summary (Overview) - **Novel training objective**: The paper introduces **Best-of-Depth (BoD)**, which trains looped Transformers by minimizing the *minimum* prediction loss across supervised depths, treating the recurrent trajectory as a candidate set of solutions rather than supervising only the final output. - **Theoretical grounding**: Under exact-match loss, BoD is proven to maximize solution coverage—the fraction of all valid solutions recovered along the trajectory—connecting the training objective directly to solution discovery. - **Strong empirical gains on combinatorial tasks**: On N-Queens completion, BoD achieves **91.03% coverage** vs. **45.75%** for final-depth training; on graph coloring, it reaches **72.16% coverage** vs. 44.39% for the best baseline (Soft-Depth), with success nearly saturating while coverage continues to grow. - **Discovery beyond training data**: BoD recovers solutions withheld from teacher pools on familiar inputs and finds new valid solutions beyond the training depth, demonstrating genuine exploratory dynamics. - **Language-model validation**: Fine-tuning the 3B-parameter Huginn looped LM with BoD yields higher pooled code-generation success (37.8% HumanEval, 44.2% MBPP) than final-depth training (34.8%, 39.2%). --- ## Introduction and Theoretical Foundation ### Background and Motivation Looped Transformers extend inference-time computation by repeatedly applying a shared transformation to a latent state, increasing computational depth without adding parameters. While greater recurrent depth can improve reasoning, the *learning objective* determines how successive updates contribute to problem-solving. The key insight: **a single recurrent trajectory produces multiple predictions at successive depths**, which can be treated as a candidate set—analogous to Best-of-N sampling but generated *sequentially* along one trajectory rather than in parallel. ### Problem with Existing Objectives - **Last-Depth training** (Eq. 2): Only supervises the final prediction, so intermediate states receive no direct learning signal for producing useful alternatives. - **Mean/Random-Depth**: Supervising every depth against the *same* target forces all readouts to reproduce that target, discouraging diversity. - **Fixed-point convergence** approaches promote stability but do not determine *which* alternatives a trajectory discovers. ### Theoretical Foundation The paper formalizes the connection between BoD and solution coverage. For exact-match loss, define: $$\mathcal{L}_{\mathrm{BoD}}^{0-1} = \min_{d \in \mathcal{D}} \mathbf{1}[\hat{y}_d \neq y]$$ **Proposition 1 (First-hit interpretation)**: Let $\tau_y = \min\{d \in \mathcal{D} : \hat{y}_d = y\}$, with $\tau_y = \infty$ if no readout matches $y$. Then: $$\mathbb{E}[\mathcal{L}_{\mathrm{BoD}}^{0-1}] = \Pr(\tau_y > D)$$ BoD evaluates whether the target is reached *anywhere* within the depth budget. **Proposition 2 (Connection to solution coverage)**: Fix $x$ and $S_D(x) = \{\hat{y}_d(x) : d \in \mathcal{D}\}$ independently of target $Y \sim \text{Uniform}(\mathcal{Y}^\star(x))$. Then: $$\mathbb{E}_Y[\mathcal{L}_{\mathrm{BoD}}^{0-1} \mid x] = 1 - \frac{|S_D(x) \cap \mathcal{Y}^\star(x)|}{|\mathcal{Y}^\star(x)|} = 1 - \text{Coverage@Depth}(D; x)$$ Minimizing BoD risk **maximizes solution coverage**; repeated candidates do not increase coverage. **Proposition 3 (Separation in solution discovery)**: There exists a multi-solution task where, under cross-entropy training with uniformly resampled valid targets, greedy coverage satisfies: $$\lim_{n \to \infty} \text{Coverage@Depth}(D; \theta_A^{(n)}) = \begin{cases} 0, & A \in \{\text{Last}, \text{Mean}, \text{Random}, \text{Soft}\}, \\ 1, & A = \text{BoD}, \end{cases}$$ almost surely. BoD breaks symmetry by selecting one loss-minimizing depth, allowing different targets to reinforce the same latent solution through different recurrent phases. --- ## Methodology ### Looped Transformer Formulation A looped Transformer encodes input $x$ and repeatedly applies a shared transformation: $$h_0 = \mathrm{Enc}_\theta(x), \qquad h_d = F_\theta(h_{d-1}; x), \qquad p_d = \mathrm{Dec}_\theta(h_d), \quad d = 1, \ldots, D$$ One application of $F_\theta$ is a loop; $d$ denotes recurrent depth. A shared decoder reads prediction distribution $p_d$ from each state. ### The Best-of-Depth Objective At training depth $D = D_{\text{train}}$, with supervised depths $\mathcal{D} \subseteq \{1, \ldots, D\}$ (including $D$), BoD minimizes: $$\mathcal{L}_{\mathrm{BoD}}(x, y) = \min_{d \in \mathcal{D}} L_d(x, y)$$ For autoregressive targets, the teacher-forced loss is: $$L_d(x, y) = -\frac{1}{T} \sum_{t \in \mathcal{T}} \log p_d(y_t \mid x, y_{