# Dory1.0-4B: A Looped Normalized-GPT

> Dory1.0-4B, a looped 4B model, outperforms models up to 9x larger by scaling compute per token in latent space rather than generating longer outputs.

- **Source:** [arXiv](https://arxiv.org/abs/2611.03998)
- **Published:** 2026-11-07
- **Permalink:** https://picx.dev/p/OWO94h
- **Whiteboard:** https://picx.dev/p/OWO94h/image

## Summary

## Summary (Overview)

- **Dory1.0-4B** is a 4-billion-parameter looped (latent-recurrent) language model with elastic depth, built on a normalized Transformer (nGPT) backbone. A single checkpoint serves different inference compute budgets by varying the loop count $k$ of its shared recurrent block.
- The model was pretrained on 6 trillion tokens (first 4.5T in NVFP4), extended to a 256K-token context, and post-trained with SFT followed by RLVR using a **difficulty-aware loop curriculum** that routes harder prompts to deeper loops.
- Increasing loop count improves accuracy, with the largest gains on coding, agentic, and long-context benchmarks. With 2+ loops, Dory1.0-4B outperforms larger models (Qwen3.5-9B, Gemma 4 12B, Gemma 4 26B-A4B) on average across 12 benchmarks.
- Looping improves token efficiency: with four loops, the model generates 1.3–2.5× fewer tokens than with one loop on agentic benchmarks, as extra computation is spent in latent space rather than on longer outputs.
- **Last-loop KV caching** keeps the KV-cache footprint equal to a non-looped model at any loop count. The model is open-sourced in BF16, MXFP8, NVFP4, and GGUF quantized checkpoints.

## Introduction and Theoretical Foundation

Looped (latent-recurrent) language models offer a complementary compute-scaling axis to standard token-space scaling (Dehghani et al., 2019; Saunshi et al., 2025; Geiping et al., 2025; Zhu et al., 2025). By repeatedly applying a shared block of layers, a looped model spends more computation per token in latent space without adding parameters. This addresses a key limitation of test-time compute scaling: models generate longer reasoning traces, and latency, KV-cache footprint, and serving cost grow with every generated token.

Dory1.0-4B brings this idea to a fully post-trained model built for reasoning and agentic tasks. It is trained with a mixture of loop counts so that a single checkpoint can be served at any loop count $k$, trading inference compute for accuracy per request with no retraining and no model swap.

The architecture is based on the **normalized Transformer (nGPT)** (Loshchilov et al., 2025), which retains the residual-update rule:

$$
s_{0} = x, \qquad u_{t} = N(s_{t-1} + x), \qquad s_{t} = N(F_{\theta}(u_{t})), \qquad t = 1, \ldots, k.
$$

where $x$ is the input block's output, $F_{\theta}$ is the shared recurrent block, $N$ is per-token L2 normalization, and $k$ is the loop count. The effective forward depth is $32 + 40k$, ranging from 72 to 192 physical-layer applications.

## Methodology

### Architecture

Dory1.0-4B uses a normalized Transformer backbone with:
- **72 physical layers** partitioned into a 16-layer input block, a 40-layer shared recurrent block, and a 16-layer output block
- **3:1 interleaving** of sliding-window attention (SWA) and full-attention layers within each 8-layer pattern
- SWA uses window $W = 1024$ tokens with RoPE base $\theta_S = 10^4$; full attention uses $\theta_F = 10^7$
- Gated grouped-query attention with 16 query heads and 4 KV groups
- Input-dependent sigmoid attention-output gates
- Initial raw residual-rate, query–key scale, and logit-scale parameters at 0.0177

### Pretraining

1. **NVFP4 stage**: ~4.535T tokens using Megatron-LM and NVIDIA Transformer Engine's NVFP4 blockscaling, yielding ~20% end-to-end speedup over BF16. Two-phase curriculum: phase-1 at sequence length 8,192, then phase-2 at 32,768. Learning rate held at $3.2 \times 10^{-4}$ through 4.5T tokens, then minus-square-root decay toward $10^{-5}$ at 6T.

2. **BF16 loop continuation**: ~1.465T additional tokens with loop count sampled uniformly from $k \in \{1, 2, 3, 4\}$. A rank-level override fraction of 0.1 forces a rotating subset of replicas to execute exactly one loop. Uses **truncated backpropagation through loops (TBPTL)** with depth one.

3. **Long-context extension**: 300B tokens at sequence length 262,144, increasing full-attention RoPE base from 500,000 to 10,000,000, with constant learning rate $10^{-5}$.

### Post-Training

**SFT**: Nine epochs over packed 256K-token sequences with both thinking and non-thinking responses, retaining random loop sampling. Between epochs 1 and 9, SciCode improved from 13.11% to 18.14%, Terminal-Bench v2.1 from 4.2% to 13.5%, and GPQA Diamond from 43.13% to 54.95%.

**RLVR**: 200 steps of asynchronous GRPO across 23 environments. Uses a **difficulty-aware loop curriculum** with routing score:

$$
s_{i} = \alpha q_{i} + (1 - \alpha) u_{i}, \qquad u_{i} \sim \mathrm{Uniform}(0, 1).\tag{1}
$$

where $q_i$ is the midpoint quantile of each prompt's profiled pass rate and $\alpha = 0.3$. Prompts with lower routing scores fill deeper-loop slots first. The loop curriculum improves all six benchmarks at both one and four loops, with the largest gains on agentic tasks at four loops (e.g., Terminal-Bench v2.1 from 14.0% to 21.4%).

### Quantization

Three weight-only formats evaluated: **MXFP8** (PTQ only), **NVFP4** and **Q5_K_M** (with quantization-aware distillation, QAD). QAD minimizes full-vocabulary forward KL divergence:

$$
\mathcal{L}_{\mathrm{KD}} = \frac{1}{\sum_{t} m_{t}} \sum_{t} m_{t} \sum_{v \in \mathcal{V}} p_{T}(v \mid x_{\leq t}, k) \log \frac{p_{T}(v \mid x_{\leq t}, k)}{p_{S}(v \mid x_{\leq t}, k)}.\tag{2}
$$

## Empirical Validation / Results

### Base Model Evaluations

Base model performance improves with loop count, saturating around 4 loops. Key results at $k \in \{1, 2, 4\}$:

| Task | Metric | k=1 | k=2 | k=4 |
|------|--------|-----|-----|-----|
| MMLU (Str) | 5-shot EM | 68.20 | 70.77 | 71.19 |
| MMLU-Pro | 5-shot CoT EM | 51.52 | 55.06 | 56.09 |
| GSM8K | 8-shot CoT EM | 86.43 | 89.99 | 89.84 |
| MATH-500 | 4-shot greedy EM | 60.60 | 65.00 | 69.40 |
| HumanEval | 0-shot pass@1 | 58.33 | 59.76 | 62.34 |
| MBPP-Sanitized | 3-shot pass@1 | 57.09 | 61.50 | 60.89 |
| ARC-Challenge | 25-shot acc_norm | 87.63 | 89.08 | 88.74 |
| HellaSwag | 10-shot acc_norm | 78.52 | 79.53 | 79.76 |
| WinoGrande | 5-shot acc | 74.11 | 76.24 | 77.43 |
| RACE-H | 0-shot acc | 83.06 | 85.26 | 86.03 |

### Main Results (Post-Trained)

| Task | Dory 1 loop | Dory 2 loops | Dory 4 loops | Qwen3.5 4B | Qwen3.5 9B | Gemma 4 12B | Gemma 4 26B-A4B | Qwen3.6 35B-A3B |
|------|------------|-------------|-------------|------------|------------|-------------|-----------------|-----------------|
| SciCode | 27.3 | 34.94 | 35.11 | 25.82 | 28.21 | 37.89 | 40.32 | 37.24 |
| SWE-Bench Verified | 50.3 | 59.5 | 61.8 | 54.7 | 61.0 | 47.3 | 57.40 | 73.4† |
| Terminal-Bench v2.1 | 12.5 | 18.1 | 22.2 | 15 | 24 | 27.0 | 35 | 43 |
| τ³-Banking | 11.6 | 12.89 | 14.69 | 5.03 | 7.22 | 8.63 | 13.66 | 9.54 |
| GDPval-AA v2 (Elo) | 815 | 946 | 967 | 424 | 546 | 657.4 | 751.6 | 987 |
| GPQA Diamond | 69.19 | 72.41 | 72.16 | 77.71 | 80.93 | 73.36 | 80.43 | 85.35 |
| HLE | 8.25 | 10.89 | 10.43 | 11.31 | 15.43 | 13.67 | 17.79 | 19.18 |
| IFBench | 74.29 | 77.94 | 79 | 53.83 | 59 | 73.44 | 77.89 | 57.44 |
| AA-LCR | 63 | 71 | 73 | 64.7 | 68.4 | 58.9 | 64.6 | 68.5 |
| GDP.pdf | 3.4 | 6.2 | 6.0 | 2.8 | 4.6 | 4.00 | 3.60 | 4.40 |
| AA-Omniscience (acc) | 10.03 | 10.08 | 10.03 | 15.02 | 18.05 | 17.20 | 21.13 | 19.95 |
| AA-Omniscience (non-halluc.) | 76.42 | 79.74 | 79.38 | 20.18 | 17.39 | 21.84 | 16.36 | 65.25 |
| **Average** | **35.17** | **39.67** | **40.60** | 28.53 | 32.21 | 32.59 | 36.73 | 42.30 |

### Token Efficiency

Going from $k=1$ to $k=4$ reduces output tokens per attempt by 1.3–2.5× on agentic benchmarks:

| Task | Turns/attempt (k=1→4) | Output tokens/attempt (k=1→4) | Reduction |
|------|------------------------|-------------------------------|-----------|
| GDPval-AA v2 | 116 → 49 | 212k → 85k | 2.5× |
| Terminal-Bench v2.1 | 134 → 82 | 109k → 54k | 2.0× |
| SWE-Bench Verified | 88 → 73 | 33.6k → 24.8k | 1.35× |
| τ³-Banking | 21.3 → 19.8 | 23.0k → 17.6k | 1.3× |

The second loop gives the largest accuracy gain per unit of added compute, making $k=2$ a good default.

### Inference Optimizations

- **Last-loop KV caching**: Keeps KV memory at one-loop footprint for any $k$. Score changes of −2.12 to +0.42 points relative to per-loop caching at $k=4$, with nearly unchanged output lengths.
- **Self-speculative decoding**: Uses $k=1$ drafting and $k=4$ verification with $\gamma=4$ draft tokens, achieving 1.11–1.53× speedups in mean prefill-plus-decode request duration.
- **Quantization speedups**: MXFP8 achieves up to 1.84× and NVFP4 up to 2.44× speedup over BF16.

## Theoretical and Practical Implications

**Theoretical significance**: Dory1.0-4B demonstrates that latent-recurrent computation is a viable and complementary scaling axis to token-space test-time compute. The results show that a 4B model with deep looping can compete with models 9× its size on agentic and coding tasks, suggesting that recurrent depth can substitute for parameter count in certain domains.

**Practical implications**:
- **Elastic depth**: A single checkpoint serves different compute budgets, enabling per-request trade-offs between latency and accuracy without model swapping.
- **Token efficiency**: Looping reduces output token count on agentic tasks, partially offsetting higher per-token compute costs. This is particularly valuable for agentic workloads where output tokens dominate cost.
- **Memory efficiency**: Last-loop KV caching decouples KV-cache footprint from loop count, making deep looping deployable with standard serving infrastructure.
- **Quantization**: Multiple quantized formats (MXFP8, NVFP4, Q5_K_M) enable deployment across different hardware constraints with minimal accuracy loss.

## Conclusion

Dory1.0-4B is the first open looped language model in the Nemotron Labs family, demonstrating that latent-recurrent architectures can be successfully pretrained, post-trained, and deployed at scale. Key findings:

1. Looping improves accuracy on coding, agentic, and long-context benchmarks, with the largest gains on agentic tasks.
2. A difficulty-aware loop curriculum in RLVR is crucial for unlocking gains at deeper loop counts.
3. Looping improves token efficiency by reducing the number of actions and output tokens needed to complete agentic tasks.
4. Last-loop KV caching and self-speculative decoding make deep looping practical for serving.
5. Quantization (MXFP8, NVFP4, Q5_K_M) preserves most accuracy while providing up to 2.44× inference speedup.

The open-sourced checkpoints (BF16, MXFP8, NVFP4, GGUF) enable the research community to further explore latent-recurrent computation, elastic depth, and the trade-offs between parameter count and recurrent depth in language models.

---

_Markdown view of https://picx.dev/p/OWO94h, served by PicX — AI-generated visual whiteboard summaries of research papers._