NeoHorse-1: Towards Recursive Self-Improvement via Agentic Post-Training with Routing Harness

Summary (Overview)

• Core contribution: NeoHorse-1 introduces a family of agent-native models (4B and 9B scales) that demonstrate how a deployed routing harness can serve as a concrete mechanism for recursive self-improvement (RSI), where execution trajectories, routing signals, and recorded outcomes are converted into the next round of training data.

• Key mechanism: The system couples a heterogeneous model pool with intelligent routing that records capability demand predictions, service tier selections, and interaction outcomes for every turn, which are then transformed into user-turn training examples preserving interleaved reasoning, tool calls, and harness context

• Methodological innovations: Routing-guided curriculum learning organizes SFT into a three-stage progression based on capability demand scores, extended to routing-guided on-policy distillation(OPD) where a teacher supervises student-generated responses under the same staged progression

• Quantitative results: Post-training lifts macro-average scores from 58.94 to 64.87 for the 4B model and from 65.60 to 69.04 for the 9B model across ten benchmarks, substantially narrowing the gap between the post-trained 4B model and the 9B base model

• RSI validation: The post-trained 4B model matches or exceeds Qwen3.5-9B on several benchmarks, providing preliminary evidence that the evaluation–selection–update loop can be operationalized across model generations

Introduction and Theoretical Foundation

The paper addresses recursive self-improvement(RSI), the vision in which AI systems take a growing part in their own improvement process. The authors argue that RSI requires a concrete mechanism through which a system observes its own capabilities and converts that evidence into the next round of learning.

Key theoretical position: A deployed routing harness already contains such a mechanism. Beyond producing task outputs, agentic interaction leaves:

• Execution trajectories that ground training in real interaction
• Routing signals that characterize capability demand
• Recorded outcomes that reveal where the model still falls short

The work builds on several established research threads:

• Agentic model post-training: Trajectory-based SFT (FireAct, AgentTuning, Agent-FLAN, AgentBank), on-policy distillation [1], and harness design research (SWE-agent, Terminal-Lego)
• LLM routing: FrugalGPT, RouteLLM, and Agentic Routing [33],which extends routing to multi-agent LLM systems where a decision layer selects models based on requests
• Curriculum learning: Training strategies presenting examples according to estimated difficulty, with routing systems providing alternative difficulty signals

The central insight is that the prediction–action–outcome separation [33] in routing records enables the harness to function both as a serving-time decision mechanism and as a source of data feedback within harness-mediated RSI

Methodology

Data Pipeline (Section 3)

The corpus is organized at three linked granularities:

Quality control involves three stages:

1. Deduplication and decontamination: Exact and near-duplicate removal; evaluation overlap screening 2.. Structural validation: Rule-based verification of payload readability, causal event order, tool-call/result closure, producing three outcomes: internally complete, partially recoverable, quarantined 3.. Semantic evaluation: Six independent quality dimensions—goal attainment, instruction adherence, tool use, evidence consistency, error recovery, termination—each assigned PASS, WARN, FAIL, or NOT_EVALUATED

Data characterization uses three axes at the subscene level:

• Scene: What the user is doing and context (task type, application domain, use context, asking/doing)
• Goal: What the user expects to achieve (acceptance criteria, cross-turn relations: new, continued, modified, resumed)
• Outcome: Verifiable result of the attempt

Agentic Routing Signals

The harness router estimates capability demand from current request, recent dialogue, previous routing decisions, and available execution state, assigning each turn to one of four service tiers:

$$C0 \text{ (bounded low-risk)}, \quad C1 \text{ (general-purpose default)}, \quad C2 \text{ (multi-step reasoning/execution)}, \quad C3 \text{ (maximum capability)}$$

For each turn, the corpus retains: the router's raw prediction, the policy-adjusted decision, and the tier actually served

Agentic Post-Training (Section 4

Agentic supervision: User turns are serialized using the Qwen3.5 chat template, retaining reasoning within the current turn while omitting reasoning from earlier turns. The SFT objective uses a binary token-level loss mask $m_{i,t}$:

$$\mathcal{L}_{\mathrm{SFT}}(\theta; \mathcal{B}) = - \frac{\sum_{i \in \mathcal{B}} \sum_{t=2}^{T_{i}} m_{i,t} \log p_{\theta}(x_{i,t} \mid x_{i,<t})}{\sum_{i \in \mathcal{B}} \sum_{t=2}^{T_{i}} m_{i,t}}. \tag{1}$$

Routing-guided curriculum: Each example receives a routing score $s_i$ using either hard ordering (assigned tier index $k_i$) or soft ordering (score-weighted mean tier index):

$$s_{i} = \left\{ \begin{array}{ll} k_{i}, & \text{hard ordering,} \\ \sum_{k=0}^{3} k \cdot \pi_{i,k}, & \text{soft ordering.} \end{array} \right. \tag{2}$$

Training proceeds over three stages, progressively introducing higher-scored examples while reserving some lower-scored examples for later stages

Routing-guided on-policy distillation: The student generates responses from starting contexts scheduled by routing scores; a fixed teacher supplies next-token distributions. The objective is a response-normalized reverse KL divergence:

\mathcal{L}_{\mathrm{OPD}}(\theta; \mathcal{R}) = \frac{1}{\sum_{r \in \mathcal{R}} w_{r}} \sum_{r \in \mathcal{R}} \frac{w_{r}}{L_{r}} \sum_{t=1}^{L_{r}}} D_{\mathrm{KL}}\left(\widetilde{P}_{\theta,r,t} \parallel \widetilde{Q}_{r,t}\right), \tag{3}

where $\widetilde{P}_{\theta,r,t}$ and $\widetilde{Q}_{r,t}$ are coarsened distributions over $K+1$ bins (top-$K$ candidates plus one bin for remaining probability mass). The stage-wise objective combines the context distribution $\rho_j$ with student generation:

$$\mathcal{L}_{\text{R-OPD}}^{(j)}(\theta) = \mathbb{E}_{\substack{\mathcal{C}\sim \rho_{j}\\ \mathcal{R}\sim p_{\bar{\theta}}(\cdot|\mathcal{C})}}\left[\mathcal{L}_{\text{OPD}}(\theta;\mathcal{R})\right],\qquad j\in\{1,2,3\}. \tag{4}$$

Empirical Validation / Results

Main Benchmark Results

Table 1 | Comparison of NeoHorse-1-4B with representative 4B-scale open-weight models. Higher is better. Best and second-best results shown in bold and underlined, respectively.

| Model | BFCL v4 | Vita Bench | τ² Bench | Pinch Bench | WorkBuddy Bench | QwenClaw Bench | HumanEval | LiveCode Bench v6 | IF Bench | IFEval | Avg. | |---|---|---|---|---|---|---|---|---|---|---| | Qwen3.5-4B | 61..02 | 21..50 | ͏84.͏29 | ͏71.͏19 | ͏24.͏62 | ͏38.͏47 | ͏87.͏20 | ͏53.͏71 | ͏60.͏33 | ͏87.͏06 | ͏58.͏94 | | Spark-X2..5-4B | ͏63.͏71 | ͏37.͏00 | ͏77.͏72 | ͏62.͏37 | ͏26.͏47 | ͏43.͏52 |│ ͏92.͏07 |│ ͏54.͏86 |│ ͏73.͏33 |│ ͏91.͏13 |│ ͏62.͏22 | | Gemma-4-E4B-it |│ ͏47.͏18 |│ ͏5.͏00 |│ ͏43.͏60 |│ ͏47.͏60 |│ ͏11.͏65 |│ ͏22.͏98 |│ ͏84.͏76 |│ ͏52.͏00 |│ ͏40.͏00 |│ ͏74.͏68 |│ ͏42.͏95 | | Nanbeige-4.2-3B |│ ͏67.͏28 |│ ͏31.͏50 |│ ͏85.͏08 |│ ͏66.͏78 |│ ͏21.͏03 |│ ͏40.͏66 |│ ͏98.͏78 |│ ͏72.͏50* |│ ͏55.͏00 |│ ͏84.͏47 |│ ͏62.͏31 | | Agents-A1-4B |│ ͏46.͏60 |│ ͏39.͏25 |│ ͏81.͏00 |│ ͏75.͏07 |│ ͏33.͏37 |│ ͏43.͏16 |│ ͏92.͏68 |│ ͏56.͏57 |│ ͏63.͏33 |│ ͏83.͏55 |│ ͏61.͏46 | | NeoHorse-1-4B |│ 61..79 |│ ͏32.͏00 |│ 88..46 |│ 77..33 |│ 34..41 |│ 44..68 |│ 96...95 |│ ͏59.͏43 |│ ͏65.͏33 |│ ͏88.͏35 |│ 64...87 |

Note: ∗ denotes a result from the model's official report

Table 2 | Comparison of NeoHorse-1-9B with representative larger-scale models. Higher is better.

| Model | BFCL v4 | Vita Bench | τ² Bench | Pinch Bench | WorkBuddy Bench | QwenClaw Bench | HumanEval | LiveCode Bench v6 | IF Bench | IFEval | Avg. | |---|---|---|---|---|---|---|---|---|---|---| | Qwen3.5-9B | 64..88 | ͏31..25 | ͏88.͏04 |│ ͏74.͏55 |│ ͏39.͏60 |│ ͏44.͏04 |│ ͏92.͏68 |│ ͏65.͏14 |│ ͏66.͏33 |│ ͏89.͏46 |│ ͏65.͏60 | | Granite-4.2-8B |│ ͏52.͏06 |│ ͏23.͏00 |│ ͏62.͏28 |│ ͏56.͏93 |│ ͏35.͏07 |│ ͏37.͏01 |│ ͏96.͏34 |│ ͏72.͏00 |│ ͏78.͏00 |│ ͏92.͏98 |│ ͏60.͏57 | | Ornith-1.5-9B |│ ͏65.͏03 |│ ͏26.͏75 |│ ͏83.͏68 |│ ͏68.͏22 |│ ͏29.͏29 |│ ͏47.͏27 |│ ͏93.͏90 |│ ͏47.͏43 |│ ͏40.͏00 |│ ͏71.͏35 |│ ͏57.͏29 | | Gemma-4-12B-it |│ ͏62.͏06 |│ ͏36.͏50 |│ ͏59.͏37 |│ ͏58.͏89 |│ ͏29.͏65 |│ ͏43.͏53 |│ ͏100.͏00 |│ ͏73.͏14 |│ ͏77.͏67 |│ ͏94.͏27 |│ ͏63.͏51 | | Muse-Glimmer-30B |│ ͏53.͏74 |│ ͏48.͏50 |│ ͏76.͏64 |│ ͏71.͏35 |│ ͏45.͏85 |│ ͏46.͏11 |│ ͏98.͏17 |│ ͏65.͏71 |│ ͏78.͏67 |│ ͏93.͏90 |│ ͏67.͏86 | | NeoHorse-1-9B |│ 67..43 |│ ͏42.͏25 |│ 90..82 |│ 82..25 |│ ͏40.͏15 |│ 48..73 |│ ͏98.͏17 |│ ͏65.͏14 |│ ͏66.͏33 |│ ͏89.͏09 |│ 69..04 |

Key Findings

1. Broad gains at both scales: NeoHorse-1-4B outperforms Qwen3.5-4B on every benchmark; gains concentrated on harness-based agent tasks and coding benchmarks 2.. Scale remains beneficial: NeoHorse-1-9B consistently outperforms NeoHorse-1-4B, with advantages concentrated on tasks requiring sustained state tracking, tool interaction, and revision of intermediate decisions 3.. Training compensates for scale: NeoHorse-1-4B already matches or exceeds Qwen3.5-9B on several benchmarks

Data Analysis Results

Table 3 | Comparison of routing-harness trajectories with public tool-agent data (Toucan) under the same routing-guided training configuration. Higher is better.

Scaling routing-harness supervision shows steady aggregate gains: the development-suite average increases from 69.31 (base model) to 71.45 at the largest data scale, measured in unique supervised tokens on a logarithmic axis

Theoretical and Practical Implications

Theoretical Significance

1. Operationalizing RSI: The paper provides a concrete instantiation of how a system can observe its own capabilities and convert that evidence into the next round of learning—the core requirement for RSI that has remained largely theoretical

2. Routing as a training signal: The prediction–action–outcome separation in routing records provides a principled way to estimate capability demand without conflating the model actually served with the difficulty label, since executed routes may reflect user overrides, service availability, and deployment policies

3. Curriculum learning from routing: Routing-derived scores offer an alternative to explicit difficulty labels or dataset-specific heuristics, which are costly or impractical to obtain

Practical Implications

1. Data efficiency: Routing-harness trajectories provide stronger and more transferable agentic supervision than public synthetic trajectories under matched training conditions (+6.26 average improvement), with largest gains on HumanEval(+8.54) and τ²-Bench (+11.31)

2.. Scalable supervision: The steady gains from scaling routing-harness supervision suggest that data quantity can be treated as a capability-dependent allocation decision, jointly determined by supervision quality, coverage, and the capability profile targeted by post-training

3.. Closed-loop improvement: The capability-guided allocation step closes an evaluation–selection–update loop where what the system learns to do shapes what it learns from next, enabling continued improvement as updated models return to the harness

Conclusion

NeoHorse-1 constitutes an initial prototype of harness-mediated recursive self-improvement, demonstrating that the everyday operation of a routing harness already supplies both the experience and the feedback needed to improve the models that run within it

Main takeaways:

• Agentic post-training with routing-guided curriculum learning and on-policy distillation delivers consistent gains at both 4B and 9B scales
• The post-trained 4B model substantially narrows the aggregate gap to the 9B base model, showing training can compensate for part of the performance gap associated with model scale
• Larger models retain advantages on tasks requiring iterative debugging, recovery from execution failures, and state maintenance over long action sequences

Future directions:

1. Iterating the feedback loop across successive model generations to test whether gains can continue to accumulate as capabilities evolve
2. Extending experience collection, capability-guided allocation, and curriculum design to the broader range of tasks, execution environments, and models served by the harness
3. Sharpening the routing signal: Turning the recorded prediction–action–outcome separation into well-calibrated estimates of difficulty and deficiency—including supervision that trains the router itself—would let the harness guide not only which data to use and where to allocate it, but what a model should attempt next

The authors view NeoHorse-1 as evidence that harness-mediated recursive self-improvement can move from design to practice, though the current validation reflects a single pass of the evaluation–selection–update loop and requires testing across successive iterations