Summary of "Explore More, Drift Less: Outcome-Only Reinforcement Learning Can Suffice for Long-Horizon Interactive Agents"

Summary (Overview)

• Core thesis: The paper challenges the prevailing belief that outcome-only reinforcement learning (RL) hits a ceiling for long-horizon interactive agents on small open models, arguing this ceiling is an artifact of two specific training failures rather than a fundamental limitation.
• Key contribution: Introduces CANOPY (Coverage-ANchored On-PolicY RL), a minimalist protocol that addresses signal starvation (insufficient exploration leading to degenerate gradient groups) and policy drift (sampling distribution collapse during repeated task revisits).
• Landmark results: A Qwen3-14B policy trained with CANOPY topped the AppWorld leaderboard (Feb. 2026) with Test-Normal TGC 86.9 and Test-Challenge 67.6, outperforming far heavier inference-time systems built around frontier closed models.
• Transferability: The same design principles improved SWE-bench Verified resolve rates by 16.6 points on Qwen3.5-9B, demonstrating generalizability to real-repository software repair.
• Position: The authors argue that plain agentic RL on small open models is not exhausted—capabilities that elaborate systems assemble at inference time can be internalized directly into model weights.

Introduction and Theoretical Foundation

Background and Motivation

LLM-driven agents are increasingly deployed for coding-centric tasks, typically through engineered harnesses wrapped around frontier (usually closed) models. The authors explore an alternative: post-training a small open model into a domain specialist via reinforcement learning, where the interaction skills are internalized into the weights.

The paper focuses on AppWorld (Trivedi et al. 2024), a long-horizon benchmark of everyday digital-application tasks where an agent iteratively writes and executes Python against a live environment over dozens of turns, judged only by held-out, state-based unit tests.

The Apparent Ceiling and Its Compensation Strategies

Prior work on AppWorld has compensated around the policy rather than strengthening RL itself:

• Denser/step-level rewards — after hardest tasks were measured as "harmful" (Chen et al. 2025; Dai et al. 2026a)
• SFT cold starts — after skipping them collapsed scores (Wang et al. 2026c)
• Skill libraries, curated memory, multi-agent orchestration — anchored to frontier closed models

The Two Failure Mechanisms

Failure 1: Signal Starvation. A group-relative estimator (like GRPO) yields useful gradient only when a task's rollout group contains both successes and failures. For per-rollout success rate $p$ and group size $n$, the probability of drawing an informative group is:

$$P_{\mathrm{sig}}(p, n) = 1 - p^n - (1-p)^n \tag{4}$$

With small groups ($n \leq 8$) used by prior work and low $p$ on hard tasks, most groups are degenerate—yielding zero gradient. The occasional isolated success is amplified by standard-deviation normalization into a high-variance spike.

Failure 2: Policy Drift. When training must extract many gradient steps from a small, repeatedly revisited task pool, the sampling distribution can collapse. Without an anchor, entropy decays and exploration dies exactly when saturation makes informative groups rare. Four causes are identified:

1. An unanchored objective lets the sampling distribution narrow
2. Reusing rollouts across updates (mini-batching) changes what the update sees
3. Length-imbalanced loss averaging down-weights long, error-recovering episodes
4. Densified or substituted reward signals carry their own errors

Methodology

The Agentic RL Loop

For a task prompt $q$, the policy $\pi_\theta$ proposes actions, the environment executes them and returns feedback, repeating until termination or budget exhaustion. Episodes are trajectories $\boldsymbol{o} = (a_1, e_1, a_2, e_2, \ldots)$ interleaving action tokens $a_t$ and environment tokens $e_t$.

Two reward formulations are considered:

Dense pass-fraction reward (partial credit):

$$r_i^{\text{dense}} = \frac{1}{M}\sum_{j=1}^{M}\text{pass}(u_j, o_i) \in [0, 1] \tag{1}$$

Sparse reward (fully-correct-only):

$$r_i^{\text{sparse}} = \mathbf{1}\left[\sum_{j=1}^{M}\text{pass}(u_j, o_i) = M\right] \in \{0, 1\} \tag{2}$$

Group-relative policy optimization (GRPO) standardizes rewards within a group of $n$ trajectories:

$$\hat{A}_i = \frac{r_i - \text{mean}(r_1, \ldots, r_n)}{\text{std}(r_1, \ldots, r_n)} \tag{3}$$

CANOPY Protocol Components

Explore More (addresses signal starvation):

1. Size the group from data: For target coverage $\tau$ and estimated hardest-tier success rate $\hat{p}_{\min}$:

$$n \gtrsim \frac{\ln(1-\tau)}{\ln(1-\hat{p}_{\min})} \quad (\text{for small } p) \tag{5}$$

2. Keep the hardest tasks: They are not intrinsically harmful, only starved—once coverage is restored, they become the last remaining gradient source
3. Uncapped per-turn generation: Environment returns are truncated, but the policy's own generation is capped only by the total response budget

Drift Less (addresses policy drift):

• Strictly on-policy updates: The gradient mini-batch is the whole rollout batch; one update per batch with no stale reuse
• Sparse reward (Equation 2): No proxy to drift toward
• KL anchoring to the base model with coefficient $\beta$
• Token-level loss over action tokens only (environment tokens masked)

The per-token importance ratio:

$$\rho_{i,t}(\theta) = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t} \mid q, o_{i,<t})} \tag{6}$$

The clipped surrogate:

$$\mathcal{S}_{i,t} = \min\left(\rho_{i,t}\hat{A}_i, \text{clip}(\rho_{i,t}, 1-\epsilon_{\text{low}}, 1+\epsilon_{\text{high}})\hat{A}_i\right) \tag{7}$$

The final loss (with $\mathcal{F}$ the fault-quarantined episodes and $N$ the pooled action-token count):

$$\mathcal{L}(\theta) = \frac{1}{N}\sum_{i \notin \mathcal{F}}\sum_{t=1}^{|o_i|} M_{i,t}\left[-\mathcal{S}_{i,t} + \beta D_{\text{KL}}(\pi_\theta, \pi_{\text{ref}})_{i,t}\right] \tag{8}$$

Key differences from standard GRPO: the single denominator $N$ pools all action tokens (not per-sequence normalization), and the KL penalty pulls the sampling distribution back toward the base model's breadth.

Fault Quarantine

Episodes ending without a verdict are classified: agent-induced terminations (infinite loops, memory exhaustion) score 0 as genuine failures; only exogenous faults attributed to the serving layer enter $\mathcal{F}$ and are quarantined before scoring.

Test-Time Budget Transfer

Training is done at a moderate budget (50 turns, 32k tokens); at test time the turn count and context length are raised (100 turns, 61k tokens)—no search, no multi-rollout selection.

Empirical Validation / Results

Main Results on AppWorld

Table 1: AppWorld Results (official leaderboard or cited papers)

Note: superscripts denote mean@k variants; ‡ uses a non-standard joint-scenario protocol.

Training Configuration (Table 2)

Metric Map (Table 3)

Ablation Results (Figure 6, Test-Normal TGC mean@4)

Transfer to SWE-bench Verified (Table 4)

Key Training Dynamics Findings

• Signal coverage: All-fail groups disappear within ~10 steps; as train reward saturates above 0.99, informative groups shrink (the "signal runs out" regime predicted by Equation 4)
• KL anchor effect: Past step ~70, the unanchored run's entropy collapses (0.038) and Dev stalls at 81.6, while the anchored run stays healthy (0.217) and improves to 87.3
• Budget transfer: Gains concentrate on the hardest tier and unseen applications; the trained policy at 50t/32k (79.5) far exceeds the base at 100t/61k (32.4), proving RL—not budget—buys the capability
• Capability expansion: Single-run TGC of 86.9 exceeds the base's best@4 of 58.9 by 28 points, contradicting claims that RL narrows capability boundaries

Theoretical and Practical Implications

Theoretical Implications

1. Reinterpreting "harmful" hard tasks: The paper provides a mechanistic explanation for why prior work found hard tasks harmful—they were starved of signal, not intrinsically damaging. Once exploration is scaled, these tasks flip from poison to the most valuable data.

2. Unifying contradictory findings: The signal starvation and policy drift mechanisms reconcile seemingly contradictory results across papers: dense rewards seem necessary at small group sizes but become unnecessary (even slightly harmful, −1.8) once exploration is adequate.

3. RL expands capability boundaries: Against claims that RL pass@1 ≤ base pass@k (Szot et al. 2026; Yue et al. 2025), CANOPY's single run exceeds the base's best@4 by 28 points, demonstrating that at this scale RL adds capability resampling cannot reach.

4. The ceiling is trainable: The field's turn toward harness engineering (skills, memory, orchestration) answers trainable ceilings that this work places higher than previously reported.

Practical Implications

1. Minimalism works: CANOPY changes no optimizer and adds no auxiliary module—all ingredients are well-understood (GRPO, KL anchoring, token-level loss, larger groups).

2. Small open models can compete: A single 14B policy trained by interaction alone holds its own against far heavier inference-time systems on stronger backbones, with the entire capability in the weights—deployable by anyone.

3. Transferable design principles: The protocol transfers across domains (application operation → software repair) with only minimal re-tuning ($n=16$ instead of 32, KL coefficient $10^{-2}$ instead of $10^{-4}$, terminal reward −0.2 for non-reviewable patches).

4. Test-time budget transfer: A simple, cost-effective strategy—train at moderate budget, raise turn/context at test time—concentrates gains where they matter most (hardest tasks, unseen applications).

Conclusion

Main Takeaways

The apparent limitations of outcome-only RL for long-horizon agents stem not from sparse rewards alone but from two practical bottlenecks: obtaining informative outcome variation (signal starvation) and limiting policy drift. CANOPY addresses both with a simple recipe—explore more, drift less—using well-understood ingredients throughout.

At submission time, a single open 14B policy trained with this recipe reached the top of the AppWorld leaderboard, and the same principles improved software repair on SWE-bench Verified by 16.6 points. The results suggest that for well-defined domains, capabilities that elaborate systems assemble at inference time can instead be internalized into a small open model's weights.

Future Directions

1. Environment scaling: Testing whether gains persist with larger, more diverse task distributions—including multilingual software engineering and harder emerging benchmarks.

2. Better RL algorithms: More sample-efficient methods could make larger-scale training affordable while managing the exploration–exploitation trade-off more directly (e.g., adapting group size or task sampling to the current success probability $p$).

3. Domain mid-training: Strengthening domain knowledge before RL could improve the base model's coverage and raise the attainable ceiling of outcome-based RL, complementing rather than replacing interaction-based post-training.

The authors plan to release the complete training stack at https://github.com/AlibabaResearch/SignalCoverageRL.