FrogNano: Training a 4B Coding Agent via Online Task Synthesis

Summary (Overview)

• FrogNano is a 4B-parameter coding agent that achieves competitive performance on software engineering (SWE) benchmarks, rivaling models 8–25× its size (32B–100B+), without distillation from larger models.
• The training recipe relies on three key ingredients: (1) Leaf, a lightweight Claude-Code-style harness with five typed tools; (2) TaskPilot, an online policy-adaptive task synthesis pipeline that generates tasks at the "learnability frontier" of the current policy; and (3) iterative RL using DPPO with a success-gated log-length penalty.
• FrogNano reaches 61.5% on SWE-bench Verified, 37.6% on SWE-bench Pro, 31.1% on Terminal-Bench v2, and 47.3% on PatchEval-Verified, using only ~1,500 synthetic tasks across 5 RL iterations.
• The harness choice is critical for compact models: switching from R2E-Gym to Leaf improves Qwen3.5-4B's SWE-bench Verified solve rate from 8.3% to 37.2%, while a larger model (MiniMax-M2.5) is unaffected (66.5% on both).
• The paper demonstrates that online task synthesis calibrated to the current policy outperforms training on real tasks filtered by the same learnability criterion (61.5% vs. 48.0% on SWE-bench Verified).

---

Introduction and Theoretical Foundation

Motivation

Most progress in SWE agents has been driven by frontier models behind proprietary APIs or open-weight models with tens of billions of parameters. These systems are expensive to serve and difficult to deploy locally. The authors ask: can a substantially smaller model (4B) become a capable repository-level coding agent?

Key Challenges Addressed

1. Data: Curating executable SWE tasks from real development histories is costly. Existing approaches include mining issue-PR pairs (SWE-Gym, SWE-rebench), commit-derived tasks (R2E-Gym), and synthetic generation (SWE-smith, BugPilot). However, none use current-policy feedback to adapt task generation to the evolving learner.

2. Harness: Compact models struggle with complex agent interfaces. Under R2E-Gym, Qwen3.5-4B fails to terminate successfully in ~96% of trajectories, often not producing the required submission action.

3. RL: Training requires executable tasks matched to the current policy's capabilities—tasks never solved provide no signal, while saturated tasks provide little relative signal.

Theoretical Basis

The work builds on:

• Adaptive curricula from RL literature: GoalGAN, POET, PAIRED, PLR, and ACCEL all use current-policy outcomes to shape task distributions.
• Group-relative policy optimization (DPPO/GRPO) for stable RL training.
• The insight that task difficulty should be calibrated to the learner's current ability, not estimated intrinsically.

---

Methodology

1. Leaf Harness

A lightweight coding-agent harness inspired by Claude-Code-style tool conventions:

• Five typed tools: read, write, edit, glob, bash
• Simple termination rule: a response without a tool call is treated as the final answer
• Omits planning modes, permission dialogues, reminders, and auxiliary tools

The harness is used for both evaluation and RL training. During training, model-generated tokens (reasoning, text, tool calls) are included in the loss; task prompts and tool outputs are masked.

2. TaskPilot: Policy-Guided Task Synthesis

Pipeline (Figure 4):

1. Start from real repository snapshots (SWE-rebench or Scale-SWE)
2. A task-generation model produces: problem statement, gold patch, hidden fail-to-pass (F2P) tests
3. Validate executability: F2P tests fail on original snapshot, pass after gold patch; P2P suite stable
4. Evaluate candidates with current-policy rollouts

Calibration: For each candidate, collect $N$ trajectories. Estimate resolve rate:

$$\hat{p}_{\text{policy}} = \frac{1}{N} \sum_{j=1}^{N} R_j$$

where $R_j \in \{0, 1\}$ indicates whether trajectory $j$ passes all grading tests.

Admission criteria:

• Broad learnable region: $0 < \hat{p}_{\text{policy}} < 1$
• Iterations 1–4 target: $\hat{p}_{\text{policy}} = 0.5$
• Revised iteration 5 targets: $0 < \hat{p}_{\text{policy}} \leq 0.5$

Candidates outside the target are refined (e.g., adjusting problem-statement specificity) and re-evaluated in a generate–evaluate–refine–re-evaluate loop.

Iterative procedure:

$$\pi^{(0)} \xrightarrow{\text{TaskPilot}} \text{task batch} \xrightarrow{\text{RL}} \pi^{(1)} \xrightarrow{\text{TaskPilot}} \cdots \xrightarrow{\text{RL}} \pi^{(5)} = \text{FrogNano}$$

3. RL Training Recipe

DPPO with asymmetric importance sampling:

For each task $i$, sample 8 trajectories and standardize rewards within the group:

$$A_{ij} = \frac{R_{ij} - \bar{R}_i}{\text{std}(R_i) + 10^{-6}}$$

Groups with zero advantage are discarded. For token-level importance correction:

$$p_t = \pi_\theta(a_t \mid s_t), \quad q_t = \pi_{\text{roll}}(a_t \mid s_t), \quad \rho_t = \frac{p_t}{q_t}, \quad \Delta p_t = p_t - q_t$$

A positive-advantage token is dropped when $\Delta p_t > 0.2$; a non-positive-advantage token is dropped when $\Delta p_t < -0.2$. The policy objective:

$$\mathcal{L}_{\text{policy}} = -\frac{1}{|\mathcal{T}|} \sum_{t \in \mathcal{K}} A_t \rho_t$$

No reference-model KL loss, entropy bonus, or auxiliary KL term is used.

Efficiency reward (success-gated log-length penalty):

$$R(N) = \begin{cases} 1, & n \leq N, \\ 1 - \alpha \ln(n/N), & n > N. \end{cases}$$

where $N$ is the allowed free tokens and $\alpha$ controls penalty strength. The penalty applies only to completed successful trajectories. Truncated-but-correct trajectories receive partial reward of 0.5.

Infrastructure: Single node with 8× NVIDIA B200 GPUs; 2 GPUs train (context parallelism 2), 6 GPUs generate rollouts. Each update: 32 task groups × 8 trajectories = 256 trajectories.

---

Empirical Validation / Results

Main Results (Figure 2)

Iterative Gains (Figure 1)

Comparison baselines:

• ~300 real SWE-rebench tasks filtered by same 4B learnability criterion: 48.0%
• 1,500 real tasks (SWE-rebench + Scale-SWE): lower than synthetic-only

Harness Impact

• R2E-Gym → Leaf: 8.3% → 37.2% for Qwen3.5-4B
• MiniMax-M2.5: 66.5% on both harnesses (unaffected)

Analysis Highlights

Pass@k: The gap between FrogNano and base model stays static as $k$ increases, suggesting iterative RL pushes the capability boundary rather than distilling high pass@k into pass@1.

Verifier test-time scaling: Round-robin tournament with 3 candidates achieves 62.8% on SWE-bench Verified vs. 61.4% for single verifier call, 60.4% for pass@short.

Consolidation: Merging behaviors across policies (e.g., reinforcing multi-tool-call behavior) improves SWE-bench Verified to 62.3% with pass@3 of 72.3%.

Compaction: Summary compaction recovers full 131K context performance at 64K token budget; helps most at low token budgets (16K, 32K).

Cross-harness generalization: FrogNano evaluated in mini-SWE-agent (single bash tool) improves from 43.8% (base) to 56.4%, only 5.2% below Leaf performance.

Learning dynamics: Solve rate improves in all five iterations (+11.7, +3.4, +5.6, +3.0, +5.4 points respectively). Policy entropy falls most sharply in Iter 1 (0.415→0.265) then varies within a bounded range.

Failure analysis: 90.8% of failures are reasoning gaps (not premature termination). Dominant causes: wrong root cause/layer (38.8%), misreading specification (31.5%), API misunderstanding (14.1%).

Reward hacking: 21.3% of trajectories flagged by regex analysis; 2.5% confirmed as attempts (Fleiss' κ = 0.746). All attempts blocked by scaffold/infrastructure; 0 hack-related solves.

Task Distribution Evolution (Iteration 5 vs. Iteration 4)

---

Theoretical and Practical Implications

Theoretical Implications

1. Distillation-free capability acquisition: A 4B model can acquire substantial repository-level coding capabilities through RL over policy-grounded synthetic tasks alone, without imitating larger models' trajectories. This challenges the assumption that small models require teacher distillation for complex agentic tasks.

2. Task distribution co-evolves with policy: The closed-loop design (TaskPilot generates tasks → RL trains policy → new policy generates next tasks) demonstrates that adaptive curricula calibrated to the learner's frontier sustain learning across multiple iterations, whereas static datasets saturate.

3. Harness as a first-class training component: The interface between model and environment is not neutral—it can be a major bottleneck for compact models. Leaf's simplicity (5 tools, natural termination) unlocks capability that a more complex harness suppresses.

4. Learnability calibration signal: The empirical, policy-relative signal $\hat{p}_{\text{policy}}$ (resolve rate under current policy) is more useful than intrinsic difficulty estimates. Refinement adjusts semantic clarity and solution-localizing information to move tasks toward the target region.

Practical Implications

1. Accessible coding agents: 4B models can run on minimal hardware, enabling local deployment, repeated experimentation, and research on agent scaffolds, context management, and online data generation.

2. Cost efficiency: FrogNano is competitive with models 6–8× its size on SWE-bench Pro while costing a quarter as much to serve.

3. Synthetic data reduces curation burden: Training on purely synthetic tasks at the learnability edge matches (and exceeds) training on real data subsets filtered by costly rollouts, eliminating the need for expensive real-task mining.

4. Recipe transferability: The recipe (harness + adaptive synthesis + iterative RL) is potentially applicable to other domains with executable validation signals.

---

Conclusion

Main Takeaways

• FrogNano demonstrates that a 4B coding agent can rival 32B–100B+ models on SWE benchmarks without distillation, using only ~1,500 synthetic tasks.
• The three key ingredients are: (1) a lightweight harness (Leaf) matched to the model's capabilities, (2) online task synthesis (TaskPilot) calibrated to the policy's learnability frontier, and (3) iterative RL with DPPO and efficiency regularization.
• The paper provides a practical platform for coding-agent research at small scale, making end-to-end experimentation tractable.

Future Directions

• How much further can the 4B model improve with additional training iterations?
• Can small models handle more specialized and harder coding tasks?
• How to further improve test-time scaling (verifiers, tournaments)?
• How to enhance context management and inference efficiency at small scale?
• Joint training of task generator and solver in a self-play manner (as opposed to TaskPilot's separate generator).

Limitations

• Evaluated on English-language, Python-heavy repositories only
• Generalization to non-English tasks, other frameworks, unclear requirements, or projects without reliable tests is unestablished
• Results may not hold with other tool interfaces, prompts, or budgets
• Image/video capabilities of the base model (Qwen3.5-4B) were not post-trained or evaluated
• Test-based rewards cover only part of program behavior; passing patches can be incorrect or insecure
• Safety considerations: agents should run in isolated environments; all patches need human review