# LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents > LEGO-RL enables native coding-agent harnesses to train with policy-gradient RL, boosting SWE-bench Verified scores by up to 9.4 points without altering their control flow. - **Source:** [arXiv](https://arxiv.org/abs/2608.17393) - **Published:** 2026-08-22 - **Permalink:** https://picx.dev/p/jBNH9m - **Whiteboard:** https://picx.dev/p/jBNH9m/image ## Summary # LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents ## Summary (Overview) - **LEGO-RL** is a framework that bridges native coding-agent harnesses (OpenHands SDK, Claude Code, OpenCode) with scalable policy-gradient optimization **without modifying their internal control flow**, preserving original agent workflows while enabling RL training. - The framework is built on **three pillars**: (1) **faithful optimization** via in-process LLM proxying that captures raw generation streams for token-level alignment and robust trainer-side log-probability recomputation (including expert-routing replay for MoE models); (2) **reliable execution** via scalable sandbox orchestration with image caching and stage-wise defenses against reward hacking; (3) **observable training** through an integrated plugin with automated validation, monitoring, and a Live UI for granular trajectory diagnostics. - Training the sparse MoE model **Qwen3.5-35B-A3B** with GSPO yields significant gains on SWE-bench Verified across all three harnesses: **OpenHands SDK 64.0% → 70.4%**, **Claude Code 62.4% → 68.2%**, **OpenCode 57.2% → 66.6%**, while maintaining **rollout-training probability correlation above 0.99**. - The framework addresses critical challenges in agentic RL: harness-side history rewriting that corrupts trajectory reconstruction, reward hacking, sandbox failures, and the heavy right-tail distribution of agentic rollouts that reduces inference utilization under synchronized generation. - LEGO-RL is open-sourced with complete framework, trained models, and datasets available at https://github.com/LegoX/Lego-RL and https://huggingface.co/LegoX/Lego-RL. ## Introduction and Theoretical Foundation ### Background and Motivation Training coding agents with reinforcement learning requires optimizing long-horizon behavior over software repositories, tools, and execution environments. Unlike learning from isolated responses, RL updates the policy from **complete trajectories and their verifier rewards**. A single rollout may involve repeated model calls, repository inspection, tool calling, code modification, dependency installation, and test execution before producing a sparse executable reward. The core challenge: these trajectories are generated by **native agent harnesses** that manage prompts, tools, context, and execution state. The optimization target is the policy behavior induced by the original agent control flow, but integrating these harnesses into policy-gradient training is difficult because: - **Harness-side prompt construction, context compaction, and history rewriting** can cause the reconstructed trajectory to differ from the exact token sequence sampled during rollout, preventing faithful trainer-side probability recomputation. - **Sparse mixture-of-experts models** introduce an additional mismatch when rollout-time expert routing is not reproduced during training. - **Sandbox failures, dependency errors, verifier misconfiguration, timeouts, and reward hacking** can discard costly trajectories or corrupt reward signals, yet are difficult to localize in asynchronous pipelines. ### Problem Setup and Objective A task instance $x = (q_x, R_x, V_x)$ pairs a problem statement $q_x$ and an initialized repository environment $R_x$ with a task-specific executable verifier $V_x$. The native coding-agent harness $\mathcal{H}$ is treated as part of the environment, and only the policy $\pi_\theta$ it calls is optimized. At turn $t = 1, \ldots, T$, the harness maps the current interaction and repository state $s_t$ to a context $c_t = \mathcal{H}(s_t)$, the policy generates an assistant token span $a_t \sim \pi_\theta(\cdot \mid c_t)$, and the harness executes the requested tool actions. The trajectory log-likelihood is: $$\log \pi_\theta(\tau) = \sum_{(t,j) \in \mathcal{M}(\tau)} \log \pi_\theta(a_{t,j} \mid c_t, a_{t,