# Agent Lightning v1.0: Towards Harnessed Agentic RL

> Agent Lightning enables harnessed agentic RL with rollout-level advantage and normalization, boosting coding agents on SWE-bench Verified from 41.8% to 56.4% using only ~6K examples.

- **Source:** [arXiv](https://arxiv.org/abs/2608.17528)
- **Published:** 2026-08-22
- **Permalink:** https://picx.dev/p/qyHb7W
- **Whiteboard:** https://picx.dev/p/qyHb7W/image

## Summary

## Summary (Overview)

- **Introduces "harnessed agentic RL"**: A paradigm where the deploy-time agent harness (not the training engine) owns the environment interaction loop, with RL training observing only LLM request-response pairs across a service boundary.
- **Identifies four key challenges**: retokenization/sample merging, advantage calculation, loss normalization, and training backend scheduling under dynamic sample counts—all arising from the disconnect between harness-owned execution and token-based training.
- **Presents Agent Lightning v1.0**: A lightweight (~3,500 lines of code) framework built on a disaggregated architecture (API Gateway, Rollout Controller, Customized Trainer) that supports arbitrary agent harnesses with minimal integration effort.
- **Demonstrates strong empirical results**: On coding agents, RL alone improves Qwen3.5-9B on SWE-bench Verified from **41.8% to 56.4%** (absolute +14.6%) using only ~6K training examples and modest compute.
- **Releases complete reproducible pipeline**: Includes data-cleaning pipeline, reward-hacking safeguards, and training scripts for coding agents—addressing a significant gap in existing RL frameworks.

---

## Introduction and Theoretical Foundation

### Background
Modern agents do not operate as standalone LLMs; they run inside **agent harnesses** that manage tools, execution environments, context, and control flow. Examples include mini-SWE-agent, OpenHands, OpenCode, Claude Code, and Codex for coding, and OpenClaw and Hermes for general purposes.

### Problem with Traditional RL Frameworks
Early RL frameworks (verl, AReaL, slime) require the agent loop to be implemented **inside** the training framework. Integrating existing harnesses is difficult due to their complex implementations and dependencies.

### The Proxy-Based Approach
The original Agent Lightning work introduced a **disaggregated architecture** connecting arbitrary agents to RL training through an LLM endpoint proxy. This approach has been adopted by verl Uni-Agent, AReaL 2.0, slime v0.3.0, and Polar.

### Formal Distinction: Traditional vs. Harnessed Agentic RL

Both paradigms admit a POMDP formulation but differ fundamentally:

| | Agentic RL | Harnessed Agentic RL |
|---|---|---|
| **State** | Environment | Harness + environment |
| **Model input** | Continuous token history | Per-call prompts |
| **Agents** | Single ReAct agent | Multi-agent, subagents, handoffs |

**Traditional agentic RL**: The policy model interacts almost directly with the environment. The token history extends as:

$$p _ { t } = ( p _ { t - 1 } , a _ { t - 1 } , o _ { t } )$$

**Harnessed agentic RL**: The harness owns context construction and control flow. A rollout is exposed at the model boundary as a sequence of request-response pairs:

$$(p _ {1}, a _ {1}), (p _ {2}, a _ {2}), \ldots$$

The latent state is:

$$s _ {t} = \left(s _ {t} ^ {\mathrm{harness}}, s _ {t} ^ {\mathrm{env}}\right)$$

with the prompt constructed as:

$$p _ {t} ^ {\mathrm{tok}} = \operatorname{Tok} \bigl (\operatorname{Template} (C _ {t} ^ {\mathrm{msg}}) \bigr)$$

Each policy decision is a call-level transition:

$$z _ {t} = (p _ {t} ^ {\mathrm{tok}}, a _ {t} ^ {\mathrm{tok}}), \qquad a _ {t} ^ {\mathrm{tok}} \sim \pi_ {\theta} (\cdot | p _ {t} ^ {\mathrm{tok}})$$

---

## Methodology

### System Architecture

Agent Lightning v1.0 bridges the training cluster and agent execution cluster through three components:

1. **API Gateway**: Stores rollouts, models, and events; forwards LLM calls from agent harnesses to registered model endpoints.
2. **Rollout Controller**: Manages agent execution on Kubernetes (or local processes), polling rollouts and launching agent tasks.
3. **Customized Trainer**: Built on VERL; registers rollouts, waits for completion, retrieves events to assemble training samples.

### Key Design Innovations

**1. Collocated Async RL**: Unlike synchronous RL (waits for slowest rollout) or asynchronous RL (requires separate GPU pools), collocated async RL **time-shares the same GPUs** between rollout and update phases. The API Gateway stops accepting new requests during updates and pauses incoming ones—making the switch invisible to the harness. Achieves ~2x end-to-end speedup over sync RL with fewer GPUs.

**2. Network Reliability**: 
- Idempotent API Gateway endpoints for safe retries
- Deduplication of repeated LLM calls (keeping only the most recent when identical prompts appear)

**3. Kubernetes Integration**: Agents run as standard Kubernetes Jobs on self-hosted compute, avoiding commercial sandbox costs (vs. Modal Sandbox, E2B used by other frameworks).

### Addressing the Four Challenges

**Challenge 1: Retokenization and Sample Merging**

Token-prefix continuity requires:

$$(p _ {i} ^ {\mathrm{tok}}, a _ {i} ^ {\mathrm{tok}}) \preceq p _ {i + 1} ^ {\mathrm{tok}}$$

Even when text-level prefix holds, token-level prefix breaks due to:
- **Chat-template non-compositionality**: $\operatorname{Template} (A \parallel B) \neq \operatorname{Template} (A) \parallel \operatorname{Template} (B)$
- **Decode-retokenize drift**: $\operatorname{Tok} \left(\operatorname{Decode} \left(a _ {i} ^ {\mathrm{tok}}\right)\right) \neq a _ {i} ^ {\mathrm{tok}}$
- **Inference-time output transformation** (tool-call parsing, reserialization)

**Agent Lightning's approach**: **Best-effort sequence merging**—merge calls only when token IDs satisfy the prefix condition; otherwise close the current sequence and start a new one. This preserves rollout prompts and works with standard dense causal kernels.

**Challenge 2: Advantage Calculation**

A rollout can produce a dynamic number of training samples $N_\rho$. The paper argues for **rollout-level advantage** (rather than sample-level): retokenization and subagent spawning are incidental phenomena that should not change group baselines. Example: two rollouts with rewards 1 and 0; if Rollout 1 splits into 3 samples, rollout-level baseline is $\bar{r}_{rollout} = (1+0)/2 = 1/2$ vs. sample-level $\bar{r}_{sample} = (1+1+1+0)/4 = 3/4$.

**Challenge 3: Loss Normalization**

Three options compared:
- **Token-mean** (DAPO): $\mathcal{L}_{token\text{-}mean} = \frac{\sum_{\rho,j,t} \ell_{\rho,j,t}}{\sum_{\rho,j} L_{\rho,j}}$
- **Seq-mean-token-mean** (GRPO): $\mathcal{L}_{seq\text{-}mean} = \frac{1}{\sum_\rho N_\rho} \sum_{\rho,j} \frac{1}{L_{\rho,j}} \sum_t \ell_{\rho,j,t}$
- **Rollout-level token-mean** (slime): $\mathcal{L}_{rollout\text{-}mean} = \frac{1}{R} \sum_\rho \frac{\sum_{j,t} \ell_{\rho,j,t}}{\sum_j L_{\rho,j}}$

**Agent Lightning's choice**: Rollout-level token-mean (Equation 16), as sample-level normalization gives disproportionate weight to rollouts with more samples, and token-mean is sensitive to long sequences.

**Challenge 4: Training Backend Scheduling**

The training batch must preserve statistical provenance:

$$\mathcal {B} _ {\text { train }} = \bigcup_ {\rho \in \mathcal {B} _ {\text { rollout }}} \left\{\left(S _ {\rho , j}, \rho , g _ {\rho}\right) \mid 1 \leq j \leq N _ {\rho} \right\}$$

Sequences from one rollout must remain in the same optimizer update to avoid within-rollout policy skew.

---

## Empirical Validation / Results

### Experiment Settings

| Setting | Model | Algorithm | Dataset | Result |
|---|---|---|---|---|
| **Search Agent** | Llama-3.2-3B-Instruct | GRPO | HotpotQA | Validation reward: 25.1% → **41.7%** (+16.6%) |
| **Instruction-Following** | Qwen3-4B-Instruct-2507 | RLOO | Instruction Pre-Training | Validation reward: 51.9% → **70.2%** (+18.3%) |
| **Coding Agent** | Qwen3.5-9B | GRPO | SWE-smith | SWE-bench Verified: 41.8% → **56.4%** (+14.6%) |

### Coding Agent Details

**Dataset Pipeline** (from 59,136 SWE-smith tasks):
- Removed 18,033 tasks with empty problem statements
- Removed 1,265 tasks with missing problem branches
- Removed tasks requiring >200 tests
- Applied model-based difficulty filter (Qwen3.5-9B run 4 times per task)
- Final: ~6,000 training examples, 400 test examples

**Reward Hacking Prevention**:
1. Disabled Git commands and hid `.git` directory (prevented finding gold commits)
2. Enforced Kubernetes network policy blocking general outbound access (prevented fetching source from GitHub/pip)

### Ablation Study on Design Choices

Three settings compared (all using GRPO objective):

| Setting | Validation Reward (step 128) |
|---|---|
| Sample-level Advantage + Token-mean Loss | 35.0% |
| Rollout-level Advantage + Token-mean Loss | 33.1% |
| **Rollout-level Advantage + Rollout-level Norm** | **38.2%** |

The full variant (rollout-level advantage + rollout-level normalization) achieved the highest validation reward with more stable policy entropy.

### Rollout Merging Statistics
- Only **36%** of rollouts remain as a single training sample on average
- Each rollout yields **2.41 training samples** on average

---

## Theoretical and Practical Implications

### Theoretical Contributions
1. **First systematic characterization** of harnessed agentic RL challenges, providing formal definitions of retokenization drift, dynamic sample counts, and their consequences.
2. **Principled argument for rollout-level statistics**: Sample counts are driven by incidental factors (retokenization, subagent spawning), so advantage and loss normalization should be computed at the rollout level to maintain algorithmic correctness.
3. **Formal distinction between traditional and harnessed agentic RL** as POMDPs with different latent states and observations.

### Practical Implications
1. **Reproducibility**: Complete data pipeline and training scripts for coding agents—filling a major gap where existing frameworks lack data and complete training examples.
2. **Resource accessibility**: Demonstrates strong results with only ~6K examples and modest compute, making agentic RL feasible for smaller teams.
3. **Cost reduction**: Self-hosted Kubernetes execution avoids commercial sandbox costs; collocated async RL reduces GPU requirements vs. async RL.
4. **Harness-agnostic design**: Any agent harness can connect by switching its LLM endpoint to the proxy, requiring almost no changes to the agent.

---

## Conclusion

Agent Lightning v1.0 characterizes **harnessed agentic RL**—a paradigm where the deploy-time harness owns the environment interaction loop—and identifies four key challenges: retokenization/sample merging, advantage calculation, loss normalization, and training backend scheduling. The framework implements rollout-level design choices for these challenges in approximately 3,500 lines of code, supporting arbitrary agent harnesses.

**Key results**: RL alone improves Qwen3.5-9B on SWE-bench Verified from 41.8% to 56.4% (+14.6%) using ~6K examples; search agents improve +16.6%; instruction-following agents improve +18.3%.

**Future directions**:
- Better credit assignment across samples within a rollout
- Further study of tree-structured training for prefix reuse
- Extension to more complex multi-agent scenarios

The full codebase and scripts are released to facilitate reproducible harnessed agentic RL research (github.com/microsoft/agent-lightning).

---

_Markdown view of https://picx.dev/p/qyHb7W, served by PicX — AI-generated visual whiteboard summaries of research papers._