Summary (Overview)

• Introduces "harnessed agentic RL": A paradigm where the deploy-time agent harness (not the training engine) owns the environment interaction loop, with RL training observing only LLM request-response pairs across a service boundary.
• Identifies four key challenges: retokenization/sample merging, advantage calculation, loss normalization, and training backend scheduling under dynamic sample counts—all arising from the disconnect between harness-owned execution and token-based training.
• Presents Agent Lightning v1.0: A lightweight (~3,500 lines of code) framework built on a disaggregated architecture (API Gateway, Rollout Controller, Customized Trainer) that supports arbitrary agent harnesses with minimal integration effort.
• Demonstrates strong empirical results: On coding agents, RL alone improves Qwen3.5-9B on SWE-bench Verified from 41.8% to 56.4% (absolute +14.6%) using only ~6K training examples and modest compute.
• Releases complete reproducible pipeline: Includes data-cleaning pipeline, reward-hacking safeguards, and training scripts for coding agents—addressing a significant gap in existing RL frameworks.

---

Introduction and Theoretical Foundation

Background

Modern agents do not operate as standalone LLMs; they run inside agent harnesses that manage tools, execution environments, context, and control flow. Examples include mini-SWE-agent, OpenHands, OpenCode, Claude Code, and Codex for coding, and OpenClaw and Hermes for general purposes.

Problem with Traditional RL Frameworks

Early RL frameworks (verl, AReaL, slime) require the agent loop to be implemented inside the training framework. Integrating existing harnesses is difficult due to their complex implementations and dependencies.

The Proxy-Based Approach

The original Agent Lightning work introduced a disaggregated architecture connecting arbitrary agents to RL training through an LLM endpoint proxy. This approach has been adopted by verl Uni-Agent, AReaL 2.0, slime v0.3.0, and Polar.

Formal Distinction: Traditional vs. Harnessed Agentic RL

Both paradigms admit a POMDP formulation but differ fundamentally:

Traditional agentic RL: The policy model interacts almost directly with the environment. The token history extends as:

$$p _ { t } = ( p _ { t - 1 } , a _ { t - 1 } , o _ { t } )$$

Harnessed agentic RL: The harness owns context construction and control flow. A rollout is exposed at the model boundary as a sequence of request-response pairs:

$$(p _ {1}, a _ {1}), (p _ {2}, a _ {2}), \ldots$$

The latent state is:

$$s _ {t} = \left(s _ {t} ^ {\mathrm{harness}}, s _ {t} ^ {\mathrm{env}}\right)$$

with the prompt constructed as:

$$p _ {t} ^ {\mathrm{tok}} = \operatorname{Tok} \bigl (\operatorname{Template} (C _ {t} ^ {\mathrm{msg}}) \bigr)$$

Each policy decision is a call-level transition:

$$z _ {t} = (p _ {t} ^ {\mathrm{tok}}, a _ {t} ^ {\mathrm{tok}}), \qquad a _ {t} ^ {\mathrm{tok}} \sim \pi_ {\theta} (\cdot | p _ {t} ^ {\mathrm{tok}})$$

---

Methodology

System Architecture

Agent Lightning v1.0 bridges the training cluster and agent execution cluster through three components:

1. API Gateway: Stores rollouts, models, and events; forwards LLM calls from agent harnesses to registered model endpoints.
2. Rollout Controller: Manages agent execution on Kubernetes (or local processes), polling rollouts and launching agent tasks.
3. Customized Trainer: Built on VERL; registers rollouts, waits for completion, retrieves events to assemble training samples.

Key Design Innovations

1. Collocated Async RL: Unlike synchronous RL (waits for slowest rollout) or asynchronous RL (requires separate GPU pools), collocated async RL time-shares the same GPUs between rollout and update phases. The API Gateway stops accepting new requests during updates and pauses incoming ones—making the switch invisible to the harness. Achieves ~2x end-to-end speedup over sync RL with fewer GPUs.

2. Network Reliability:

• Idempotent API Gateway endpoints for safe retries
• Deduplication of repeated LLM calls (keeping only the most recent when identical prompts appear)

3. Kubernetes Integration: Agents run as standard Kubernetes Jobs on self-hosted compute, avoiding commercial sandbox costs (vs. Modal Sandbox, E2B used by other frameworks).

Addressing the Four Challenges

Challenge 1: Retokenization and Sample Merging

Token-prefix continuity requires:

$$(p _ {i} ^ {\mathrm{tok}}, a _ {i} ^ {\mathrm{tok}}) \preceq p _ {i + 1} ^ {\mathrm{tok}}$$

Even when text-level prefix holds, token-level prefix breaks due to:

• Chat-template non-compositionality: $\operatorname{Template} (A \parallel B) \neq \operatorname{Template} (A) \parallel \operatorname{Template} (B)$
• Decode-retokenize drift: $\operatorname{Tok} \left(\operatorname{Decode} \left(a _ {i} ^ {\mathrm{tok}}\right)\right) \neq a _ {i} ^ {\mathrm{tok}}$
• Inference-time output transformation (tool-call parsing, reserialization)

Agent Lightning's approach: Best-effort sequence merging—merge calls only when token IDs satisfy the prefix condition; otherwise close the current sequence and start a new one. This preserves rollout prompts and works with standard dense causal kernels.

Challenge 2: Advantage Calculation

A rollout can produce a dynamic number of training samples $N_\rho$. The paper argues for rollout-level advantage (rather than sample-level): retokenization and subagent spawning are incidental phenomena that should not change group baselines. Example: two rollouts with rewards 1 and 0; if Rollout 1 splits into 3 samples, rollout-level baseline is $\bar{r}_{rollout} = (1+0)/2 = 1/2$ vs. sample-level $\bar{r}_{sample} = (1+1+1+0)/4 = 3/4$.

Challenge 3: Loss Normalization

Three options compared:

• Token-mean (DAPO): $\mathcal{L}_{token\text{-}mean} = \frac{\sum_{\rho,j,t} \ell_{\rho,j,t}}{\sum_{\rho,j} L_{\rho,j}}$
• Seq-mean-token-mean (GRPO): $\mathcal{L}_{seq\text{-}mean} = \frac{1}{\sum_\rho N_\rho} \sum_{\rho,j} \frac{1}{L_{\rho,j}} \sum_t \ell_{\rho,j,t}$
• Rollout-level token-mean (slime): $\mathcal{L}_{rollout\text{-}mean} = \frac{1}{R} \sum_\rho \frac{\sum_{j,t} \ell_{\rho,j,t}}{\sum_j L_{\rho,j}}$

Agent Lightning's choice: Rollout-level token-mean (Equation 16), as sample-level normalization gives disproportionate weight to rollouts with more samples, and token-mean is sensitive to long sequences.

Challenge 4: Training Backend Scheduling

The training batch must preserve statistical provenance:

$$\mathcal {B} _ {\text { train }} = \bigcup_ {\rho \in \mathcal {B} _ {\text { rollout }}} \left\{\left(S _ {\rho , j}, \rho , g _ {\rho}\right) \mid 1 \leq j \leq N _ {\rho} \right\}$$

Sequences from one rollout must remain in the same optimizer update to avoid within-rollout policy skew.

---

Empirical Validation / Results

Experiment Settings

Coding Agent Details

Dataset Pipeline (from 59,136 SWE-smith tasks):

• Removed 18,033 tasks with empty problem statements
• Removed 1,265 tasks with missing problem branches
• Removed tasks requiring >200 tests
• Applied model-based difficulty filter (Qwen3.5-9B run 4 times per task)
• Final: ~6,000 training examples, 400 test examples

Reward Hacking Prevention:

1. Disabled Git commands and hid .git directory (prevented finding gold commits)
2. Enforced Kubernetes network policy blocking general outbound access (prevented fetching source from GitHub/pip)

Ablation Study on Design Choices

Three settings compared (all using GRPO objective):

The full variant (rollout-level advantage + rollout-level normalization) achieved the highest validation reward with more stable policy entropy.

Rollout Merging Statistics

• Only 36% of rollouts remain as a single training sample on average
• Each rollout yields 2.41 training samples on average

---

Theoretical and Practical Implications

Theoretical Contributions

1. First systematic characterization of harnessed agentic RL challenges, providing formal definitions of retokenization drift, dynamic sample counts, and their consequences.
2. Principled argument for rollout-level statistics: Sample counts are driven by incidental factors (retokenization, subagent spawning), so advantage and loss normalization should be computed at the rollout level to maintain algorithmic correctness.
3. Formal distinction between traditional and harnessed agentic RL as POMDPs with different latent states and observations.

Practical Implications

1. Reproducibility: Complete data pipeline and training scripts for coding agents—filling a major gap where existing frameworks lack data and complete training examples.
2. Resource accessibility: Demonstrates strong results with only ~6K examples and modest compute, making agentic RL feasible for smaller teams.
3. Cost reduction: Self-hosted Kubernetes execution avoids commercial sandbox costs; collocated async RL reduces GPU requirements vs. async RL.
4. Harness-agnostic design: Any agent harness can connect by switching its LLM endpoint to the proxy, requiring almost no changes to the agent.

---

Conclusion

Agent Lightning v1.0 characterizes harnessed agentic RL—a paradigm where the deploy-time harness owns the environment interaction loop—and identifies four key challenges: retokenization/sample merging, advantage calculation, loss normalization, and training backend scheduling. The framework implements rollout-level design choices for these challenges in approximately 3,500 lines of code, supporting arbitrary agent harnesses.

Key results: RL alone improves Qwen3.5-9B on SWE-bench Verified from 41.8% to 56.4% (+14.6%) using ~6K examples; search agents improve +16.6%; instruction-following agents improve +18.3%.

Future directions:

• Better credit assignment across samples within a rollout
• Further study of tree-structured training for prefix reuse
• Extension to more complex multi-agent scenarios

The full codebase and scripts are released to facilitate reproducible harnessed agentic RL research (github.com/microsoft/agent-lightning).