# CompAct-Bench: A Benchmark for Working-Context Compaction in Long-Horizon Agent Tasks

> Working-context compaction drops even the best LLM's task success to 68.5% on a new 200-task benchmark, with failures driven by whether task state persists in the environment or only in context.

- **Source:** [arXiv](https://arxiv.org/abs/2611.25673)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/2XFGzi
- **Whiteboard:** https://picx.dev/p/2XFGzi/image

## Summary

## Summary (Overview)

- **CompAct-Bench** is a new 200-task benchmark for evaluating **working-context compaction** — the ability of LLMs to compress accumulated interaction history in long-horizon agent tasks while preserving information needed for task completion.
- The benchmark spans three domains: **BrowseComp** (79 information-seeking tasks), **SWE-bench** (96 software-engineering tasks), and **CompanyBench** (25 workspace tasks), with compaction boundaries at 25%, 50%, and 75% of execution progress.
- Evaluation of **12 frontier LLMs** as compactors shows that even the best model (Claude Opus 5) achieves only **68.5%** post-compaction accuracy, with the average at **56.7%** — despite all tasks being solved in pre-compaction screening runs.
- Key findings reveal **domain-specific stage-dependent patterns**: Search performance degrades at later boundaries (52.9%→34.6%), while Coding improves (64.0%→86.1%), explained by **task-state persistence** — whether progress is stored in the environment or only in the working context.
- Failure analysis identifies **two distinct pathways**: omissions cause repeated work and budget exhaustion, while state distortions lead to premature termination from incorrect task representations.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Long-horizon LLM agents accumulate extensive working contexts containing user instructions, intermediate reasoning, tool observations, failed attempts, and partial progress. As these contexts approach model context-window limits, systems must manage this pressure through either:

- **Truncation**: Directly discarding earlier interactions
- **Context compaction**: Replacing accumulated history with a shorter representation, often produced by an LLM

Modern agent harnesses (e.g., Claude Code, DeepSeek Harness) increasingly rely on model-based compaction. However, the paper notes:

> "Effective compaction is not merely about producing a concise summary of the past, but about retaining the information required for subsequent execution."

### Key Research Question

The central question: **How much does working-context compaction disrupt downstream task completion in long-horizon agents?**

The paper frames working-context compaction as a distinct capability from generic summarization — it is **prospective** (what matters is what must be carried forward for future execution) rather than retrospective (preserving past content).

---

## Methodology

### Benchmark Construction Pipeline

The construction process involves three main stages:

**1. Task and Trajectory Collection:**
- Sample tasks from each source benchmark
- Run a fixed executor **three times** per task
- Retain only tasks solved in all three runs (pass@3 filter)
- Remove trajectories with fewer than 8 interaction rounds
- Select one successful rollout as the reference trajectory per task

**Table 1: Task Statistics**

| Benchmark | # Tasks | Avg. Rounds | Avg. Tokens | Median Tokens |
|-----------|---------|-------------|-------------|---------------|
| BrowseComp | 79 | 30.4 | 35.3K | 23.0K |
| SWE-bench | 96 | 19.4 | 8.8K | 7.7K |
| CompanyBench | 25 | 58.5 | 122.8K | 115.0K |

**2. Compaction Instance Construction:**
- Boundaries placed at 25%, 50%, and 75% of execution progress
- Trajectories stratified by length before boundary assignment to reduce sampling bias
- Each trajectory assigned to one boundary to control evaluation cost

**3. Evaluation Protocol:**

The reference trajectory is formalized as:

$$\tau = (a_1, o_1, \ldots, a_{T-1}, o_{T-1}, a_T) \tag{1}$$

where $a_i$ is an executor action, $o_i$ is the corresponding observation, and $a_T$ is the terminal action.

The accumulated working context before boundary $b$ is:

$$H_b = (a_1, o_1, \ldots, a_b, o_b) \tag{2}$$

The compactor $C_\phi$ maps this history to a compacted representation:

$$M_b = C_\phi(H_b; r) \tag{3}$$

with actual retention ratio:

$$r_{\text{actual}} = \frac{\ell(M_b)}{\ell(H_b)} \tag{4}$$

**Iterative Ratio Calibration (IRC)** progressively contracts $H_b$ until output ratio falls within tolerance of the target (10% character-level retention by default).

**Matched Continuation Budget:** Each continuation receives at most $T - b$ additional actions, preventing compensation for information loss through extra interaction.

**Evaluation Metric:**

$$\text{Acc}(C_\phi) = \frac{1}{|\mathcal{D}|} \sum_{i \in \mathcal{D}} y_i \tag{5}$$

where $y_i = 1$ if instance $i$ succeeds after compaction.

### Models Evaluated

Twelve frontier LLMs tested as compactors: Claude Opus 5, GPT-5.6-Sol, Gemini 3.8 Flash, Gemini 3.1 Pro Preview, Kimi K3, DeepSeek V4 Pro, HY3, GLM 5.3 Flash, GLM 5.2, Qwen 3.8 Flash, Qwen 3.7 Plus, and Doubao Seed 2.1 Pro.

---

## Empirical Validation / Results

### RQ1: Capability and Generalizability of Current LLMs

**Table 2: Post-Compaction Task Accuracy (%)**

| Model | Search | Coding | Workspace | Overall |
|-------|--------|--------|-----------|---------|
| **Claude Opus 5 (Max)** | **62.0** | **81.2** | 40.0 | **68.5** |
| Gemini 3.8 Flash (High) | 57.0 | 75.0 | **56.0** | 65.5 |
| Kimi K3 (Max) | 50.6 | 79.2 | 32.0 | 62.0 |
| GPT-5.6-Sol (Max) | 51.9 | 76.0 | 20.0 | 59.5 |
| DeepSeek V4 Pro (Max) | 44.3 | 72.4 | 44.0 | 57.8 |
| HY3 (High) | 44.3 | 72.9 | 40.0 | 57.5 |
| GLM 5.3 Flash (Max) | 39.2 | 76.0 | 40.0 | 57.0 |
| Qwen 3.8 Flash (Max) | 34.2 | 77.1 | 32.0 | 54.5 |
| GLM 5.2 (Max) | 32.9 | 75.0 | 24.0 | 52.0 |
| Gemini 3.1 Pro Preview (High) | 39.2 | 70.8 | 16.0 | 51.5 |
| Qwen 3.7 Plus (64K*) | 31.6 | 66.7 | 32.0 | 48.5 |
| Doubao Seed 2.1 Pro (High) | 31.6 | 67.7 | 8.0 | 46.0 |

*Best result bolded, second-best underlined.*

Key findings:
- Overall performance ranges from **46.0% to 68.5%** (average 56.7%)
- Strong performance partially transfers across domains, suggesting a shared ability
- Claude Opus 5 leads on Search and Coding; Gemini 3.8 Flash leads on Workspace

### RQ2: Performance Across Execution Stages

**Stage-dependent patterns (model average):**
- **Search**: Decreases monotonically — 52.9% → 42.6% → 34.6% at 25%, 50%, 75% boundaries
- **Coding**: Increases monotonically — 64.0% → 72.7% → 86.1%
- **Workspace**: Non-monotonic, peaking at 50% boundary

The amount of accumulated history alone is insufficient to explain these patterns, as input length grows consistently across all domains (e.g., Search: 8.2K → 29.3K tokens; Coding: 2.3K → 5.5K tokens).

### RQ3: Task-State Persistence Analysis

**Figure 4 findings:**
- (a) Without $H_b$ or $M_b$, continuing from reconstructed boundary state achieves **66.7%, 58.1%, 71.9%** at 25%, 50%, 75% boundaries (vs. 48.5%, 35.5%, 15.6% from initial state) — gains of +18.2, +22.6, +56.3 points
- (b) The utility of $M_b$ differs by domain:
  - **Search**: $M_b$ provides consistently large gains (context-resident evidence)
  - **Coding**: At 25% boundary, $M_b$ provides no gain (66.7%→64.0%), but at 75% it improves from 71.9%→86.1%

### RQ4: Failure Mode Analysis

Six error categories annotated: **critical omission (CO), factual distortion (FD), premature commitment (PC), lost resolution (LR), progress/state loss (PSL), constraint/goal distortion (CGD)**.

Key findings from Figure 5:
- **Early failures** are more associated with PC (56.7% vs. 46.0%) and PSL (55.2% vs. 44.6%)
- **Budget exhaustion** is less concentrated around any single error category
- **Critical omission** → +18.2 points in repeated work
- **Progress/state loss** → +13.8 points in repeated work
- **Constraint/goal distortion** → +10.5 points in unverified commitment
- **Factual distortion** → +7.3 points unverified commitment, +7.4 points repeated work

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Working-context compaction as a distinct capability**: The paper demonstrates that compaction is fundamentally different from generic summarization or document compression — it is prospective and execution-critical.

2. **Task-state persistence framework**: Introduces a conceptual distinction between:
   - **Context-resident state**: Information existing only in the interaction history (e.g., Search evidence)
   - **Externally persisted state**: Progress stored in the environment (e.g., Coding repository changes)
   
   This distribution determines how much information $M_b$ must carry and explains domain-specific compaction difficulty.

3. **Two failure pathways**:
   - **Omission pathway**: Lost information → repeated work → budget exhaustion
   - **Distortion pathway**: Incorrect representation → premature/misdirected actions → early failure

### Practical Implications

- **Agent design**: Systems should externalize task state to persistent environments where possible, reducing the information burden on compaction
- **Compaction strategy**: Should be stage-aware — later-stage compaction in state-persistent domains (Coding) is more robust
- **Model development**: Compaction quality should be treated as a first-class evaluation metric for long-horizon agent reliability
- **Benchmark extensibility**: The pipeline generalizes to new domains without manual reference summaries

---

## Conclusion

CompAct-Bench provides a systematic foundation for evaluating working-context compaction through downstream task execution. Key takeaways:

1. **Current models remain substantially lossy** under aggressive (10%) compaction, with even the best model failing on ~31.5% of tasks
2. **Compaction difficulty depends on more than context length** — it depends critically on where task state is retained and how faithfully it is carried forward
3. **Different failure modes require different mitigation strategies**: omissions suggest better information selection; distortions require more faithful state representation

**Future directions** implied by the work:
- Developing compactors specifically optimized for prospective information retention
- Adaptive compaction strategies that account for task-state persistence
- Training models on compaction-specific objectives rather than generic summarization
- Extending the benchmark to additional domains and execution settings

---

_Markdown view of https://picx.dev/p/2XFGzi, served by PicX — AI-generated visual whiteboard summaries of research papers._
