Full text not available for this paper
Summary (Overview)
- CompAct-Bench is a new 200-task benchmark for evaluating working-context compaction — the ability of LLMs to compress accumulated interaction history in long-horizon agent tasks while preserving information needed for task completion.
- The benchmark spans three domains: BrowseComp (79 information-seeking tasks), SWE-bench (96 software-engineering tasks), and CompanyBench (25 workspace tasks), with compaction boundaries at 25%, 50%, and 75% of execution progress.
- Evaluation of 12 frontier LLMs as compactors shows that even the best model (Claude Opus 5) achieves only 68.5% post-compaction accuracy, with the average at 56.7% — despite all tasks being solved in pre-compaction screening runs.
- Key findings reveal domain-specific stage-dependent patterns: Search performance degrades at later boundaries (52.9%→34.6%), while Coding improves (64.0%→86.1%), explained by task-state persistence — whether progress is stored in the environment or only in the working context.
- Failure analysis identifies two distinct pathways: omissions cause repeated work and budget exhaustion, while state distortions lead to premature termination from incorrect task representations.
Introduction and Theoretical Foundation
Background and Motivation
Long-horizon LLM agents accumulate extensive working contexts containing user instructions, intermediate reasoning, tool observations, failed attempts, and partial progress. As these contexts approach model context-window limits, systems must manage this pressure through either:
- Truncation: Directly discarding earlier interactions
- Context compaction: Replacing accumulated history with a shorter representation, often produced by an LLM
Modern agent harnesses (e.g., Claude Code, DeepSeek Harness) increasingly rely on model-based compaction. However, the paper notes:
"Effective compaction is not merely about producing a concise summary of the past, but about retaining the information required for subsequent execution."
Key Research Question
The central question: How much does working-context compaction disrupt downstream task completion in long-horizon agents?
The paper frames working-context compaction as a distinct capability from generic summarization — it is prospective (what matters is what must be carried forward for future execution) rather than retrospective (preserving past content).
Methodology
Benchmark Construction Pipeline
The construction process involves three main stages:
1. Task and Trajectory Collection:
- Sample tasks from each source benchmark
- Run a fixed executor three times per task
- Retain only tasks solved in all three runs (pass@3 filter)
- Remove trajectories with fewer than 8 interaction rounds
- Select one successful rollout as the reference trajectory per task
Table 1: Task Statistics
| Benchmark | # Tasks | Avg. Rounds | Avg. Tokens | Median Tokens |
|---|---|---|---|---|
| BrowseComp | 79 | 30.4 | 35.3K | 23.0K |
| SWE-bench | 96 | 19.4 | 8.8K | 7.7K |
| CompanyBench | 25 | 58.5 | 122.8K | 115.0K |
2. Compaction Instance Construction:
- Boundaries placed at 25%, 50%, and 75% of execution progress
- Trajectories stratified by length before boundary assignment to reduce sampling bias
- Each trajectory assigned to one boundary to control evaluation cost
3. Evaluation Protocol:
The reference trajectory is formalized as:
where is an executor action, is the corresponding observation, and is the terminal action.
The accumulated working context before boundary is:
The compactor maps this history to a compacted representation:
with actual retention ratio:
Iterative Ratio Calibration (IRC) progressively contracts until output ratio falls within tolerance of the target (10% character-level retention by default).
Matched Continuation Budget: Each continuation receives at most additional actions, preventing compensation for information loss through extra interaction.
Evaluation Metric:
where if instance succeeds after compaction.
Models Evaluated
Twelve frontier LLMs tested as compactors: Claude Opus 5, GPT-5.6-Sol, Gemini 3.8 Flash, Gemini 3.1 Pro Preview, Kimi K3, DeepSeek V4 Pro, HY3, GLM 5.3 Flash, GLM 5.2, Qwen 3.8 Flash, Qwen 3.7 Plus, and Doubao Seed 2.1 Pro.
Empirical Validation / Results
RQ1: Capability and Generalizability of Current LLMs
Table 2: Post-Compaction Task Accuracy (%)
| Model | Search | Coding | Workspace | Overall |
|---|---|---|---|---|
| Claude Opus 5 (Max) | 62.0 | 81.2 | 40.0 | 68.5 |
| Gemini 3.8 Flash (High) | 57.0 | 75.0 | 56.0 | 65.5 |
| Kimi K3 (Max) | 50.6 | 79.2 | 32.0 | 62.0 |
| GPT-5.6-Sol (Max) | 51.9 | 76.0 | 20.0 | 59.5 |
| DeepSeek V4 Pro (Max) | 44.3 | 72.4 | 44.0 | 57.8 |
| HY3 (High) | 44.3 | 72.9 | 40.0 | 57.5 |
| GLM 5.3 Flash (Max) | 39.2 | 76.0 | 40.0 | 57.0 |
| Qwen 3.8 Flash (Max) | 34.2 | 77.1 | 32.0 | 54.5 |
| GLM 5.2 (Max) | 32.9 | 75.0 | 24.0 | 52.0 |
| Gemini 3.1 Pro Preview (High) | 39.2 | 70.8 | 16.0 | 51.5 |
| Qwen 3.7 Plus (64K*) | 31.6 | 66.7 | 32.0 | 48.5 |
| Doubao Seed 2.1 Pro (High) | 31.6 | 67.7 | 8.0 | 46.0 |
Best result bolded, second-best underlined.
Key findings:
- Overall performance ranges from 46.0% to 68.5% (average 56.7%)
- Strong performance partially transfers across domains, suggesting a shared ability
- Claude Opus 5 leads on Search and Coding; Gemini 3.8 Flash leads on Workspace
RQ2: Performance Across Execution Stages
Stage-dependent patterns (model average):
- Search: Decreases monotonically — 52.9% → 42.6% → 34.6% at 25%, 50%, 75% boundaries
- Coding: Increases monotonically — 64.0% → 72.7% → 86.1%
- Workspace: Non-monotonic, peaking at 50% boundary
The amount of accumulated history alone is insufficient to explain these patterns, as input length grows consistently across all domains (e.g., Search: 8.2K → 29.3K tokens; Coding: 2.3K → 5.5K tokens).
RQ3: Task-State Persistence Analysis
Figure 4 findings:
- (a) Without or , continuing from reconstructed boundary state achieves 66.7%, 58.1%, 71.9% at 25%, 50%, 75% boundaries (vs. 48.5%, 35.5%, 15.6% from initial state) — gains of +18.2, +22.6, +56.3 points
- (b) The utility of differs by domain:
- Search: provides consistently large gains (context-resident evidence)
- Coding: At 25% boundary, provides no gain (66.7%→64.0%), but at 75% it improves from 71.9%→86.1%
RQ4: Failure Mode Analysis
Six error categories annotated: critical omission (CO), factual distortion (FD), premature commitment (PC), lost resolution (LR), progress/state loss (PSL), constraint/goal distortion (CGD).
Key findings from Figure 5:
- Early failures are more associated with PC (56.7% vs. 46.0%) and PSL (55.2% vs. 44.6%)
- Budget exhaustion is less concentrated around any single error category
- Critical omission → +18.2 points in repeated work
- Progress/state loss → +13.8 points in repeated work
- Constraint/goal distortion → +10.5 points in unverified commitment
- Factual distortion → +7.3 points unverified commitment, +7.4 points repeated work
Theoretical and Practical Implications
Theoretical Contributions
-
Working-context compaction as a distinct capability: The paper demonstrates that compaction is fundamentally different from generic summarization or document compression — it is prospective and execution-critical.
-
Task-state persistence framework: Introduces a conceptual distinction between:
- Context-resident state: Information existing only in the interaction history (e.g., Search evidence)
- Externally persisted state: Progress stored in the environment (e.g., Coding repository changes)
This distribution determines how much information must carry and explains domain-specific compaction difficulty.
-
Two failure pathways:
- Omission pathway: Lost information → repeated work → budget exhaustion
- Distortion pathway: Incorrect representation → premature/misdirected actions → early failure
Practical Implications
- Agent design: Systems should externalize task state to persistent environments where possible, reducing the information burden on compaction
- Compaction strategy: Should be stage-aware — later-stage compaction in state-persistent domains (Coding) is more robust
- Model development: Compaction quality should be treated as a first-class evaluation metric for long-horizon agent reliability
- Benchmark extensibility: The pipeline generalizes to new domains without manual reference summaries
Conclusion
CompAct-Bench provides a systematic foundation for evaluating working-context compaction through downstream task execution. Key takeaways:
- Current models remain substantially lossy under aggressive (10%) compaction, with even the best model failing on ~31.5% of tasks
- Compaction difficulty depends on more than context length — it depends critically on where task state is retained and how faithfully it is carried forward
- Different failure modes require different mitigation strategies: omissions suggest better information selection; distortions require more faithful state representation
Future directions implied by the work:
- Developing compactors specifically optimized for prospective information retention
- Adaptive compaction strategies that account for task-state persistence
- Training models on compaction-specific objectives rather than generic summarization
- Extending the benchmark to additional domains and execution settings
Related papers
- Late Entry in Looped Language Models
Late entry into deep context caches outperforms early exit in looped language models, yielding up to 8.4x exact and 12.5x lossy decoding speedups.
- RIPPLE: Recovery Internalization through Process-Level Policy Learning and Effect-Aware Credit for Long-Horizon Agentic Coding
RIPPLE internalizes prefix-only recovery guidance into a single policy via stage-specific credit, improving coding agents by 4.8 points across four benchmarks without deployment-time critics.
- Inside the Loop: A Mechanistic Study of Looped Transformers on Permutation Traversal
Looped transformers solve sequential algorithmic tasks untied models of equal effective depth cannot, by executing one algorithm step per iteration without intermediate supervision.