Summary (Overview)

• Compact-Bench is a novel benchmark for evaluating whether context compaction preserves a coding agent's ability to resume long-horizon work, combining three complementary evaluation tracks: Task Preservation, Recurring-Detail Retention, and Production-Chain Retention.
• Key finding: Across 14 summarizer backbones, stable task damage ranges from 5–64% of 22 full-history successes, with Claude Opus 4.8 preserving the most repairs and DeepSeek-Flash damaging the most.
• Critical insight: Retaining repair evidence (target files, code tokens) does not guarantee successful continuation—summaries can preserve all ingredients while losing the progress needed to complete a task.
• Prompt choice matters: With Kimi-K3 as summarizer, Kimi Code preserves the most task continuations (19/22), while Claude Code retains the most recurring details (15.0%), demonstrating that task preservation and detail retention are distinct properties.
• LLM summaries retain at most 16.0% of recurring details compared to 79.1% for mechanical extraction, revealing a large gap between generated summarization and direct extraction.

Introduction and Theoretical Foundation

Long-horizon coding agents accumulate instructions, tool observations, test results, and partial decisions. Context compaction allows them to work beyond their context window, but the resulting summary must carry enough state to resume unfinished work. The paper identifies a gap in existing evaluations:

• Long-context benchmarks test retrieval, reasoning, and generation from extended inputs but do not address agent histories' specific requirement: preserving which actions have been taken, which hypotheses have been rejected, and what remains to be done.
• End-to-end benchmarks evaluate repository repair and code generation but a final score does not isolate the effect of replacing a history midway through a task.
• Recent compaction evaluations examine constraint loss, archived-fact retrieval, and future-trajectory support, but none directly compare full-history vs. summary-only continuation at a shared boundary.

The central research question is: For the same task and pre-compaction state, how does continuation change when an agent receives only a summary instead of its full history?

The benchmark introduces three evaluation dimensions:

1. Task Preservation — paired full-history vs. summary-only continuations from the same repository snapshot
2. Recurring-Detail Retention — preservation of structured details that recur in later requests
3. Production-Chain Retention — tracking details across successive compactions in real sessions

Methodology

Evaluation Framework

Let $H$ denote the serialized agent history before a compaction boundary. The compaction method $C$ (summarizer + prompt) produces summary $S = C(H)$.

• PRE condition: continuation with full history $H$
• POST condition: continuation with only summary $S$
• Both start from the same repository working-tree snapshot $D$

Task Preservation Scoring

For each repair, outcomes are classified into four categories:

• Preserved: both PRE and POST pass
• Damaged: PRE passes, POST fails
• Rescue: PRE fails, POST passes
• Unsolved: both fail

The damage rate is the fraction of PRE-pass cases that fail after compaction. Stable damage requires POST failure in at least two of three runs.

Recurring-Detail Retention Scoring

For session $i$, let $H_i$ be the pre-boundary history, $F_i$ the next 3–8 request payloads, and $S_i$ the tested summary. The reference set is:

$$A_i = A(H_i) \cap A(F_i)$$

where $A(T)$ is a deterministic extractor returning distinct (type, value) pairs. The per-session score is:

$$\text{retention}_i = \frac{\text{matching pairs in } A_i}{|A_i|}$$

Production-Chain Retention Scoring

For session $i$ and its $k$-th compaction boundary:

$$R_{i,k} = \frac{\text{matching detail pairs in } A_{i,k}}{|A_{i,k}|}$$

where $A_{i,k} = A(F_{i,k})$ and $F_{i,k}$ is the next at most 5 request payloads.

Dataset Composition

Empirical Validation / Results

Task Preservation — Backbone Comparison

Table 2: Task Preservation damage under Claude Code prompt and Kimi-K3 continuation agent

• Claude Opus 4.8 preserves 21 of 22 validated repairs; DeepSeek-Flash preserves only 8
• 120 of 126 first-round damaged pairs confirmed by repeated runs
• Damage sets overlap only partially across backbones

Task Preservation — Prompt Comparison

Table 3: Task Preservation POST-pass counts and Detail Retention (Kimi-K3 summarizer)

Detail Retention Results

Table 4: Recurring-Detail Retention score by summarizer backbone (Claude Code prompt)

Production-Chain Retention

• First boundary: 15.7% retention (n=106)
• Second boundary: 19.9% (n=27)
• Third boundary: 4.9% (n=6)
• Fourth boundary: 1.8% (n=1)

Diagnosing Task Damage

Table 5: Retained repair evidence in stably damaged cells under Kimi-K3 summaries

The sphinx-9230 trace reveals the key failure pattern: POST reaches the relevant file, searches broadly, and reads test/reproduction code—but never makes the repair. The summary retained strings from the fix but omitted the reasoning that split(None, 1) must become rsplit.

Theoretical and Practical Implications

1. Retention ≠ Preservation: The benchmark establishes a central distinction—retaining paths and code strings can coexist with losing the progress needed to finish a task. This challenges the assumption that information retention metrics alone can predict task success.

2. Configuration sensitivity: Both summarizer model choice and prompt instruction are consequential configuration choices. The interaction between summarizer and prompt (e.g., DeepSeek-Flash leads under some prompts but not others) demonstrates that compaction performance is not a property of the model alone.

3. Mechanical extraction gap: The 63.1-percentage-point gap between LLM summarization (16.0%) and mechanical extraction (79.1%) suggests that current summarization approaches lose substantial structured information, motivating hybrid approaches.

4. Evaluation methodology: Compact-Bench's paired continuation design isolates the effect of history replacement, providing instance-level evidence (changed test verdict = task damage; missing literal value = detail loss) rather than aggregate scores alone.

5. Practical guidance: For deployed coding agents, the results suggest that compaction prompts should focus on preserving progress (what has been done, what remains) rather than merely retaining relevant code strings.

Conclusion

Compact-Bench measures what survives context compaction through three complementary lenses: paired task continuation, recurring-detail matching, and real production chains. The shared-boundary comparison separates summarizers by their ability to preserve executable repair progress, with Claude Opus 4.8 having the lowest stable damage (5%) and DeepSeek-Flash the highest (64%). Prompt choice also matters significantly, with different prompts leading in task preservation vs. detail retention.

The central finding is that retaining relevant information alone does not ensure successful continuation. Evaluating both information retention and subsequent agent behavior makes this loss visible and provides a more complete picture of compaction quality.

Limitations: Results are conditional on tested repository repairs, continuation agent, and budget. Detail-retention scores capture literal recurrence only, and later production-boundary estimates use small, changing session subsets. Private session payloads and cached repair states are unavailable.

Future directions implied by this work include: developing compaction methods that preserve task progress explicitly, investigating hybrid mechanical+LLM summarization approaches, and extending the benchmark to more repositories and agent configurations.