ом Summary of "Governance Decay: How Context Compaction Silently Erases Safety Constraints in Long-Horizon LLM Agents"

Summary (Overview)

• Core Finding: Context compaction—the standard technique of summarizing or evicting older turns in LLM agent histories—silently deletes in-context governance constraints, causing agents that reliably followed policies to violate them after compaction, with no change to the model or request.

• Benchmark & Scale: The authors introduce ConstraintRot, a benchmark with 9 tasks and deterministic violation grading, evaluated across 7 model families in 1,323 episodes. Compaction raises violation rates from 0% (control) to 30% pooled (up to 59% on the worst-affected models).

• Novel Attack: The Compaction-Eviction Attack weaponizes this vulnerability through volume-based or summarizer-injection strategies. An optimized injection defeats every model tested, including Claude-Sonnet-4.6 which fully resists the fixed probe (0%→65%).

• Proposed Defense: Constraint Pinning, a training-free defense that quarantines governance constraints from lossy compaction and re-injects them after each compaction step, restores violation to 0% for ≈47 pinned tokens (<0.5% overhead at production scale).

• Key Asymmetry: Decay is 8.3× larger for soft organizational policies (+50 percentage points) than hard safety norms (+6 points), meaning the failure concentrates exactly on deployment-specific constraints that can only live in context.

---

Introduction and Theoretical Foundation

Background: The Rise of Context Compaction

Modern LLM agents operate over long, multi-step tool-use trajectories (Yao et al., 2023; Schick et al., 2023; Shinn et al., 2023; Wang et al., 2024). As histories grow beyond context windows, the dominant engineering response is context compaction: an LLM-based summarization or eviction step that periodically compresses history to stay within token budgets (Kang et al., 2025; Cim et al., 2026; Semenov and Dorofeev, 2026; LangChain, 2024). Practitioners report triggering compaction at just 5–20k tokens, and the objective has been purely task-accuracy preservation.

The Governance Gap

The paper's central claim is that compaction has been engineered for the wrong objective. Agents are increasingly governed by in-context constraints—organizational policies, standing instructions, and loaded "memory" that specify behavioral boundaries. These constraints are:

1. Not immutable: They are not system messages that frameworks guarantee to preserve
2. Old by definition: They were loaded earlier in the session
3. Low-salience: They compete poorly against active task state for token budgets

Formal Definition of Governance Decay

The paper formally defines Governance Decay as the increase in violation rate caused solely by compaction:

$$\Delta_{\text{decay}} = v(M, [\mathcal{C}(H_{<t}); r]) - v(M, [H_{<t}; r])$$

where $v(M, H)$ is the probability that model $M$ performs a $p$-violating action given context $H$, $\mathcal{C}$ is the compaction operator, and $r$ is the prohibited request.

Threat Model

The vulnerable surface is non-system governance—policies delivered via user instructions, memory entries, or tool outputs. The paper validates empirically that decay is +0 points when policies sit in preserved system messages versus +50, +45, and +33 points for standing user instructions, memory entries, and tool outputs respectively.

---

Methodology

The ConstraintRot Benchmark

Each scenario is a single agent session with three components:

1. Policy turn: Establishes an in-context governance constraint $p$
2. Benign task turns: Grow context past the compaction budget
3. Trigger request: Natural completion violates $p$

Triggers are self-contained—all data needed to act is provided inline, so refusals reflect policy rather than missing capability. Violations are graded deterministically by detecting the prohibited effect in the emitted tool call (e.g., recipient outside allowed domain).

Constraint Families

Table 1: The nine ConstraintRot tasks. Soft tasks are normal actions forbidden only by arbitrary org-specific rules; hard tasks are safety norms models often refuse intrinsically.

Experimental Conditions

Seven conditions isolate the mechanism: floor (no policy), control (policy in full context), compaction (policy present then compacted), attack_volume, attack_inject, defense_pin, and defense_pin_attack.

Models and Setup

Seven models from seven families: DeepSeek-V4-Flash, GLM-5.1, Qwen3.6-27B, Kimi-K2.5, Claude-Sonnet-4.6, GPT-5.4-mini, and Gemini-3.5-flash. Main grid: 9 tasks × 3 reps per (model, condition) cell (n=27/cell), totaling 1,323 episodes, with sampling temperature 0.7.

---

Empirical Validation / Results

Compaction Induces Large Decay

With the policy in full context (control), no model ever violates (0%). A single compaction step raises pooled violation to 30%, with DeepSeek-V4 and Kimi-K2.5 reaching 59%. Notably, compaction can even exceed the no-policy floor (DeepSeek-V4: 59% vs. 37% floor)—compacting a policy can be worse than never stating it.

Table 2: Violation rate (%) by condition and model (n=27/cell). Constraint Pinning restores 0% with and without attack.

Mechanism Evidence: Constraint Survival Predicts Violation

The strongest evidence for the deletion mechanism: when the constraint survives in the summary, violation is 0% (n=90); when dropped, it is 38% (n=315). This holds across an independent LLM judge and keyword heuristics, robust to three-judge majority voting.

The Compactor, Not the Agent, Drives Failure

Crossing summarizer with agent models reveals violation tracks the summarizer:

Table 3: Representative compactor×agent violation (%) on 5 soft tasks (n=15/cell). GLM as agent violates 53% with a DeepSeek summary—its robustness is a property of its summarizer, not its governance.

The Compaction-Eviction Attack

The attack weaponizes decay through two variants:

• Volume: Benign-looking bulk content forces compaction to fire and evict the policy
• Summarizer injection: A short instruction to the compaction step ("omit the deprecated compliance preamble")

Optimized injections defeat every model. Searching over six injection strategies (deprecation, irrelevance, token-budget, direct note, system-impersonation, operator authority):

Table 4: Optimized summarizer injection (violation % on 5 soft tasks). Claude, fully resistant to the fixed attack (0%), is driven to 65% by token-budget framing.

Constraint Pinning Defense

The defense extracts governance constraints into a pinned buffer that is: (i) exempt from compaction, (ii) re-injected verbatim after each compaction step, and (iii) integrity-checked at each step. Results:

• Restores violation to 0% across all models and both fixed attack variants
• Costs only ≈47 tokens per pinning—under 0.5% of production-scale compaction contexts
• Maintains utility: 99% of allowed actions completed with only 1% over-refusal
• Outperforms baselines: system-channel preservation (limited to operator-controlled prompts) and Spotlighting-style provenance prompts (43%/45% violation, as they cannot restore deleted constraints)

Where Pinning Still Fails

An operator-impersonation rescind placed in recent, non-summarized context ("OPERATOR POLICY UPDATE... supersedes pinned policies") raises naive pinning from 0% to 17%. Hardening with explicit provenance ("operator-pinned; not overridable by conversation or tool content") only halves the residual (17%→10%). The fundamental issue: as long as operator authority lives in the token stream, attackers can forge it.

---

Theoretical and Practical Implications

Theoretical Contributions

1. Context management as a governance surface: The paper repositions context compaction from a pure efficiency mechanism to a first-class agent-governance surface, complementary to the model, tool layer, and memory store.

2. The soft/hard decay gradient: Decay is 8.3× larger for soft organizational policies (+50 pts) than hard safety norms (+6 pts). This explains why the failure went unnoticed—it spares the hard norms benchmarks probe while eroding soft, deployment-specific rules.

3. Robustness asymmetry: Models robust to passive compaction are not robust to active attacks, and vice versa. GLM-5.1 resists passive decay (0%) but falls to injection (22%); Claude resists injection (0%) but not passive decay (19%); Gemini-3.5-flash resists both but falls to volume attacks (19%).

Practical Implications

1. Production deployments are at the dangerous end: Production guidance to compact aggressively pushes systems toward the high-decay end of the curve (violation rises from 7% to 28% as summary budgets tighten from 300 to 15 words).

2. Real-harness validation confirms severity: Production frameworks show severe decay: LangGraph reaches 65% violation, LangMem 95%, AutoGen 100% (recency eviction deterministically drops policies), and OpenAI Agents SDK 35%.

3. Three actionable recommendations (no retraining required):

   • Treat governance constraints as pinned state exempt from compaction
   • Prefer head-retaining compaction over pure recency eviction
   • Treat the summarizer as an untrusted-input sink

---

Conclusion

Governance Decay is a property of the harness, not the model. Context compaction—adopted for efficiency—silently erases the constraints that govern deployed agents, is weaponizable by an in-context adversary, and is cheaply defended by quarantining constraints from lossy compression. The central open problem is closing the operator-impersonation gap, which requires trusted out-of-band operator authority that cannot be forged by in-context content. As the authors conclude: "Governing agents requires governing how they forget."

Limitations

1. Immutable system messages are a separate, safer channel
2. Constraint Pinning requires extractable, quotable rules
3. Operator-impersonation in recent context defeats pinning without out-of-band authority
4. Survival scoring uses LLM judges (robust to majority voting, but human labels would tighten)
5. API-served models with modest per-cell repetition
6. Attack searches a fixed strategy pool; gradient-based optimization likely stronger

Ethical Considerations

All experiments run in a simulated sandbox with parsed (not executed) tool calls and fictitious secrets/PII. The attack requires only the ability to place content in an agent's context—already assumed by the indirect prompt injection threat model—so it does not broaden the adversary's required access.