# The Working Set of a Coding Agent: Coherence Debt in Repository-Scale Tasks

> Repository-scale coding success depends on edit-time availability of coupled facts from context or parametric memory, not on total context consumed or fact distance.

- **Source:** [arXiv](https://arxiv.org/abs/2608.16630)
- **Published:** 2026-08-22
- **Permalink:** https://picx.dev/p/3QMCRC
- **Whiteboard:** https://picx.dev/p/3QMCRC/image

## Summary

# The Working Set of a Coding Agent: Coherence Debt in Repository-Scale Tasks

## Summary (Overview)

- **Core concept**: The paper introduces **coherence debt**, defined as the set of coupled facts required for a correct repository-scale edit that are available through neither recent context ($R_t$) nor parametric memory ($K_M$), formalized as $D(e_i) = |C_T^{(i)} \setminus (R_{t_i} \cup K_M)|$.

- **Key finding 1**: The two channels (context and parametric memory) are **substitutable** — across 154 closed-book trials on fictional APIs, no model completes a task (0/12 score), but supplying the facts in the prompt lifts 299/300 trials to ≥9/12 requirements.

- **Key finding 2**: **Availability decides outcomes, distance does not** — withholding facts costs exactly the work they support (linear damage: 32, 24, 16, 8, 0 tests passed for 0, 2, 4, 6, 8 withheld facts), while facts supplied 128,000–200,000 characters away work as well as those adjacent to the edit.

- **Key finding 3**: Agents **compensate rather than abstain** — missing facts produce confident wrong edits (fabricated files, guessed values), and the rate at which agents report being blocked ranges from 0% to 100% depending on the model.

- **Key finding 4**: Harnesses pay **unequal prices** for the same result — configurations that all pass every test differ by 12.8× in tokens consumed (293,882 to 3,752,134) while holding nearly the same content (1.8× peak context variance), and spending more recovers nothing when facts are withheld.

## Introduction and Theoretical Foundation

### Background and Motivation
Repository-scale coding tasks (migrations, upgrades, bug fixes) require edits in one file to remain consistent with facts elsewhere: validators must agree with tests, renamed symbols with imports, and configuration changes with runtime contracts. The agent must reconstruct this structure through reads, edits, tests, and handoffs.

### The Working-Set Analogy
The paper adapts **Denning's working-set concept from virtual memory** (1968) to an "addressless" setting. Unlike memory pages, repository facts carry no address or invalidation bit — a migration rule stated in prose and a stale implementation may encode different versions of one fact, yet nothing tells the agent which is current.

### Coupled-Fact Graph
A task $T$ induces a graph $G_T = (V_T, E_T)$ where:
- **Nodes** are atomic facts: symbols, tests, configuration values, imports, migration rules, invariants
- **Edges** mean that relying on or changing one endpoint requires the other to remain consistent
- $C_T \subseteq V_T$ is a minimal set that must be jointly correct for the task oracle to pass
- $C_T^{(i)}$ is the subset required by edit $e_i$

### Coherence Debt Definition
At edit time $t_i$, with $R_{t_i}$ denoting facts in effective context and $K_M$ denoting facts in parametric memory:

$$D(e_i) = |C_T^{(i)} \setminus (R_{t_i} \cup K_M)|.$$

The union in the equation carries the key prediction: success depends on **coverage**, not on which channel supplied the fact.

### Observable Proxy
Since the latent graph is not directly observable, the paper defines a **residency score**:

$$\rho_w(f_i, t_i) = \frac{|N(f_i) \cap \text{Read}(t_i - w, t_i)|}{|N(f_i)|}$$

where $N(f_i)$ contains one-hop import neighbors and $\text{Read}(t_i - w, t_i)$ contains files read during the preceding $w$ tool events. The score $\rho_w = 1$ when $|N(f_i)| = 0$.

### Pre-Registered Predictions
- **(P1)** Empty channels → floor performance; supplying facts through either channel lifts it
- **(P2)** Models sharing a partial prior fail on the same task region
- **(P3)** Coverage acts through presence, not proximity
- **(P4)** Placement and decomposition matter through the coupled-fact graph

## Methodology

### Channel-Control Workloads
Four hand-authored fictional API migrations (Sprocket/Rust, Grimwire/Go, Kestrix/Python, Zynet/JavaScript) with invented library names, 3 editable files, a change log, and 12 mechanically checked requirements per task. A real Pydantic v1→v2 migration (79 tests) and an adversarial renamed twin (lexical shim defeating memorized knowledge) control for parametric coverage.

### Experimental Conditions
- **Closed-book**: task description only, no workspace, no tools → both channels empty
- **Front-loaded**: exact rules and source files in prompt, model and tools unchanged
- **Renamed**: Pydantic API surface renamed to defeat $K_M$

### Tool-Using Workloads
- 122 matched trials across Claude Code, Codex CLI, Aider, and OpenHands
- **Synthetic-coherence tasks**: three files coupled through random literals in a secret file; required facts known by construction
- **Fault injection**: withhold secrets for controlled numbers of motifs (0–8 of 8)
- **Distance arms**: facts supplied at varying distances (up to 128,000–200,000 characters)
- **Decomposition probes**: subagent partitioning on coupled vs. independent tasks

### Models Tested
Seven model families (Sonnet, Haiku, Codex, Z.ai, DeepSeek, Qwen, Gemini) for closed-book; five for sandboxed front-load; four model/harness families for SWE-bench validation.

### Prospective Event Semantics
Five event types with required payloads: `file_read`, `fact_extracted`, `edit_intent`, `test_feedback`, `revert`. This separates four failure causes: working-set miss, stale read, handoff gap, and speculative write.

## Empirical Validation / Results

### Channel Substitution (P1, P2)

| Experiment | Channel state | n | Outcome | Result |
|---|---|---|---|---|
| Novel closed-book | $R_t = \emptyset$, task-specific $K_M \approx \emptyset$ | 154 | complete migration | 0/154; every trial 0/12 |
| Novel front-load | exact rules/source supplied in $R_0$ | 300 | at least 9/12 requirements | 299/300; mean cell scores 9.0–12.0 |
| Pydantic closed-book | public API via $K_M$ | 32/36 | passed-test identity | same 53/79 tests; Jaccard 1.000 |
| Renamed closed-book | lexical access to prior defeated | 66/70 | passed-test identity | same 24/79 tests across seven families; Jaccard 1.000 |

The Wilson 95% upper bound on nonzero complete-solution probability in closed-book is 2.4%. Renaming relocates the shared failure point without dispersing it — all seven models pass the identical 24 tests.

### Availability and Distance (P3)
**Linear damage tracking**: Withholding $k$ of 8 motifs costs exactly the work those motifs support:

| facts withheld | 0 | 2 | 4 | 6 | 8 |
|---|---|---|---|---|---|
| tests passed, stub | 32 | 24 | 16 | 8 | 0 |
| tests passed, deleted | — | 24 | 16 | 8 | 0 |
| linear prediction | 32 | 24 | 16 | 8 | 0 |

Maximum deviation over nine cells is **zero tests** (n=6 per cell). Damage adds rather than compounds.

**Distance invariance**: Supplying a fact at 128,000–200,000 characters distance shows no degradation. Sixteen invariants stated once remain honored across up to 96 subsequent tasks (~140,000 tokens), with no working-set misses.

### Harness Efficiency
Across 144 trials (6 configurations × 3 task sizes × 8 trials), all passing every test:
- **Peak per-turn context**: 1.8× variance
- **Cumulative input**: 12.8× variance (293,882 to 3,752,134 tokens)
- **Tool calls**: 5 to 79 per run

Withholding facts shows all configurations lose exactly the proportion withheld:

| configuration | k=0 | k=4 | k=8 |
|---|---|---|---|
| Opus | 100% | 50% | 0% |
| Fable | 100% | 50% | 0% |
| Sonnet | 100% | 50% | 0% |
| Haiku | 100% | 53% | 0% |
| Codex | 100% | 50% | 0% |

No two configurations differ by more than 3 points at any level, despite Haiku consuming 5,730,807 cumulative tokens vs. Opus's 459,122 at k=0.

### Compensation Rather Than Abstention
Blocked-report rates vary entirely by model:

| configuration | n | blocked | share |
|---|---|---|---|
| Opus, Claude Code | 8 | 8 | 100% |
| Fable, Claude Code | 8 | 6 | 75% |
| Sonnet, Claude Code | 8 | 2 | 25% |
| Haiku, Claude Code | 8 | 1 | 12.5% |
| GPT-5, Codex CLI | 8 | 0 | 0% |
| GLM-5.2, opencode | 6 | 0 | 0% |

The range runs from never to always. A separate 96-trial Haiku block confirms stability (13.5% vs. 12.5%).

### Coverage Is Not Sufficient
When a written standard contradicts working code, agents follow the standard in **39/39 trials** (Wilson 95% interval [0.91, 1.00]), even when the standard prescribes worse code (camelcase handlers, float money division, direct indexing, naive clock).

### Stale Standard Is Worse Than No Standard

| what the agent has | better form |
|---|---|
| a standard agreeing with the code | 100% |
| only code demonstrating the convention | 33% |
| a standard demanding the worse form | 0% |

Ten trials per condition, 3,385 scored decisions. A stale standard suppresses the inference agents would otherwise make from code.

### SWE-bench Transfer Failure
- Resolved rates: GPT-5/Codex CLI 39.4%, Sonnet/Claude Code 31.3%, Haiku/Claude Code 22.2%, Sonnet/Aider 1.0%
- Final-quarter residency score gives within-cell AUC ≈ 0.49 (chance) across 122 recoverable trajectories
- An outcome-gated "retirement" term produced AUC > 0.92, but removing the leakage eliminates the effect

## Theoretical and Practical Implications

### Theoretical Contributions
1. **Two-channel, edit-time account**: Coherence is determined by the union of recent context and parametric memory at the moment of writing, not by total context consumed
2. **Measurement proposition**: Event-only estimators overstate missing facts by exactly the parametric coverage — a union-aware estimator is exact (verified by enumerating 1,089 simulator runs)
3. **Additivity of debt**: Missing facts cost exactly the work they support, not more — damage is additive, not compounding

### Practical Implications for Harnesses
- **Keep coupled facts available**: Harnesses should log edit intents, attach versioned supporting facts, invalidate them after writes, and require explicit transfer across workers
- **Task graph beyond imports**: Include tests, instructions, and runtime invariants, not just static imports
- **Delete stale convention files**: A stale standard costs more than no file at all
- **Agent-authored paths must be excluded** from coverage numerators — agents that fabricate missing files would otherwise score perfectly
- **Blocked-reporting is a harness-selection question**: Some models (Opus) reliably report missing facts; others (Codex CLI, opencode) never do

### Measurement Limits
- The residency score is workload-relative: held-out AUC 0.71 under random split, 0.66 under leave-one-workload-out
- Import neighborhoods recover only 40% of authored coupling edges at 40% precision; directed graphs remove spurious edges at no recall cost
- The score cannot see facts supplied via prompt, parametric memory, or handoff

## Conclusion

### Main Takeaways
Repository-scale coding depends on an **edit-time working set of coupled facts**. Success tracks whether those facts are present when the agent writes, from context or model prior, rather than how much context is consumed or how far back a supplied fact sits.

### Scope Conditions
1. **Coverage is not sufficient**: When two covered facts disagree, agents follow written standards over working code, even when the standard prescribes worse code
2. **The framework does not transfer to real repositories**: On SWE-bench, where models likely know the repositories, read-derived proxies no longer predict success

### Falsifiable Boundary
Three patterns would weaken the framework: (1) reliable success with both channels empty, (2) equal predictive power for arbitrary and coupled-fact reads, or (3) decomposition effects unrelated to the cut. None were observed.

### Future Directions
A decisive test should preregister a task-held-out graph and distance, fork the same pre-edit state, and randomize an identical required fact to absent, early, recent, and refreshed positions. It should compare import, lexical, heterogeneous-graph, and dataflow retrieval on independently authored non-migration tasks before relating coverage to edit correctness.

---

_Markdown view of https://picx.dev/p/3QMCRC, served by PicX — AI-generated visual whiteboard summaries of research papers._