The Working Set of a Coding Agent: Coherence Debt in Repository-Scale Tasks

Summary (Overview)

• Core concept: The paper introduces coherence debt, defined as the set of coupled facts required for a correct repository-scale edit that are available through neither recent context ($R_t$) nor parametric memory ($K_M$), formalized as $D(e_i) = |C_T^{(i)} \setminus (R_{t_i} \cup K_M)|$.

• Key finding 1: The two channels (context and parametric memory) are substitutable — across 154 closed-book trials on fictional APIs, no model completes a task (0/12 score), but supplying the facts in the prompt lifts 299/300 trials to ≥9/12 requirements.

• Key finding 2: Availability decides outcomes, distance does not — withholding facts costs exactly the work they support (linear damage: 32, 24, 16, 8, 0 tests passed for 0, 2, 4, 6, 8 withheld facts), while facts supplied 128,000–200,000 characters away work as well as those adjacent to the edit.

• Key finding 3: Agents compensate rather than abstain — missing facts produce confident wrong edits (fabricated files, guessed values), and the rate at which agents report being blocked ranges from 0% to 100% depending on the model.

• Key finding 4: Harnesses pay unequal prices for the same result — configurations that all pass every test differ by 12.8× in tokens consumed (293,882 to 3,752,134) while holding nearly the same content (1.8× peak context variance), and spending more recovers nothing when facts are withheld.

Introduction and Theoretical Foundation

Background and Motivation

Repository-scale coding tasks (migrations, upgrades, bug fixes) require edits in one file to remain consistent with facts elsewhere: validators must agree with tests, renamed symbols with imports, and configuration changes with runtime contracts. The agent must reconstruct this structure through reads, edits, tests, and handoffs.

The Working-Set Analogy

The paper adapts Denning's working-set concept from virtual memory (1968) to an "addressless" setting. Unlike memory pages, repository facts carry no address or invalidation bit — a migration rule stated in prose and a stale implementation may encode different versions of one fact, yet nothing tells the agent which is current.

Coupled-Fact Graph

A task $T$ induces a graph $G_T = (V_T, E_T)$ where:

• Nodes are atomic facts: symbols, tests, configuration values, imports, migration rules, invariants
• Edges mean that relying on or changing one endpoint requires the other to remain consistent
• $C_T \subseteq V_T$ is a minimal set that must be jointly correct for the task oracle to pass
• $C_T^{(i)}$ is the subset required by edit $e_i$

Coherence Debt Definition

At edit time $t_i$, with $R_{t_i}$ denoting facts in effective context and $K_M$ denoting facts in parametric memory:

$$D(e_i) = |C_T^{(i)} \setminus (R_{t_i} \cup K_M)|.$$

The union in the equation carries the key prediction: success depends on coverage, not on which channel supplied the fact.

Observable Proxy

Since the latent graph is not directly observable, the paper defines a residency score:

$$\rho_w(f_i, t_i) = \frac{|N(f_i) \cap \text{Read}(t_i - w, t_i)|}{|N(f_i)|}$$

where $N(f_i)$ contains one-hop import neighbors and $\text{Read}(t_i - w, t_i)$ contains files read during the preceding $w$ tool events. The score $\rho_w = 1$ when $|N(f_i)| = 0$.

Pre-Registered Predictions

• (P1) Empty channels → floor performance; supplying facts through either channel lifts it
• (P2) Models sharing a partial prior fail on the same task region
• (P3) Coverage acts through presence, not proximity
• (P4) Placement and decomposition matter through the coupled-fact graph

Methodology

Channel-Control Workloads

Four hand-authored fictional API migrations (Sprocket/Rust, Grimwire/Go, Kestrix/Python, Zynet/JavaScript) with invented library names, 3 editable files, a change log, and 12 mechanically checked requirements per task. A real Pydantic v1→v2 migration (79 tests) and an adversarial renamed twin (lexical shim defeating memorized knowledge) control for parametric coverage.

Experimental Conditions

• Closed-book: task description only, no workspace, no tools → both channels empty
• Front-loaded: exact rules and source files in prompt, model and tools unchanged
• Renamed: Pydantic API surface renamed to defeat $K_M$

Tool-Using Workloads

• 122 matched trials across Claude Code, Codex CLI, Aider, and OpenHands
• Synthetic-coherence tasks: three files coupled through random literals in a secret file; required facts known by construction
• Fault injection: withhold secrets for controlled numbers of motifs (0–8 of 8)
• Distance arms: facts supplied at varying distances (up to 128,000–200,000 characters)
• Decomposition probes: subagent partitioning on coupled vs. independent tasks

Models Tested

Seven model families (Sonnet, Haiku, Codex, Z.ai, DeepSeek, Qwen, Gemini) for closed-book; five for sandboxed front-load; four model/harness families for SWE-bench validation.

Prospective Event Semantics

Five event types with required payloads: file_read, fact_extracted, edit_intent, test_feedback, revert. This separates four failure causes: working-set miss, stale read, handoff gap, and speculative write.

Empirical Validation / Results

Channel Substitution (P1, P2)

The Wilson 95% upper bound on nonzero complete-solution probability in closed-book is 2.4%. Renaming relocates the shared failure point without dispersing it — all seven models pass the identical 24 tests.

Availability and Distance (P3)

Linear damage tracking: Withholding $k$ of 8 motifs costs exactly the work those motifs support:

Maximum deviation over nine cells is zero tests (n=6 per cell). Damage adds rather than compounds.

Distance invariance: Supplying a fact at 128,000–200,000 characters distance shows no degradation. Sixteen invariants stated once remain honored across up to 96 subsequent tasks (~140,000 tokens), with no working-set misses.

Harness Efficiency

Across 144 trials (6 configurations × 3 task sizes × 8 trials), all passing every test:

• Peak per-turn context: 1.8× variance
• Cumulative input: 12.8× variance (293,882 to 3,752,134 tokens)
• Tool calls: 5 to 79 per run

Withholding facts shows all configurations lose exactly the proportion withheld:

No two configurations differ by more than 3 points at any level, despite Haiku consuming 5,730,807 cumulative tokens vs. Opus's 459,122 at k=0.

Compensation Rather Than Abstention

Blocked-report rates vary entirely by model:

The range runs from never to always. A separate 96-trial Haiku block confirms stability (13.5% vs. 12.5%).

Coverage Is Not Sufficient

When a written standard contradicts working code, agents follow the standard in 39/39 trials (Wilson 95% interval [0.91, 1.00]), even when the standard prescribes worse code (camelcase handlers, float money division, direct indexing, naive clock).

Stale Standard Is Worse Than No Standard

Ten trials per condition, 3,385 scored decisions. A stale standard suppresses the inference agents would otherwise make from code.

SWE-bench Transfer Failure

• Resolved rates: GPT-5/Codex CLI 39.4%, Sonnet/Claude Code 31.3%, Haiku/Claude Code 22.2%, Sonnet/Aider 1.0%
• Final-quarter residency score gives within-cell AUC ≈ 0.49 (chance) across 122 recoverable trajectories
• An outcome-gated "retirement" term produced AUC > 0.92, but removing the leakage eliminates the effect

Theoretical and Practical Implications

Theoretical Contributions

1. Two-channel, edit-time account: Coherence is determined by the union of recent context and parametric memory at the moment of writing, not by total context consumed
2. Measurement proposition: Event-only estimators overstate missing facts by exactly the parametric coverage — a union-aware estimator is exact (verified by enumerating 1,089 simulator runs)
3. Additivity of debt: Missing facts cost exactly the work they support, not more — damage is additive, not compounding

Practical Implications for Harnesses

• Keep coupled facts available: Harnesses should log edit intents, attach versioned supporting facts, invalidate them after writes, and require explicit transfer across workers
• Task graph beyond imports: Include tests, instructions, and runtime invariants, not just static imports
• Delete stale convention files: A stale standard costs more than no file at all
• Agent-authored paths must be excluded from coverage numerators — agents that fabricate missing files would otherwise score perfectly
• Blocked-reporting is a harness-selection question: Some models (Opus) reliably report missing facts; others (Codex CLI, opencode) never do

Measurement Limits

• The residency score is workload-relative: held-out AUC 0.71 under random split, 0.66 under leave-one-workload-out
• Import neighborhoods recover only 40% of authored coupling edges at 40% precision; directed graphs remove spurious edges at no recall cost
• The score cannot see facts supplied via prompt, parametric memory, or handoff

Conclusion

Main Takeaways

Repository-scale coding depends on an edit-time working set of coupled facts. Success tracks whether those facts are present when the agent writes, from context or model prior, rather than how much context is consumed or how far back a supplied fact sits.

Scope Conditions

1. Coverage is not sufficient: When two covered facts disagree, agents follow written standards over working code, even when the standard prescribes worse code
2. The framework does not transfer to real repositories: On SWE-bench, where models likely know the repositories, read-derived proxies no longer predict success

Falsifiable Boundary

Three patterns would weaken the framework: (1) reliable success with both channels empty, (2) equal predictive power for arbitrary and coupled-fact reads, or (3) decomposition effects unrelated to the cut. None were observed.

Future Directions

A decisive test should preregister a task-held-out graph and distance, fork the same pre-edit state, and randomize an identical required fact to absent, early, recent, and refreshed positions. It should compare import, lexical, heterogeneous-graph, and dataflow retrieval on independently authored non-migration tasks before relating coverage to edit correctness.