# Do They Do as They Are Told? Investigating Coding Agents' Adherence to Explicit Instructions in Context Files

> Only 59.8% of agent context file instructions are followed, with adherence sharply bimodal: 54.4% always followed, 37.3% never.

- **Source:** [arXiv](https://arxiv.org/abs/2611.13709)
- **Published:** 2026-11-14
- **Permalink:** https://picx.dev/p/30T0tp
- **Whiteboard:** https://picx.dev/p/30T0tp/image

## Summary

## Summary (Overview)

- This paper presents the first instruction-level empirical study of whether coding agents actually follow the explicit instructions written in Agent Context Files (ACFs) such as AGENTS.md, CLAUDE.md, and copilot-instructions.md.
- The study analyzes 621 unique ACF snapshots linked to 6,541 agent-generated commits across 268 repositories, using a three-phase pipeline: instruction characterization, operationalization into executable measurement scripts, and compliance assessment.
- Key findings: 62.9% of explicit, actionable, and observable instructions can be operationalized into measurement scripts; among evaluable instructions (2,053), mean compliance is 59.8%, but compliance is strongly **bimodal**—54.4% of instructions are followed in every commit, while 37.3% are **never** followed.
- Compliance varies significantly by instruction category (e.g., Development Process instructions are least followed) and by coding agent (Copilot follows its ACFs more than Claude Code or OpenAI Codex).
- The paper provides a replication package including the extraction/classification pipeline, generated measurement scripts, and the compliance dataset.

## Introduction and Theoretical Foundation

**Motivation:** Developers increasingly rely on Agent Context Files (ACFs)—persistent, project-scoped artifacts like `AGENTS.md` or `CLAUDE.md`—to steer autonomous coding agents' behavior. Tools such as Claude Code, OpenAI Codex, and GitHub Copilot automatically load these files at session start. However, the core assumption behind ACF-based governance—that instructions written in these files are actually followed—remains untested.

**Key Gap:** Existing studies (Chatlatanagulchai et al. [4, 5]) describe ACF *content* through high-level categories (e.g., Build commands, Implementation details) but do not assess whether individual instructions are followed. The related work on instruction following shows:
- CodeIF-Bench [34] and AgentIF [26] measure instruction adherence but only within single interactions, not persistent project artifacts.
- Watanabe et al. [36] find agent PRs need substantial human revision.
- Yang et al. [37] measure compliance with repository-level AI policies but not broader ACF development guidance.

**Theoretical Foundation:** The paper frames ACFs as a form of "procedural memory" in Wang et al.'s perception–memory–action framework for agent systems. ACFs represent developer-authored, persistent guidance that injects project-specific knowledge into agent context. The study takes an **artifact-centric view**—measuring whether instructions are reflected in produced software artifacts without inferring agent intent or internal reasoning.

**Three Criteria for Instructions:**
- **Explicitness**: directly and clearly stated in the ACF, not inferred
- **Actionability**: describes a concrete development behavior an agent can perform
- **Observability**: compliance checkable through produced/modified artifacts without inferring intent

## Methodology

The study is structured as a three-phase empirical pipeline:

### RQ₁: Instruction Characterization
- **Data**: 621 unique context snapshots (deduplicated ACF versions) from repositories using Copilot, Claude Code, or OpenAI Codex.
- **Phase 1 (Manual Calibration)**: Iterative manual analysis of 15 snapshots until qualitative saturation, establishing extraction procedures and three coding criteria (explicitness, actionability, observability).
- **Phase 2 (LLM-assisted classification)**: Qwen 3.5 9B used as annotator with few-shot prompting, validated against Ministral 3 14B (74.8% cross-model agreement; Qwen selected for less over-extraction).
- **Phase 3 (Manual validation)**: Statistically representative sample of 84 snapshots (95% confidence, 10% margin) validated by a single experienced annotator.

### RQ₂: Operationalization
- **Objective**: Generate executable measurement scripts for each instruction using **test-driven code generation** adapted from Chen et al.'s CodeT [6].
- **Process**: For each instruction:
  - The code LLM (Qwen2.5-Coder-14B-Instruct) generates N=10 candidate measurement scripts and M=10 candidate test fixtures (pairs of synthetic compliant/non-compliant commits).
  - Scripts are validated against fixtures with two acceptance bars:
    - **Minimum-correctness bar**: must pass ≥ 2 fixtures
    - **Corroboration bar**: ≥ 60% overlap with best-performing script's passed-fixture set
  - Only 62.9% of instructions (4,123 of 6,552) yield accepted scripts.

### RQ₃: Compliance Assessment
- **Applicability filter**: For each instruction, derive file patterns (extensions or basenames) from its validated fixtures. Only commit–instruction pairs where the commit touches a matching file are evaluated.
- **Compliance signal**: Binary (compliant/non-compliant) per instruction–commit pair, based on script output.
- **Statistical analysis**: 
  - Aggregation to one compliance rate per instruction to avoid pseudo-replication
  - Kruskal-Wallis tests for cross-category and cross-agent differences
  - Post-hoc Mann-Whitney U tests with Bonferroni correction
  - Cliff's delta for effect sizes
  - Sensitivity analysis at ≥3 and ≥5 applicable-commit thresholds

**Key validation data**: Manual spot-check of 94 accepted scripts found only 71.3% faithfully implement their instruction's specification; most common error is hardcoding ACF examples as literal requirements.

## Empirical Validation / Results

### RQ₁ Results: Instruction Characterization
The classification funnel on 14,902 candidate instructions:
- **Explicit**: 10,324 (69.3%)
- **+ Actionable**: 8,726 (58.6%)
- **+ Observable**: 6,552 (44.0%)

**Category distribution** (final observable dataset): Implementation Details dominates (50.0%), followed by Testing (14.1%) and Documentation (12.4%). Process-oriented categories drop sharply at observability: Build and Run falls from 8.8% of explicit-and-actionable to 2.9% of final.

**Validation**: Extraction precision 96.3%, recall 99.9%, F1-score 98.1%. Observability is the least reliably labeled criterion (85.0% accuracy, biased toward over-inclusion).

### RQ₂ Results: Operationalization Rate

**Overall rate**: 62.9% (4,123 of 6,552 instructions).

| Category | Total | Accepted | Rate |
|---|---|---|---|
| Implementation Details | 3,279 | 2,008 | 61.2% |
| Testing | 927 | 626 | 67.5% |
| Documentation | 810 | 505 | 62.3% |
| Configuration and Environment | 356 | 269 | **75.6%** |
| Security | 99 | 48 | **48.5%** |
| Management | 65 | 31 | **47.7%** |

Operationalization rate is consistent across agents (62.1%–63.5%). Exclusions are primarily due to correctness failures (57.3%) and lack of corroborating scripts (41.1%).

### RQ₃ Results: Compliance

**Coverage**: Only 49.8% of operationalized instructions (2,053 of 4,123) ever meet an applicable commit; 18,594 instruction–commit pairs evaluated.

**Overall compliance**: Mean rate 59.8%, median 100.0%. Distribution is strongly **bimodal**:
- 54.4% (1,116) followed in *every* applicable commit
- 37.3% (766) followed in *none*
- Only 8.3% (171) partially followed

**Category-level variation** (significant, $H = 59.13, p < .001$):
- **Development Process** least compliant (mean 34.0%, median 0.0%), significantly lower than Build and Run, Documentation, and Implementation Details (but not robust to sensitivity check)
- **Implementation Details** (mean 63.6%) significantly higher than **Testing** (mean 49.2%)—the only category comparison robust at every evidence threshold
- **Architecture** (mean 38.2%) and **Debugging** (mean 21.9%) low but not statistically significant after correction

**Agent-level variation** (significant, $H = 41.58, p < .001$):

| Agent | Evaluated | Mean Compliance | Median |
|---|---|---|---|
| Copilot | 1,137 | **66.0%** | 100.0% |
| Claude Code | 799 | 52.2% | 66.7% |
| OpenAI Codex | 117 | 51.2% | 100.0% |

- Copilot significantly higher than Claude Code ($p_{Bonf} < .001$, $\delta = 0.15$)—robust at all thresholds
- Copilot vs. OpenAI Codex significant only in full sample ($p_{Bonf} = .018$); Codex sample is small (117)
- Claude Code and OpenAI Codex statistically indistinguishable at every threshold

**Sensitivity analysis**: The bimodal split persists (86.2% at extremes even with ≥5 commits). Implementation Details vs. Testing and Copilot vs. Claude Code gaps remain significant; Development Process gap and Copilot vs. Codex gap do not survive.

## Theoretical and Practical Implications

**For ACF Authors:**
- A large share of ACF content (69.3%) is not even explicit; write for the agent, not the human reader—use imperative phrasing, concrete behaviors, avoid vague qualifiers.
- Prefer **observable** instructions whose compliance leaves artifact traces over process-only directives ("run the linter") that cannot be audited.
- Workflow-level conventions (Development Process guidance) may be better encoded in project tooling (commit hooks, CI configs) than in ACF prose.

**For Engineering Agentic Systems:**
- The **bimodal compliance** shape means spot-checking is insufficient; continuous, automated verification is needed, targeted at least reliable categories and agents.
- Verification mechanisms should be **calibrated per ACF–agent pair**; compliance differences across agents suggest reliability is a property of both the ACF and the agent together.
- Test-driven script generation with dual-bar acceptance demonstrates that artifact-grounded, per-instruction verification is practical today for a majority of observable ACF content—but ~30% of generated scripts still need human review before trust.

**For Researchers:**
- The **single instruction**, not the file or category, should be the unit of analysis; aggregate claims obscure the bimodal reality.
- The full pipeline (from 14,902 candidates to 2,053 evaluable instructions, only 13.8%) shows which guidance is verifiable at all.

## Conclusion

This paper provides the first instruction-level assessment of whether coding agents follow ACF guidance in real-world development. The key findings are: (1) ACFs encode a much smaller proportion of explicit, actionable, and observable instructions (44%) than raw content suggests; (2) where compliance can be measured, agents follow ACF guidance more often than not (59.8% mean), but adherence is sharply bimodal—instructions are either consistently followed or consistently ignored; and (3) compliance varies meaningfully across instruction categories and coding agents, with Copilot showing higher adherence than Claude Code or Codex.

**Limitations**: Scope limited to three agents and public GitHub repositories; relies on LLM-based extraction with single-annotator validation; scripts receive full-file content but not line-level diffs; potential confounds in agent-level comparisons (category mix, repository differences).

**Future work**: Extend to additional agents and ACF conventions; investigate underlying causes of category- and agent-level compliance gaps; explore enforcement mechanisms that ensure ACF adherence at generation time rather than measuring it post-hoc.

---

_Markdown view of https://picx.dev/p/30T0tp, served by PicX — AI-generated visual whiteboard summaries of research papers._