Summary (Overview)

• This paper presents the first instruction-level empirical study of whether coding agents actually follow the explicit instructions written in Agent Context Files (ACFs) such as AGENTS.md, CLAUDE.md, and copilot-instructions.md.
• The study analyzes 621 unique ACF snapshots linked to 6,541 agent-generated commits across 268 repositories, using a three-phase pipeline: instruction characterization, operationalization into executable measurement scripts, and compliance assessment.
• Key findings: 62.9% of explicit, actionable, and observable instructions can be operationalized into measurement scripts; among evaluable instructions (2,053), mean compliance is 59.8%, but compliance is strongly bimodal—54.4% of instructions are followed in every commit, while 37.3% are never followed.
• Compliance varies significantly by instruction category (e.g., Development Process instructions are least followed) and by coding agent (Copilot follows its ACFs more than Claude Code or OpenAI Codex).
• The paper provides a replication package including the extraction/classification pipeline, generated measurement scripts, and the compliance dataset.

Introduction and Theoretical Foundation

Motivation: Developers increasingly rely on Agent Context Files (ACFs)—persistent, project-scoped artifacts like AGENTS.md or CLAUDE.md—to steer autonomous coding agents' behavior. Tools such as Claude Code, OpenAI Codex, and GitHub Copilot automatically load these files at session start. However, the core assumption behind ACF-based governance—that instructions written in these files are actually followed—remains untested.

Key Gap: Existing studies (Chatlatanagulchai et al. [4, 5]) describe ACF content through high-level categories (e.g., Build commands, Implementation details) but do not assess whether individual instructions are followed. The related work on instruction following shows:

• CodeIF-Bench [34] and AgentIF [26] measure instruction adherence but only within single interactions, not persistent project artifacts.
• Watanabe et al. [36] find agent PRs need substantial human revision.
• Yang et al. [37] measure compliance with repository-level AI policies but not broader ACF development guidance.

Theoretical Foundation: The paper frames ACFs as a form of "procedural memory" in Wang et al.'s perception–memory–action framework for agent systems. ACFs represent developer-authored, persistent guidance that injects project-specific knowledge into agent context. The study takes an artifact-centric view—measuring whether instructions are reflected in produced software artifacts without inferring agent intent or internal reasoning.

Three Criteria for Instructions:

• Explicitness: directly and clearly stated in the ACF, not inferred
• Actionability: describes a concrete development behavior an agent can perform
• Observability: compliance checkable through produced/modified artifacts without inferring intent

Methodology

The study is structured as a three-phase empirical pipeline:

RQ₁: Instruction Characterization

• Data: 621 unique context snapshots (deduplicated ACF versions) from repositories using Copilot, Claude Code, or OpenAI Codex.
• Phase 1 (Manual Calibration): Iterative manual analysis of 15 snapshots until qualitative saturation, establishing extraction procedures and three coding criteria (explicitness, actionability, observability).
• Phase 2 (LLM-assisted classification): Qwen 3.5 9B used as annotator with few-shot prompting, validated against Ministral 3 14B (74.8% cross-model agreement; Qwen selected for less over-extraction).
• Phase 3 (Manual validation): Statistically representative sample of 84 snapshots (95% confidence, 10% margin) validated by a single experienced annotator.

RQ₂: Operationalization

• Objective: Generate executable measurement scripts for each instruction using test-driven code generation adapted from Chen et al.'s CodeT [6].
• Process: For each instruction:
  • The code LLM (Qwen2.5-Coder-14B-Instruct) generates N=10 candidate measurement scripts and M=10 candidate test fixtures (pairs of synthetic compliant/non-compliant commits).
  • Scripts are validated against fixtures with two acceptance bars:
    • Minimum-correctness bar: must pass ≥ 2 fixtures
    • Corroboration bar: ≥ 60% overlap with best-performing script's passed-fixture set
  • Only 62.9% of instructions (4,123 of 6,552) yield accepted scripts.

RQ₃: Compliance Assessment

• Applicability filter: For each instruction, derive file patterns (extensions or basenames) from its validated fixtures. Only commit–instruction pairs where the commit touches a matching file are evaluated.
• Compliance signal: Binary (compliant/non-compliant) per instruction–commit pair, based on script output.
• Statistical analysis:
  • Aggregation to one compliance rate per instruction to avoid pseudo-replication
  • Kruskal-Wallis tests for cross-category and cross-agent differences
  • Post-hoc Mann-Whitney U tests with Bonferroni correction
  • Cliff's delta for effect sizes
  • Sensitivity analysis at ≥3 and ≥5 applicable-commit thresholds

Key validation data: Manual spot-check of 94 accepted scripts found only 71.3% faithfully implement their instruction's specification; most common error is hardcoding ACF examples as literal requirements.

Empirical Validation / Results

RQ₁ Results: Instruction Characterization

The classification funnel on 14,902 candidate instructions:

• Explicit: 10,324 (69.3%)
• + Actionable: 8,726 (58.6%)
• + Observable: 6,552 (44.0%)

Category distribution (final observable dataset): Implementation Details dominates (50.0%), followed by Testing (14.1%) and Documentation (12.4%). Process-oriented categories drop sharply at observability: Build and Run falls from 8.8% of explicit-and-actionable to 2.9% of final.

Validation: Extraction precision 96.3%, recall 99.9%, F1-score 98.1%. Observability is the least reliably labeled criterion (85.0% accuracy, biased toward over-inclusion).

RQ₂ Results: Operationalization Rate

Overall rate: 62.9% (4,123 of 6,552 instructions).

Operationalization rate is consistent across agents (62.1%–63.5%). Exclusions are primarily due to correctness failures (57.3%) and lack of corroborating scripts (41.1%).

RQ₃ Results: Compliance

Coverage: Only 49.8% of operationalized instructions (2,053 of 4,123) ever meet an applicable commit; 18,594 instruction–commit pairs evaluated.

Overall compliance: Mean rate 59.8%, median 100.0%. Distribution is strongly bimodal:

• 54.4% (1,116) followed in every applicable commit
• 37.3% (766) followed in none
• Only 8.3% (171) partially followed

Category-level variation (significant, $H = 59.13, p < .001$):

• Development Process least compliant (mean 34.0%, median 0.0%), significantly lower than Build and Run, Documentation, and Implementation Details (but not robust to sensitivity check)
• Implementation Details (mean 63.6%) significantly higher than Testing (mean 49.2%)—the only category comparison robust at every evidence threshold
• Architecture (mean 38.2%) and Debugging (mean 21.9%) low but not statistically significant after correction

Agent-level variation (significant, $H = 41.58, p < .001$):

• Copilot significantly higher than Claude Code ($p_{Bonf} < .001$, $\delta = 0.15$)—robust at all thresholds
• Copilot vs. OpenAI Codex significant only in full sample ($p_{Bonf} = .018$); Codex sample is small (117)
• Claude Code and OpenAI Codex statistically indistinguishable at every threshold

Sensitivity analysis: The bimodal split persists (86.2% at extremes even with ≥5 commits). Implementation Details vs. Testing and Copilot vs. Claude Code gaps remain significant; Development Process gap and Copilot vs. Codex gap do not survive.

Theoretical and Practical Implications

For ACF Authors:

• A large share of ACF content (69.3%) is not even explicit; write for the agent, not the human reader—use imperative phrasing, concrete behaviors, avoid vague qualifiers.
• Prefer observable instructions whose compliance leaves artifact traces over process-only directives ("run the linter") that cannot be audited.
• Workflow-level conventions (Development Process guidance) may be better encoded in project tooling (commit hooks, CI configs) than in ACF prose.

For Engineering Agentic Systems:

• The bimodal compliance shape means spot-checking is insufficient; continuous, automated verification is needed, targeted at least reliable categories and agents.
• Verification mechanisms should be calibrated per ACF–agent pair; compliance differences across agents suggest reliability is a property of both the ACF and the agent together.
• Test-driven script generation with dual-bar acceptance demonstrates that artifact-grounded, per-instruction verification is practical today for a majority of observable ACF content—but ~30% of generated scripts still need human review before trust.

For Researchers:

• The single instruction, not the file or category, should be the unit of analysis; aggregate claims obscure the bimodal reality.
• The full pipeline (from 14,902 candidates to 2,053 evaluable instructions, only 13.8%) shows which guidance is verifiable at all.

Conclusion

This paper provides the first instruction-level assessment of whether coding agents follow ACF guidance in real-world development. The key findings are: (1) ACFs encode a much smaller proportion of explicit, actionable, and observable instructions (44%) than raw content suggests; (2) where compliance can be measured, agents follow ACF guidance more often than not (59.8% mean), but adherence is sharply bimodal—instructions are either consistently followed or consistently ignored; and (3) compliance varies meaningfully across instruction categories and coding agents, with Copilot showing higher adherence than Claude Code or Codex.

Limitations: Scope limited to three agents and public GitHub repositories; relies on LLM-based extraction with single-annotator validation; scripts receive full-file content but not line-level diffs; potential confounds in agent-level comparisons (category mix, repository differences).

Future work: Extend to additional agents and ACF conventions; investigate underlying causes of category- and agent-level compliance gaps; explore enforcement mechanisms that ensure ACF adherence at generation time rather than measuring it post-hoc.