# SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations

> SWE-CoWork shows frontier coding agents satisfy 63% of tests but complete under 8% of tasks, failing at sustained requirement tracking across organizational workflows.

- **Source:** [arXiv](https://arxiv.org/abs/2611.20218)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/W9tmBp
- **Whiteboard:** https://picx.dev/p/W9tmBp/image

## Summary

# SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations

## Summary (Overview)

- **New benchmark for organizational software engineering**: SWE-CoWork comprises 101 development assignments across 101 real repositories, 11 programming languages, and 12 application domains, where agents must recover requirements distributed across 26 digital employees and workplace records rather than receiving pre-scoped tasks.

- **Large gap between partial and complete delivery**: The two leading models (Claude-Opus-5.5 and GPT-6 Astra) satisfy ~63% of fail-to-pass (F2P) tests but complete fewer than 8% of assignments; the remaining nine models reach at most 39.09% F2P and 0.99% complete task success.

- **Providing complete requirements upfront helps partial but not complete success**: Oracle-Spec improves F2P by ~16 percentage points for Seed Evolving and GPT-6 Astra, yet complete task pass rates remain unchanged (0.99% and 7.92%).

- **Requirement-level trajectory analysis reveals failures throughout delivery**: Even Claude-Opus-5.5, which directs 93% of first questions to accountable owners, obtains only 64% of colleague-held requirements, abandons 153 of 450 deferred threads, and reports 31% of requirements as "done" that fail their tests.

- **Sustained requirement tracking identified as key challenge**: From clarification through implementation, verification, and handoff, agents fail to maintain coherent state of individual requirements across the full engineering workflow.

## Introduction and Theoretical Foundation

Real-world software development is inherently collaborative rather than an isolated coding exercise. Requirements originate from human stakeholders, evolve continuously, and require coordination across diverse roles where asking the right clarifying questions is indispensable. Developers must navigate project boards, versioned design documents, chat logs, merge requests, and CI histories—artifacts that frequently disagree with one another.

While LLM coding agents have driven standard benchmarks from single-digit scores to near saturation (with Anthropic reporting over 80% of merged code authored by Claude as of May 2026), existing benchmarks evaluate **isolated, pre-scoped task descriptions**. Current agents function as passive "spec executors" that require humans to curate organizational context and translate evolving stakeholder needs into self-contained specifications.

The paper identifies **four core organizational capabilities** required for agents to become true digital software engineers:

- **Continuous**: Interacting across workplace interfaces while maintaining coherent context across multiple sources and evolving revisions
- **Proactive**: Operating without step-by-step prompting from a brief handoff, identifying accountable owners while exercising restraint against scope distractors
- **Efficient**: Communicating within a bounded working budget and advancing multiple requirements in parallel
- **Persistent**: Remaining embedded in realistic environments over long horizons, tracking every requirement from clarification through implementation, verification, and handoff

## Methodology

### Digital Employees

Each task places an agent in an organization of 26 modeled colleagues (2,626 employee profiles across the benchmark). Each digital employee has three properties:

1. **Responsibility**: The module or decision for which the employee is accountable
2. **Local knowledge**: Which requirements the employee can explain and which require referral
3. **Collaboration behavior**: Governed by rules (e.g., requesting reproduction, addressing one topic at a time, deferring answers)

Colleague interactions are controlled by fixed knowledge boundaries and disclosure rules, with a language model generating responses. The agent interacts through a dedicated tool interface for searching staff directories, inspecting project boards, opening versioned documents, searching chat, messaging colleagues, and checking CI records.

### Task Composition

- **101 tasks** from 101 upstream repositories across 11 languages and 12 domains
- Median solution: **2,955 production lines across 60 files**
- **3,261 executable checks total**: 2,699 F2P and 562 P2P test nodes
- Workplace records include 10,329 board cards, 5,097 versioned documents, 40,836 chat messages, 1,194 merge requests

### Scoring Metrics

For task $i$, let $f_i$ and $p_i$ denote passing F2P and P2P test nodes, with totals $F_i$ and $P_i$:

$$
\mathrm{F2P} = 100 \frac{\sum_{i=1}^{N} f_i}{\sum_{i=1}^{N} F_i}
$$

$$
\mathrm{P2P} = 100 \frac{\sum_{i=1}^{N} p_i}{\sum_{i=1}^{N} P_i}
$$

$$
\mathrm{Passrate} = \frac{100}{N} \sum_{i=1}^{N} \mathbf{1}[f_i = F_i \land p_i = P_i]
$$

### Experimental Design

- **CoWork evaluation**: Eleven models at max effort (Claude-Opus-5.5, GPT-6 Astra, Seed 2.1 Pro, Seed Evolving, Kimi K3, DeepSeek V4 Pro, DeepSeek V4.1 Flash, GLM 5.3, GLM 5.2, Qwen3.8-Max, Qwen3.8-Flash)
- **Oracle-Spec comparison**: Complete requirements provided upfront for Seed Evolving and GPT-6 Astra, holding repository, tests, scaffold, and budget constant
- **Trajectory analysis**: Claude-Opus-5.5 runs analyzed via programmatic disclosure logs and rubric-based LLM trajectory review

### Workplace Information Challenges

Five situations requiring engineering judgment: **late evidence** (new traces arriving mid-implementation), **superseded conclusions** (earlier proposals remaining searchable after corrections), **ambiguous provenance** (logs mislabeled), **scope distractors** (canceled cards and neighboring projects), and **fragmented ownership** (referrals between colleagues).

## Empirical Validation / Results

### Cross-Model Results (Table 2)

| Model | F2P (%) | P2P (%) | Pass rate (%) |
|---|---|---|---|
| **Claude-Opus-5.5** | **63.84** | 93.24 | 6.93 |
| **GPT-6 Astra** | **62.91** | 89.15 | 7.92 |
| Seed Evolving | 39.09 | 94.48 | 0.99 |
| Qwen3.8-Max | 35.70 | 91.80 | 0.00 |
| GLM 5.3 | 31.27 | 87.72 | 0.99 |
| Qwen3.8-Flash | 30.70 | 93.40 | 0.00 |
| DeepSeek V4.1 Flash | 28.20 | 94.30 | 0.00 |
| Kimi K3 | 27.08 | 84.88 | 0.99 |
| GLM 5.2 | 24.40 | 95.20 | 0.00 |
| Seed 2.1 Pro | 21.01 | 81.85 | 0.00 |
| DeepSeek V4 Pro | 13.04 | 90.21 | 0.00 |

### Key Findings

**Requirement acquisition tracks delivery**: Spearman $\rho = 0.94$ between F2P and share of colleague-held requirements obtained. GPT-6 Astra and Claude-Opus-5.5 obtain 67.8% and 64.2% respectively; DeepSeek V4 Pro obtains only 1.8%.

**Tool calls don't predict success**: Spearman $\rho = 0.08$ between tool-call counts and F2P. More frequent colleague communication ($\rho = 0.71$) and earlier first code edit ($\rho = -0.62$) associate with higher F2P.

### Oracle-Spec Comparison (Table 3)

| Metric | Seed Evolving CoWork | Seed Evolving Oracle | Gain | GPT-6 Astra CoWork | GPT-6 Astra Oracle | Gain |
|---|---|---|---|---|---|---|
| F2P | 39.09 | 55.76 | +16.67 | 62.91 | 78.84 | +15.93 |
| P2P | 94.48 | 89.50 | -4.98 | 89.15 | 91.81 | +2.66 |
| Pass rate | 0.99 | 0.99 | +0.00 | 7.92 | 7.92 | +0.00 |

Even with complete requirements, Seed Evolving leaves 44% of F2P test nodes unsatisfied and GPT-6 Astra 21%.

### Claude-Opus-5.5 Trajectory Analysis

**Clarification failures**:
- 93% of first questions directed to accountable owners, yet only 64% of 2,176 colleague-held requirements obtained
- 16% raised without disclosure, 17% unraised with already-contacted owners, 3% belong to never-contacted owners
- Among failed F2P test nodes with linked requirements: 56% reached the agent but remained unsatisfied (implementation gap), 42% never obtained through clarification

**Deferred questions abandoned**: 153 of 450 deferred threads receive no follow-up, with a median of 7.1 hours of execution budget remaining at deferral. The agent continues working for a median of 3.4 hours afterward.

**Handoff discrepancies**: Among 826 requirements reported as completed, 255 (31%) fail their corresponding tests. The trajectory reviewer identifies 49 premature stops with actionable next steps remaining.

## Theoretical and Practical Implications

### Benchmark Design Implications

SWE-CoWork shifts evaluation from isolated spec execution to autonomous collaboration inside software organizations. Unlike prior benchmarks (DeepSWE, SWE Atlas, FrontierSWE v2, CursorBench 4.0, Real-SWE, RealSWE, τ^T-Bench), it requires agents to establish requirements from organizational records and colleagues before implementing changes, with each requirement linked to a fixed executable test contract.

### Capability Assessment

The paired CoWork/Oracle-Spec design distinguishes **organizational overhead** (cost of acquiring requirements through colleagues) from **implementation bottlenecks** (failures persisting even with complete requirements). The ~16-point F2P gain from Oracle-Spec quantifies organizational overhead, while the unchanged pass rates reveal implementation/verification as equally critical bottlenecks.

### Behavioral Insights

- Weaker models exhibit passive spec-executor behavior (DeepSeek V4 Pro sends a median of 0 colleague messages per run)
- Frontier models find the right people but fail at **sustained requirement tracking**: unraised cards, abandoned deferrals, unsatisfied disclosed requirements, premature stops, and completion claims contradicted by tests
- Evidence-to-code translation shows the strongest association with F2P ($\rho = +0.36$), followed by causal debugging and verification design (both $\rho = +0.30$)

## Conclusion

SWE-CoWork demonstrates that reliable autonomous software engineering requires **sustained requirement tracking**—maintaining the state of each requirement from initial clarification through implementation, verification, and handoff. The substantial gap between partial implementation (~63% F2P) and complete task delivery (<8%) highlights that current agents cannot reliably deliver entire assignments even when they can implement individual behaviors.

Future directions include:
- Developing agents that explicitly track each requirement's accountable owner, unresolved questions, implementation status, verification evidence, and handoff claims
- Extending the benchmark to evaluate **teams of autonomous programming agents**
- Investigating whether explicit requirement-tracking mechanisms improve end-to-end delivery

The benchmark provides a foundation for studying these capabilities, with the finding that requirements remain unsatisfied at multiple stages—clarification, implementation, verification, and reporting—suggesting that continuity across the entire engineering workflow is the central challenge for autonomous software engineering.

---

_Markdown view of https://picx.dev/p/W9tmBp, served by PicX — AI-generated visual whiteboard summaries of research papers._
