# SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving

> SWE-Serve reveals a production correctness gap where removing end-to-end serving tests boosts agent benchmark pass rates by 23.4 points, exposing that local success often fails in real inference systems.

- **Source:** [arXiv](https://arxiv.org/abs/2609.26777)
- **Published:** 2026-09-26
- **Permalink:** https://picx.dev/p/zHxYf8
- **Whiteboard:** https://picx.dev/p/zHxYf8/image

## Summary

## Summary (Overview)

- **SWE-Serve** is a new benchmark for evaluating AI agents on **production inference engineering** tasks, built from 53 repository-grounded tasks derived from recent production changes to SGLang, an open-source inference-serving system.
- The benchmark spans **six engineering families** (model enablement, decoding, kernels/quantization, caching, distributed execution, serving APIs) and executes on either CPU or a single H100 GPU with hidden functional and regression tests.
- A key finding is the **"production correctness gap"**: removing model-serving end-to-end (E2E) tests from scoring raises pass rates by **23.4 percentage points** (from 45.9% to 69.4%), demonstrating that agents often complete tasks locally without achieving production correctness.
- Across 11 models and 31 model–effort configurations, the best-performing configuration achieves **75% mean pass@1**, with a 40-point spread between the top and bottom models.
- The benchmark introduces a rigorous **validity and integrity methodology** including no-op/oracle controls, adversarial probing, human adjudication, and closed-book execution.

---

## Introduction and Theoretical Foundation

The paper addresses a critical gap in agentic coding evaluation: **repository-scale benchmarks do not necessarily test whether a locally working implementation remains correct across the production path**. In production inference systems, a change can work in isolation yet fail when exercised through the complete system.

**Key motivation example**: On the Gemma 4 mixture-of-experts (MoE) core-serving task, 48.5% of agent-created patches (16 of 33) passed every test except the end-to-end tests but failed to serve the specified model correctly through the live server's public interfaces.

The paper defines **production correctness** as: *satisfying a task's behavioral requirements within the inference system, including preserving existing functionality and meeting any task-specific performance constraints.*

**Related work gaps**:
- **SWE-bench Verified, SWE-bench Pro, DeepSWE**: Repository-level benchmarks but not focused on inference engineering or GPU-dependent execution.
- **Terminal-Bench 3**: Only 3 of 74 tasks (4.1%) concern inference.
- **KernelBench, SOL-ExecBench, InferenceBench**: Focus on isolated kernel generation or performance optimization, not repository-scale production features.
- **ISO-Bench**: Only inference optimization at repository level, drawn exclusively from performance changes.

---

## Methodology

### Task Formulation

Each SWE-Serve task includes:
- A task instruction
- A sandboxed execution environment specification
- An oracle solution (used only for qualification)
- A hidden executable verifier

**Scoring**: Following the SWE-bench standard, the verifier assesses behavior through:
- **Fail-to-pass (F2P) tests**: require behavior absent at the base commit
- **Pass-to-pass (P2P) tests**: protect relevant existing behavior
- **Model-serving E2E tests**: launch a standalone serving process and assess behavior through public interfaces (19 tasks)
- **Calibrated performance gates**: enforce performance constraints (3 tasks)

### Task Construction and Qualification

- **Discovery**: Five paths (release-note scans, manual review, maintainer roadmap issues, SGLang team suggestions) screened 786 task-source records → 203 source candidates.
- **Construction**: 156 task candidates → **53 admitted** (34.0% admission rate).
- **Qualification controls**:
  - No-op control must fail every F2P and pass every P2P test
  - Oracle must pass all tests
  - Agent-assisted adversarial probing for verifier vulnerabilities
  - Human adjudication
  - Closed-book execution with trajectory auditing

### Experimental Setup

- **Harness**: mini-SWE-agent v2.4.3, model-agnostic
- **Limits**: 350 steps or 210 minutes per task; 120-second command timeout
- **Models**: 11 models including Claude Opus 5, GPT-5.6 Sol/Luna/Terra, Kimi K3, DeepSeek V4 Flash, GLM-5.2, Gemini 3.6 Flash, Laguna S 2.1, Inkling S
- **Configurations**: 31 model–effort configurations, each evaluated 3 times (K=3)

---

## Empirical Validation / Results

### Primary Leaderboard Results

| Model | Reasoning effort | pass@1 (95% CI) | pass^3 | Mean cost | Mean tokens | Mean steps | Mean time (min) |
|---|---|---|---|---|---|---|---|
| Claude Opus 5 | max | 75% ± 4% | 70% | $17.40 | 122k | 134 | 57.5 |
| GPT-5.6 Sol | max | 75% ± 6% | 68% | $12.26 | 52k | 63 | 29.5 |
| Claude Sonnet 5 | xhigh | 64% ± 4% | 51% | $6.61 | 98k | 148 | 40.6 |
| Kimi K3 | max | 64% ± 6% | 51% | $7.24 | 79k | 117 | 99.9 |
| GPT-5.6 Luna | max | 64% ± 4% | 51% | $0.95 | 65k | 110 | 28.9 |
| GPT-5.6 Terra | max | 64% ± 4% | 55% | $5.06 | 61k | 82 | 25.5 |
| DeepSeek V4 Flash | max | 55% ± 5% | 42% | $0.69 | 99k | 173 | 36.4 |
| GLM-5.2 | max | 48% ± 2% | 34% | $2.10 | 53k | 101 | 34.0 |
| Gemini 3.6 Flash | high | 48% ± 7% | 30% | $4.84 | 99k | 92 | 37.3 |
| Laguna S 2.1 | max | 46% ± 5% | 26% | $0.33 | 147k | 184 | 56.9 |
| Inkling S | xhigh | 35% ± 3% | 25% | $0.44 | 36k | 96 | 17.6 |

### The Production Correctness Gap

**Key result**: Removing E2E tests from scoring (while keeping patches and all other tests unchanged):
- Pass rate increases from **45.9% to 69.4%** (+23.4 percentage points)
- Pass rate increases for **all 11 top-per-model configurations**

**Matched test-removal control**: Across 12 eligible tasks with 10,000 randomized pairs:
- E2E test removal produces **16.1 fail-to-pass transitions** vs. **8.0** for matched non-E2E removal (2.0× difference)
- E2E removal produces more fail-to-pass transitions in 92.2% of randomized pairs

### Task Property Analysis

- **Runtime-domain breadth**: Single-domain tasks: 69.0% pass rate vs. multi-domain: 47.7% (21.3-point gap, consistent across all configurations)
- **Persistent state**: 20.8-point lower pass rate on GPU tasks requiring persistent state
- **Concurrent coordination**: 29.0-point lower pass rate on the six tasks testing concurrent coordination

### Harness Sensitivity

Switching to model-specific harnesses **did not improve** performance:
- mini-SWE-agent: 75.5% for both GPT-5.6 Sol and Claude Opus 5
- Codex (GPT-5.6 Sol): 73.6%
- Claude Code (Opus 5): 69.8%

---

## Theoretical and Practical Implications

1. **Local correctness ≠ production correctness**: The benchmark provides direct, executable evidence that agents frequently satisfy component/integration tests yet fail end-to-end serving requirements. This is a critical insight for the field.

2. **Resource-efficiency trade-offs**: Among configurations tied at 64% pass@1, mean per-task cost varies by **7.6×** ($0.95–$7.24) and wall-clock time by **3.9×** (25.5–99.9 minutes). Optimizing for pass@1 alone can incur substantially higher costs for marginal gains (e.g., Claude Opus 5: high→max effort raises pass@1 from 74% to 75% but increases cost by 76%).

3. **Repeated-run reliability matters**: Configurations with identical pass@1 can differ substantially in pass@3 (retry coverage) and pass^3 (consistency), which is critical for practical deployment decisions.

4. **Task complexity dimensions**: Multi-runtime-domain tasks, persistent state, and concurrent coordination are reliable predictors of lower agent performance, suggesting these are key areas for future improvement.

5. **Benchmark methodology**: The validity/integrity framework (no-op/oracle controls, adversarial probing, closed-book execution, canary strings) provides a template for future benchmark construction.

---

## Conclusion

SWE-Serve is a production inference engineering benchmark that enables direct measurement of whether agents achieve **production correctness** rather than mere local task completion. Key takeaways:

- **Best performance**: 75% mean pass@1 (Claude Opus 5, GPT-5.6 Sol at max effort)
- **Production gap is substantial and consistent**: 23.4-point pass-rate increase when E2E tests are removed, affecting all 11 top configurations
- **Task properties matter**: Multi-domain, persistent-state, and concurrent-coordination tasks are significantly harder

**Future directions**:
- Extending executable coverage beyond SGLang to other inference systems (vLLM, TensorRT-LLM, Triton)
- Multi-GPU and multi-node tasks (parallelism, disaggregation, distributed coordination)
- A private held-out task set as a complement to canary strings
- Tracking whether future agents close the gap between local completion and production correctness

---

_Markdown view of https://picx.dev/p/zHxYf8, served by PicX — AI-generated visual whiteboard summaries of research papers._