Summary (Overview)

• SWE-Serve is a new benchmark for evaluating AI agents on production inference engineering tasks, built from 53 repository-grounded tasks derived from recent production changes to SGLang, an open-source inference-serving system.
• The benchmark spans six engineering families (model enablement, decoding, kernels/quantization, caching, distributed execution, serving APIs) and executes on either CPU or a single H100 GPU with hidden functional and regression tests.
• A key finding is the "production correctness gap": removing model-serving end-to-end (E2E) tests from scoring raises pass rates by 23.4 percentage points (from 45.9% to 69.4%), demonstrating that agents often complete tasks locally without achieving production correctness.
• Across 11 models and 31 model–effort configurations, the best-performing configuration achieves 75% mean pass@1, with a 40-point spread between the top and bottom models.
• The benchmark introduces a rigorous validity and integrity methodology including no-op/oracle controls, adversarial probing, human adjudication, and closed-book execution.

---

Introduction and Theoretical Foundation

The paper addresses a critical gap in agentic coding evaluation: repository-scale benchmarks do not necessarily test whether a locally working implementation remains correct across the production path. In production inference systems, a change can work in isolation yet fail when exercised through the complete system.

Key motivation example: On the Gemma 4 mixture-of-experts (MoE) core-serving task, 48.5% of agent-created patches (16 of 33) passed every test except the end-to-end tests but failed to serve the specified model correctly through the live server's public interfaces.

The paper defines production correctness as: satisfying a task's behavioral requirements within the inference system, including preserving existing functionality and meeting any task-specific performance constraints.

Related work gaps:

• SWE-bench Verified, SWE-bench Pro, DeepSWE: Repository-level benchmarks but not focused on inference engineering or GPU-dependent execution.
• Terminal-Bench 3: Only 3 of 74 tasks (4.1%) concern inference.
• KernelBench, SOL-ExecBench, InferenceBench: Focus on isolated kernel generation or performance optimization, not repository-scale production features.
• ISO-Bench: Only inference optimization at repository level, drawn exclusively from performance changes.

---

Methodology

Task Formulation

Each SWE-Serve task includes:

• A task instruction
• A sandboxed execution environment specification
• An oracle solution (used only for qualification)
• A hidden executable verifier

Scoring: Following the SWE-bench standard, the verifier assesses behavior through:

• Fail-to-pass (F2P) tests: require behavior absent at the base commit
• Pass-to-pass (P2P) tests: protect relevant existing behavior
• Model-serving E2E tests: launch a standalone serving process and assess behavior through public interfaces (19 tasks)
• Calibrated performance gates: enforce performance constraints (3 tasks)

Task Construction and Qualification

• Discovery: Five paths (release-note scans, manual review, maintainer roadmap issues, SGLang team suggestions) screened 786 task-source records → 203 source candidates.
• Construction: 156 task candidates → 53 admitted (34.0% admission rate).
• Qualification controls:
  • No-op control must fail every F2P and pass every P2P test
  • Oracle must pass all tests
  • Agent-assisted adversarial probing for verifier vulnerabilities
  • Human adjudication
  • Closed-book execution with trajectory auditing

Experimental Setup

• Harness: mini-SWE-agent v2.4.3, model-agnostic
• Limits: 350 steps or 210 minutes per task; 120-second command timeout
• Models: 11 models including Claude Opus 5, GPT-5.6 Sol/Luna/Terra, Kimi K3, DeepSeek V4 Flash, GLM-5.2, Gemini 3.6 Flash, Laguna S 2.1, Inkling S
• Configurations: 31 model–effort configurations, each evaluated 3 times (K=3)

---

Empirical Validation / Results

Primary Leaderboard Results

The Production Correctness Gap

Key result: Removing E2E tests from scoring (while keeping patches and all other tests unchanged):

• Pass rate increases from 45.9% to 69.4% (+23.4 percentage points)
• Pass rate increases for all 11 top-per-model configurations

Matched test-removal control: Across 12 eligible tasks with 10,000 randomized pairs:

• E2E test removal produces 16.1 fail-to-pass transitions vs. 8.0 for matched non-E2E removal (2.0× difference)
• E2E removal produces more fail-to-pass transitions in 92.2% of randomized pairs

Task Property Analysis

• Runtime-domain breadth: Single-domain tasks: 69.0% pass rate vs. multi-domain: 47.7% (21.3-point gap, consistent across all configurations)
• Persistent state: 20.8-point lower pass rate on GPU tasks requiring persistent state
• Concurrent coordination: 29.0-point lower pass rate on the six tasks testing concurrent coordination

Harness Sensitivity

Switching to model-specific harnesses did not improve performance:

• mini-SWE-agent: 75.5% for both GPT-5.6 Sol and Claude Opus 5
• Codex (GPT-5.6 Sol): 73.6%
• Claude Code (Opus 5): 69.8%

---

Theoretical and Practical Implications

1. Local correctness ≠ production correctness: The benchmark provides direct, executable evidence that agents frequently satisfy component/integration tests yet fail end-to-end serving requirements. This is a critical insight for the field.

2. Resource-efficiency trade-offs: Among configurations tied at 64% pass@1, mean per-task cost varies by 7.6× ($0.95–$7.24) and wall-clock time by 3.9× (25.5–99.9 minutes). Optimizing for pass@1 alone can incur substantially higher costs for marginal gains (e.g., Claude Opus 5: high→max effort raises pass@1 from 74% to 75% but increases cost by 76%).

3. Repeated-run reliability matters: Configurations with identical pass@1 can differ substantially in pass@3 (retry coverage) and pass^3 (consistency), which is critical for practical deployment decisions.

4. Task complexity dimensions: Multi-runtime-domain tasks, persistent state, and concurrent coordination are reliable predictors of lower agent performance, suggesting these are key areas for future improvement.

5. Benchmark methodology: The validity/integrity framework (no-op/oracle controls, adversarial probing, closed-book execution, canary strings) provides a template for future benchmark construction.

---

Conclusion

SWE-Serve is a production inference engineering benchmark that enables direct measurement of whether agents achieve production correctness rather than mere local task completion. Key takeaways:

• Best performance: 75% mean pass@1 (Claude Opus 5, GPT-5.6 Sol at max effort)
• Production gap is substantial and consistent: 23.4-point pass-rate increase when E2E tests are removed, affecting all 11 top configurations
• Task properties matter: Multi-domain, persistent-state, and concurrent-coordination tasks are significantly harder

Future directions:

• Extending executable coverage beyond SGLang to other inference systems (vLLM, TensorRT-LLM, Triton)
• Multi-GPU and multi-node tasks (parallelism, disaggregation, distributed coordination)
• A private held-out task set as a complement to canary strings
• Tracking whether future agents close the gap between local completion and production correctness