# ViBench 2.0: A Benchmark for Long-Horizon Software Building

> ViBench 2.0 reveals a reliability gap: top coding agents pass up to 98% of test plans yet deliver a fully working product in at most 65% of long-horizon builds.

- **Source:** [arXiv](https://arxiv.org/abs/2611.05620)
- **Published:** 2026-11-07
- **Permalink:** https://picx.dev/p/T0rh5u
- **Whiteboard:** https://picx.dev/p/T0rh5u/image

## Summary

# ViBench 2.0: A Benchmark for Long-Horizon Software Building

## Summary (Overview)

- **ViBench 2.0** is a benchmark designed to evaluate whether AI coding agents can build complete, fully-functional software products over long horizons, with agents receiving feature requests one at a time in a single conversation, starting from an empty project.
- The benchmark comprises **17 full-stack web products** (modeled on real software like Amazon, Asana, Discord, Figma, GitHub, Google Docs, Jira, and Uber) grown over **241 feature requests**, judged only when finished through **544 test plans** executed by a browser agent.
- Key finding: a **reliability gap** exists—the three strongest models pass up to 98% of test plans yet deliver a fully working product in at most **65% of builds**, and no open-weight model delivers a single fully working product.
- The benchmark shows rapid progress: across three releases of one model family, pass@1 on public products rises from **3% to 53%** while cost per build falls by more than half.
- Eight products are released publicly with the harness and grader; nine are held out to guard against training contamination.

## Introduction and Theoretical Foundation

### Background and Motivation

The paper addresses a fundamental shift in software engineering: **agentic engineering**, where users describe software in natural language and a coding agent builds and extends it one request at a time. This practice, which has matured from "vibe coding" (Karpathy, 2025), now involves agents that can sustain long-horizon work for hours or days.

### The Benchmark Saturation Problem

> Over 40% of benchmarks less than two years old have already saturated (Akhtar et al., 2026).

Existing coding benchmarks suffer from critical limitations:
- **Single-task benchmarks** (e.g., SWE-bench, ViBench) give agents one fully-specified task and are nearing saturation—GPT-6.1 Sol reaches 82–89% pass@1 on ViBench's Vibe-on-Vibe tasks
- **Multi-turn benchmarks** score each step as completed, rather than evaluating the final product as a whole
- **Vibe Code Bench 1-100** grows an app over up to ten requests but tops out at 34.0% and starts from a finished reference app, measuring progress rather than delivery

### Theoretical Position

The paper argues that the question users actually face is: **whether a product built from nothing still works as a whole after the last request**. Such a product can only be checked through its interface, the way users who never read the code judge it. A build should count only if *every* test plan passes.

## Methodology

### Task Formulation

The benchmark defines a rigorous pipeline (Figure 1):

**Building Phase:**
- The builder agent starts from an empty project and receives the PRD for an MVP
- After each stage, it receives the next feature's PRD in the **same conversation**
- Each stage is limited to **300 agent steps and 2 hours**
- Beyond 200k tokens, the agent's own model summarizes older conversation parts
- The agent works in a container with a PostgreSQL database and must maintain setup/start scripts

**Evaluation Phase:**
- No grading until the last stage is complete
- Test plans check only what a user of the finished product would see
- Plans grade outcomes rather than specific designs (wording, layout, placement are left to the agent)
- Every seed and grade starts a **fresh container** from final code with an **empty database**
- The grader operates a real Chromium browser through Playwright, acting as a computer-use agent

### Task Construction

**Products:** Each of the 17 tasks is modeled on real software with features that the real software ships, as described in public help pages. Real products impose requirements beyond primary paths (e.g., a chat message must reach everyone without reload on Discord; split fares must divide to the cent on Uber).

**PRDs:** Specify only business-facing content (user flows, roles, business rules, observable states), omitting all technical details (data models, APIs, schemas, libraries). Features are ordered along a dependency graph so each arrives after those it builds on.

**Test Plans:** Sequence of steps executed as a user would, with 85% of plans exercising multiple features together. Each behavior is graded by exactly one plan to prevent a single bug from costing more than one step.

### Verification Protocol

Four criteria for task verification:

1. **Specificity:** A plan passes iff the product behaves as PRDs require, tracing to explicit PRD statements or five basics (app starts on empty DB, saved data persists, no false save confirmations, one click does one thing, refused actions show messages)
2. **Solvability:** Every product has been built to pass all its plans
3. **Integrity:** No shortcuts possible—PRDs enter the container only at their own stage, builders never see test plans, no web-search or browser tools, canary strings for training-data filtering
4. **Grader Accuracy:** An all-or-nothing score makes wrong fails costly, so a failing plan is re-graded up to four more times until three grades agree

$$P(\text{wrong fail per plan}) = 0.2\% \text{ for a single grade; majority of five removes such errors}$$

## Empirical Validation / Results

### Main Results (Table 1)

| Model | pass@1 | Plan pass | Min. | Cost |
|-------|--------|-----------|------|------|
| GPT-6.1 Sol | $64.7 \pm 9.3$ | $96.9 \pm 1.2$ | 178 | \$12.3 |
| GPT-6 Astra | $60.3 \pm 10.4$ | $98.0 \pm 0.6$ | 198 | \$89.9 |
| Opus 5.5 | $48.5 \pm 10.9$ | $93.7 \pm 4.2$ | 73 | \$23.1 |
| Sonnet 5.5 | $19.1 \pm 9.8$ | $79.2 \pm 7.3$ | 26 | \$4.4 |
| Grok 4.7 | $1.5 \text{ (0–7.9)}$ | $52.9 \pm 8.9$ | 118 | \$63.9 |
| DeepSeek V4.1 Flash | $0.0 \text{ (0–5.3)}$ | $77.4 \pm 3.2$ | 221 | \$2.5 |
| GLM 5.3 | $0.0 \text{ (0–5.3)}$ | $66.0 \pm 4.9$ | 231 | \$27.8 |
| Kimi K3 | $0.0 \text{ (0–5.3)}$ | $56.7 \pm 8.1$ | 221 | \$33.1 |
| Gemini 3.8 Flash | $0.0 \text{ (0–5.3)}$ | $38.1 \pm 5.6$ | 249 | \$27.0 |

**Key findings:**
- The three strongest models pass 93.7–98.0% of plans but deliver fully working products in only 48.5–64.7% of builds
- Of 84 failed seeded builds of top models, **42 fail exactly one plan** and **62 fail at most two**
- Astra has at least one passing build on 16/17 products; Sol on 15; Opus on 13
- No open-weight model passes any of their 204 builds
- The Figma-modeled product fails for all 36 independent builds

### Model Performance over Versions (Table 2)

| Model | pass@1 | Plan pass | Min. | Cost |
|-------|--------|-----------|------|------|
| GPT-6.1 Sol* | 53.1 | 94.9 | 170 | \$11.8 |
| GPT-6.1 Terra | 46.9 | 94.5 | 122 | \$6.2 |
| GPT-6 Sol | 31.2 | 84.0 | 88 | \$19.4 |
| GPT-5.6 Sol | 3.1 | 67.5 | 73 | \$26.5 |
| GPT-6 Luna | 0.0 | 59.8 | 188 | \$1.7 |
| GPT-5.5 | 0.0 | 51.5 | 46 | \$30.1 |
| GPT-5.4 | 0.0 | 41.5 | 78 | \$22.9 |
| GPT-5.6 Terra | 0.0 | 26.1 | 39 | \$5.3 |
| GPT-5.6 Luna | 0.0 | 6.6 | 35 | \$0.5 |

**Pattern:** The plan pass rate rises before pass@1; pass@1 only rises once plan pass rate exceeds ~70%. GPT-6.1 Sol costs \$11.8 per build, down from \$30.1 peak for GPT-5.5.

### Failure Analysis

- Failures are **not concentrated where features interact** (6.4% vs 6.2% for Opus)
- Difficulty lies in **breadth**: with 93.7% plan pass rate, fewer than half of Opus's builds pass entirely
- Complete failures are rare (2/204 top-model builds), caused by seeding failures from schema initialization bugs
- **Visual quality gap**: Sonnet produces plain, list-like pages versus polished product-like pages from top models; Sonnet writes half the output tokens (0.20M vs 0.36–0.39M)

## Theoretical and Practical Implications

### Deployment Lessons from Production Use

Replit uses ViBench 2.0 for deployment decisions, yielding six key lessons:

1. **Two metrics answer different questions**: highest pass@1 models serve the core loop; cheaper models with close plan-pass rates serve specialized workflows
2. **Context handoff matters over long horizons**: compaction is a first-order concern—what survives determines what later requests can build on
3. **The plan pass rate moves first**: it rises before pass@1, serving as an earlier signal
4. **Platforms should check deployments**: agents don't reliably test that their product deploys successfully
5. **Fair, difficult tests need repeated review**: about a fifth of draft plans required edits without loosening checks
6. **Both kinds of grading error must be measured**: re-grading removes wrong fails but not wrong passes

### Grading Protocol Validation

- **Wrong fails** are infrequent and seldom recur: single grade fails working plans 0.2% of the time; majority of five corrects
- **Wrong passes** measured with a second grader: only 4 of 477 passed plans (0.8%) were false
- **Planted 34 bugs**, 24 subtle (e.g., total off by a cent); the grader caught every one
- Audit of 178 failed plans found 13 wrong verdicts, all fixed before release

## Conclusion

ViBench 2.0 measures whether an agent can build a fully working product over a long horizon. The benchmark reveals that while strongest models implement nearly every feature, at least a third of their builds still fail one or more plans. However, progress is rapid: across three Sol releases, pass@1 on public products rises from 3% to 53% while cost per build falls from \$27 to \$12.

**Future directions:**
- Expect open-weight models to follow the same trajectory
- Continue adding products to keep the benchmark unsaturated
- Developing a lighter benchmark variant to reduce grading costs (~\$1,440 per full round currently)
- The remaining reliability gap serves as an engineering target for production agents

### Key Limitations

- **Contamination**: 9 products held out under NDA; public products carry canary strings for filtering
- **Grading**: passed plans aren't re-checked; wrong passes estimated at 0.8%
- **Scope**: covers only full-stack web products with PostgreSQL, English specifications, functional correctness (not visual design, speed, or cost)
- **Variance**: ±10 point 95% intervals on pass@1 with 4 builds per product; plan pass rate offers finer comparison (±1 point)
- **Reproducibility**: public subset is a faithful proxy (Spearman ρ = 0.99) but a harder one, with top models scoring 7–12 points lower

---

_Markdown view of https://picx.dev/p/T0rh5u, served by PicX — AI-generated visual whiteboard summaries of research papers._