Summary (Overview)

• FDE-Gym is a benchmark of 95 end-to-end customer delivery tasks from 20 customer scenarios across 18 sectors, each adapted by industry experts from real-world delivery projects. It evaluates agents on their ability to deliver working solutions that measurably improve a customer's business metric.
• The benchmark addresses the "last mile" of AI delivery: forward-deployed engineers (FDEs) must clarify ambiguous requirements, work in live systems, and uncover unstated constraints—challenges not captured by existing benchmarks that assume well-defined problems.
• Key finding: The best model configuration (Claude Opus 5.5) achieves a mean FDE-Gym score of only 0.56 and resolves 14 of 95 tasks, while five of thirteen configurations resolve none. This reveals a substantial gap between solving well-specified tasks and delivering effective AI systems for real customers.
• Primary bottleneck: Discovering unstated constraints is the largest source of performance loss. When rules and constraints are provided (L0), leading models score 0.73–0.82; requiring discovery drops scores to 0.39–0.62. However, even obtaining all key facts does not ensure effective delivery—leading models achieve only 0.60–0.70 of the reference uplift.
• Practical implication: Weak models frequently leave customers worse off than the do-nothing baseline—Claude Haiku 4.5 scores below zero on 79 of 95 tasks, meaning delivering nothing would have been better.

Introduction and Theoretical Foundation

The paper addresses a fundamental gap in AI agent evaluation. Existing benchmarks—from software engineering (SWE-bench, Terminal-Bench) to enterprise solutions (TheAgentCompany) and clarification benchmarks (XYEval)—all assume the problem is already well-defined. Real-world customer delivery, however, requires forward-deployed engineers (FDEs) to:

• Identify what will actually help the customer, not just what the sponsor requests
• Work with live production systems and uncover unstated constraints
• Navigate stakeholder relationships with limited patience and hidden costs

The motivating example is a drilling operator's vice president requesting a daily summary email and "Ask your wells" chat box, while explicitly excluding the downtime-coding system from scope. A literal solution passes every functional check but fails to move the customer's business metric—because the actual bottleneck lies in monthly downtime credit claims caused by a flawed keyword-based script in the system the sponsor excluded.

The paper introduces four task instruction levels (L0–L3) that isolate distinct delivery challenges:

• L0: Problem, rules, and scoring fully specified (tests implementation)
• L1: Problem stated, but constraints must be discovered (tests constraint discovery)
• L2: Vague but directionally correct request (tests problem clarification)
• L3: Sponsor's original request, which may be misdirected (tests problem framing)

The theoretical foundation builds on the observation that delivery quality must be measured by customer outcomes, not by task completion.

Methodology

Task Construction

Experts adapted real-world delivery projects (China, Singapore, USA, since 2024) into 20 scenarios across 18 sectors. Each scenario yields an original task plus 2–7 variants that alter key facts (which rule is outdated, who holds critical information, whether the request is misdirected), yielding 95 tasks total. This design prevents reliance on memorized solutions.

Environment and Stakeholder Simulation

Agents work in a workspace with:

• Task instruction, production system copy, documents, historical data, staff directory
• Python and SQLite runtime; network access restricted to company LLM gateway
• 130+ scripted stakeholders with fixed knowledge, misconceptions, patience limits, and contact restrictions (scripted for reproducibility, avoiding LLM-as-user variance)

Scoring Metrics

The core evaluation uses the Normalized Business Uplift:

$$U = \frac {\mathrm{KPI} _ {\text { delivered }} - \mathrm{KPI} _ {\text { baseline }}}{\mathrm{KPI} _ {\text { reference }} - \mathrm{KPI} _ {\text { baseline }}}$$

where KPI_baseline, KPI_delivered, and KPI_reference are the customer's business metric for the unchanged, delivered, and reference systems respectively. U = 0 means no improvement, U = 1 matches the reference, U < 0 means worse than doing nothing.

Incident Adjustment (zeroes positive uplift if operational incidents occur):

$$U _ {\text { adj }} = \left\{ \begin{array}{l l} U, & \text { if no incident occurs }, \\ \min (U, 0), & \text { otherwise }. \end{array} \right.$$

Stakeholder Contact Cost (charges for messages, trial runs, waiting time):

$$C = \sum_ {k \in K} c _ {k}$$

FDE-Gym Score:

$$S = U _ {\mathrm{adj}} - C$$

A task is resolved if S ≥ 0.8 with no incident. The Average FDE-Gym Score floors negative incident-adjusted uplift at -1 for aggregation.

Admission Anchors (Anti-Gaming)

Every task must satisfy four anchors:

1. Reference solution scores ≥ 0.95
2. Do-nothing baseline scores 0
3. Literal solution (implementing exactly what's asked) scores ≤ 0.2
4. Shortcut strategies (overfitting, round-number guessing, deferring to humans) score ≤ 0.8

Over 270 shortcut and bypass strategies were tested against the graders.

Experimental Setup

• 13 model configurations from seven providers (Claude Opus 5.5, Claude Sonnet 5.5, GPT-6 Astra, GPT-6.1 Sol, GLM-5.3 Flash, DeepSeek V4.1 Flash, Gemini 3.8 Flash, Qwen 3.8 Flash, GPT-5.5, GPT-6 Luna, Mistral Small 4, Claude Haiku 4.5, Claude Haiku 4.5 thinking)
• Each run limited to 200 turns, 150 minutes, or USD 8 in model usage cost
• Each task–instruction level combination run three times
• Total: 1,976 task–instruction level combinations

Empirical Validation / Results

Finding 1: Best Model Falls Far Short of Reference

At L3, Claude Opus 5.5 leads with 0.56 (95% CI: 0.50–0.62), followed by Claude Sonnet 5.5 (0.49), GPT-6 Astra (0.38), and GPT-6.1 Sol (0.37). Even selecting the best configuration per task yields only 0.70 with 30 tasks resolved; on 16 tasks no configuration reaches 0.5.

Finding 2: Constraint Discovery Is the Primary Bottleneck

The L0→L1 drop is 0.28 (95% interval: 0.17–0.39), while L1→L3 adds only 0.04. Questions per task rise from 3.9–7.0 (L0) to 11.3–13.0 (L1), with contact costs increasing from 0.07–0.10 to 0.18–0.24.

Finding 3: Obtaining Facts Does Not Ensure Effective Delivery

Leading models obtain 72–85% of key stakeholder facts, yet deliver only 0.60–0.70 of the reference uplift. In 73 task–model cases where all key facts were obtained, average uplift is 0.66, reaching ≥0.8 in only 37%. Stakeholder contact cost further reduces scores: Claude Opus 5.5 loses 0.14 of its 0.70 uplift; GPT-6 Astra loses 0.22 of 0.60.

Finding 4: Weak Models Often Score Below Do-Nothing Baseline

• Claude Haiku 4.5: below zero on 79 of 95 tasks
• Mistral Small 4: below zero on 51 tasks before contact costs
• GPT-6 Luna: below zero on 63 tasks
• Extended thinking does not help: Claude Haiku 4.5 scores 0.33 without, 0.35 with thinking

Finding 5: Performance Varies Substantially Across Scenarios

On five scenarios no model averages ≥0.5; on the hardest (news_incident, revenue_dispute), best scores are only 0.18 and 0.27. The median score range for a leading model within a scenario is 0.34–0.44, exceeding 0.5 in 6–8 of 20 scenarios.

Comparison with Other Benchmarks

Models solving up to 88% (Terminal-Bench 2.1), 90% (Vibe Code Bench), and 61% (Finance Agent) of tasks score only 0.19–0.56 on FDE-Gym—at most one in seven engagements improves the customer's business metric.

Theoretical and Practical Implications

For benchmark design: FDE-Gym establishes that delivery quality must be measured by business outcomes on the customer's own metric, not by task completion. The four-anchor admission system (reference ≥0.95, baseline = 0, literal ≤0.2, shortcuts ≤0.8) provides a methodological template for preventing gaming and ensuring tasks measure genuine delivery skill.

For model development: The findings identify specific capability gaps. The largest loss occurs between L0 and L1—not between L2 and L3—indicating that leading models handle misdirected requests well (using surrounding context to correct course) but struggle to discover unstated constraints. More fundamentally, even with complete information, models cannot translate facts into effective system design at the level of expert engineers.

For practitioners: A weak agent is not a cheaper version of a strong one—its expected contribution is negative on most tasks. The stakeholder contact cost mechanism reveals that agents ask questions as if they were free, consuming 0.14–0.22 of potential uplift. Cost and effort do not correlate with quality: GPT-6.1 Sol spends USD 0.24 and 25 turns (scores 0.37) while Gemini 3.8 Flash spends USD 1.54 and 121 turns (scores 0.19).

For evaluation: The online platform (fde-gym.com) enables human practice and benchmarking under identical grading criteria, allowing direct human–agent comparison (human baseline currently being collected).

Conclusion

FDE-Gym measures AI delivery the way customers do: by whether the customer's business improved, normalized between a do-nothing baseline (0) and an expert reference solution (1). The results reveal that the "last mile" of AI delivery remains long:

• Best model: 0.56 average score, 14/95 tasks resolved
• Primary bottleneck: constraint discovery, not problem framing
• Critical insight: obtaining facts ≠ effective delivery; contact costs and implementation quality remain major gaps
• Practical warning: weak models frequently deliver negative value

The paper's contributions are threefold: (1) a benchmark measuring agent performance on customer business metrics, (2) a construction methodology combining task variants, scripted stakeholders, and scoring anchors for reproducible anti-gaming evaluation, and (3) an evaluation of 13 model configurations identifying specific delivery capability gaps.

Limitations include: scripted stakeholders (not real humans), the L0–L2 evaluation limited to 20 original tasks, no human baseline yet, reliance on a single agent framework, and potential training-data contamination. Future work includes collecting human expert baselines, expanding scenario coverage, and refining measurement of stakeholder interaction costs.