# FrontierChallenge: Evaluating Scientific Workflow Completion

> FRONTIERCHALLENGE shows that even top AI agents complete only ~20% of scientific workflows, with high partial scores masking rare full delivery and unreliable self-reported completion.

- **Source:** [arXiv](https://arxiv.org/abs/2608.24979)
- **Published:** 2026-08-28
- **Permalink:** https://picx.dev/p/EuFxsc
- **Whiteboard:** https://picx.dev/p/EuFxsc/image

## Summary

# FrontierChallenge: Evaluating Scientific Workflow Completion

## Summary (Overview)

- **FRONTIERCHALLENGE** is a cross-domain benchmark of 300 end-to-end scientific workflows, with 97 tasks released and evaluated spanning quantum chemistry, molecular dynamics, materials characterization, analytical chemistry, life science, and electrochemistry/environment.
- **Key finding**: The best-performing configurations (GPT-5.6 Sol with Codex and Grok 4.6 with Claude Code) completed only 20 of 97 tasks (Pass Rate of 20.6%), despite average scores reaching as high as 87.9.
- **Critical gap**: High partial scores did not translate to complete delivery—analytical chemistry reached Avg. Scores of 87.6 but only 4% Pass Rate, while electrochemistry/environment achieved 94.9 Avg. Score with 0% Pass Rate.
- **Failure analysis**: 75.5% of non-passing Claude Code trajectories ended with language claiming completion, demonstrating that confident self-reports do not reliably indicate successful delivery.
- **Three contributions**: (1) cross-domain scientific workflow benchmark with task-specific Graders, (2) contract-level evaluation using Pass Rate as primary metric, (3) frontier model study with failure analysis across 970 trajectories.

## Introduction and Theoretical Foundation

The paper addresses a fundamental shift in how language models are evaluated—from text generators to agents that can plan, call tools, execute code, and modify persistent files. The authors argue that existing benchmarks (HLE, AgentBench, SWE-bench, etc.) focus on final answers, isolated programs, or single-domain workflows, which fails to capture whether an agent can complete *heterogeneous scientific work* where success depends on multiple analytical stages and deliverables.

**Core research question**: Given a specified scientific task and fixed data, can an agent independently complete the workflow from input processing to final deliverables and satisfy the complete task contract?

The benchmark targets a narrower capability than autonomous science: executing a *specified* scientific workflow where objective, inputs, and required outputs are pre-defined. The agent is not asked to set research agendas or formulate problems.

**Key design principles** for task curation:
1. **Representativeness**: Workflows must reflect plausible professional practice
2. **Complexity**: Tasks require end-to-end, dependency-aware processes
3. **Diversity**: Vary by scientific knowledge, workflow type, and difficulty
4. **Verifiability**: Outputs must be assessable through files, numerical values, or explicit acceptance criteria

## Methodology

### Task Collection and Packaging
- **300 workflows** constructed across six domains: quantum chemistry (20), molecular dynamics (16), materials characterization (22), analytical chemistry (23), life science (10), electrochemistry/environment (6)
- Tasks require domain software including ORCA, CP2K, LAMMPS, AmberTools, and PLUMED
- Each task package contains: task metadata, agent-facing instructions, input data, expected-output references, stepwise scoring rubric, executable Grader, and documentation
- Tasks classified as 74 Hard and 23 Medium difficulty

### Agent Scaffolds and Models
Three agent scaffolds were evaluated:
- **Codex** (OpenAI) with GPT-5.6 Sol and GPT-5.6 Terra (max)
- **Claude Code** (Anthropic) with ten models
- **Frontier Agent** (ApodexAI) with Apodex 1.1 in Agent Team configuration

### Evaluation Metrics
Each task has a task-specific Grader returning a native score $s_{mi} \in [0, 100]$ for configuration $m$ on task $i$. A Judge (GPT-5.6 Sol) handles rubric-defined semantic criteria, with three passes per criterion.

The full-completion indicator is defined as $f_{mi} = 1[s_{mi} \geq 99.9]$ (the 99.9 threshold only absorbs minor numerical variation from averaging judge scores). The primary metrics are:

$$
\mathrm{PassRate}_{m} = \frac{1}{N} \sum_{i=1}^{N} f_{mi}, \tag{1}
$$

$$
\mathrm{Avg.Score}_{m} = \frac{1}{N} \sum_{i=1}^{N} s_{mi}, \tag{2}
$$

where $N = 97$ is the number of evaluated tasks.

## Empirical Validation / Results

### Overall Performance

| Model | Agent Scaffold | Overall ↑ | Medium ↑ | Hard ↑ |
|-------|----------------|-----------|----------|---------|
| GPT-5.6 Sol | Codex | 20.6% | 39.1% | 14.9% |
| Grok 4.6 | Claude Code | 20.6% | 43.5% | 13.5% |
| Kimi K3 | Claude Code | 17.5% | 39.1% | 10.8% |
| Claude Opus 5 | Claude Code | 17.5% | 39.1% | 10.8% |
| GPT-5.6 Terra (max) | Codex | 15.5% | 39.1% | 8.1% |
| Qwen 3.8 Max | Claude Code | 15.5% | 34.8% | 9.5% |
| DeepSeek V4 Pro-0813 | Claude Code | 13.4% | 30.4% | 8.1% |
| DeepSeek V4 Flash-0731 | Claude Code | 12.4% | 34.8% | 5.4% |
| Apodex 1.1 | Frontier Agent (Agent Team) | 12.4% | 30.4% | 6.8% |
| Gemini 3.7 Flash | Claude Code | 10.3% | 21.7% | 6.8% |
| Apodex 1.1 | Claude Code | 10.3% | 30.4% | 4.1% |
| Qwen3.5-397B-A17B | Claude Code | 4.1% | 13.0% | 1.4% |
| GLM-5.2 | Claude Code | 3.1% | 4.3% | 2.7% |

### Domain-Level Performance
- **Quantum chemistry**: Grok 4.6 achieved highest Pass Rate (60%), followed by GPT-5.6 Sol and Claude Opus 5 at 55%
- **Molecular dynamics**: GPT-5.6 Sol, GPT-5.6 Terra (max), and Grok 4.6 each achieved 38% Pass Rate
- **Materials characterization**: highest Avg. Score 88.1 but no configuration exceeded 9% Pass Rate
- **Analytical chemistry**: highest Avg. Score 87.6, but only DeepSeek V4 Pro-0813 completed any task (4% Pass Rate)
- **Electrochemistry/environment**: maximum Avg. Score 94.9 but 0% Pass Rate for every configuration
- **Life science**: GPT-5.6 Sol achieved highest Avg. Score (76.8); Kimi K3 achieved highest Pass Rate (20%)

### Resource Usage
- Reported input tokens varied more than sixfold: from 2.183 million per task (Grok 4.6) to 13.730 million (Apodex 1.1 with Claude Code)
- Mean execution time ranged from 21.8 to 112.8 minutes per task
- Frontier Agent (Agent Team) had the longest mean time (112.8 min) with median 54.9 min and 90th percentile of 272.2 min

### Failure Mode Analysis
- **Judge-assessed artifact shortfalls** appeared in 97% of non-passing materials-characterization submissions, 95% of analytical chemistry, and 85% of electrochemistry/environment, but only 43% in quantum chemistry
- **Deterministic grader diagnostics** were more common in life science (50%) and quantum computing (40%)
- **Completion language**: 641 of 849 non-passing Claude Code runs (75.5%) had final messages containing lexical completion language; only 13 (1.5%) explicitly indicated work was still in progress
- **Tool errors**: At least one tool error occurred in 80.7% of non-passing and 94.2% of passing trajectories; median error rates were 4.4 and 5.6 per 100 tool results, respectively

## Theoretical and Practical Implications

The findings have several significant implications:

1. **High partial scores do not guarantee complete delivery**: The persistent gap between Avg. Score and Pass Rate shows that substantial progress on tasks often misses at least one critical requirement, suggesting that current models lack the ability to maintain consistency across multiple interdependent deliverables.

2. **Self-reported completion is unreliable**: The finding that 75.5% of non-passing trajectories end with confident completion language has significant implications for trust and evaluation—final self-reports cannot be used as a substitute for evaluating the actual submitted artifact bundle.

3. **Domain-specific variations matter**: The distinct domain profiles show that aggregate rankings do not capture scientific setting-specific strengths and weaknesses. This suggests that models may require domain-specific optimization rather than a one-size-fits-all approach.

4. **Tool errors are not predictive**: The presence of tool errors in both passing and non-passing runs indicates that error recovery capability, not error avoidance, is a critical capability that current evaluation metrics may not adequately capture.

### Practical Implications
- **Contract tracking**: More reliable scientific agents will require explicit contract tracking, cross-artifact validation, and evidence-based completion checks
- **Evaluation design**: Benchmarks should evaluate the complete artifact bundle, not just single answers or code
- **Trustworthiness**: Scientific deliverables must be inspectable, reproducible, and mutually consistent for trustworthy handoffs

## Conclusion

FRONTIERCHALLENGE establishes that complete, contract-level delivery is a distinct and unresolved capability for current frontier models. The paper demonstrates that:

1. **Completion is rare**: Even the best configurations complete only ~20% of tasks
2. **Partial progress is misleading**: High Avg scores do not translate to complete delivery, particularly in analytical chemistry and electrochemistry/environment
3. **Self-reports are unreliable**: Final self-reports and the mere presence of errors are weak indicators of successful delivery
4. **Domain-specific evaluation is necessary**: Aggregate rankings mask important domain-level differences

**Limitations** acknowledged by the authors include:
- The findings are limited to the released task set (97 of 300)
- Results are specific to the evaluated configurations and Claude Code trajectory scope
- Resource accounting is provider-specific
- Single runs were used (no averaging across multiple runs)

Future directions point toward developing agents with **explicit contract tracking, cross-artifact validation, and evidence-based completion checks**, as well as expanding the benchmark to include the remaining held-out tasks and GPU-requiring workflows.

---

_Markdown view of https://picx.dev/p/EuFxsc, served by PicX — AI-generated visual whiteboard summaries of research papers._