# OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models

> OSReward reveals that VLM judges for computer-use agents systematically over-accept false successes, with accuracy collapsing to near chance on hard cases, motivating cheaper leniency-resistant reward models.

- **Source:** [arXiv](https://arxiv.org/abs/2607.28609)
- **Published:** 2026-08-08
- **Permalink:** https://picx.dev/p/lfAQBy

## Summary

# OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models

## Summary (Overview)

- **OSReward Benchmark**: Introduces the first standardized benchmark for evaluating VLM judges on computer-using agent (CUA) trajectories, built from scratch with 1,019 human-gold trajectories across four platforms (Web, Windows, Ubuntu, Mobile), plus two variants: OSReward-Hard (284 hard cases) and OSReward-Multi (fine-grained quality labels).

- **Key Finding – Leniency Bias**: The broadest evaluation of 27 VLM judges reveals a systematic leniency bias: judges are easily fooled by false successes (agents claiming completion without actually completing tasks), with over-accepts comprising roughly two-thirds of all errors across every model family.

- **Performance Collapse on Hard Cases**: On OSReward-Hard, even the best judge (Claude-Opus-4-8) drops to 69.7% accuracy (from 89.7% on the full set), while the mean judge falls to 52% — near chance on a 30/70 SUCCESS/FAIL split.

- **Cost-Accuracy Gap**: Reliable judges (Claude-Opus-4-8, GPT-5.5) cost $45–100 to judge the full benchmark once, making them unaffordable for training-scale reward signals; affordable open models trail by wide margins.

- **OS-Shepherd Models**: Release of OS-Shepherd-100K (open corpus of 96.6K reasoning-annotated trajectory judgments) and trained OS-Shepherd-9B and OS-Shepherd-35B reward models that match commercial judges at 30–60× lower cost, with learned leniency resistance that transfers across unseen benchmarks.

## Introduction and Theoretical Foundation

### Background

Computer-using agents (CUAs) operate across the digital world—web, mobile apps, and desktop software. A CUA trajectory records the agent's interleaved actions, states, and reasoning. Scaling CUA progress through evaluation, data curation, and reinforcement learning requires verifying whether each trajectory fulfilled its task instruction. However:

- **Human-written verifiers** cover only curated tasks and cannot apply to static corpora or previously collected trajectories
- **Human annotation** cannot keep pace with the scale of data collection
- **VLM judges** have become the de-facto practice for trajectory verification

### The Core Problem

The paper identifies a fundamental gap: **whether VLM judges are reliable for CUA trajectory verification had never been systematically tested**. Unlike judging text or general multimodal agents, judging a CUA trajectory requires:

> "reading a long, interleaved record of states, actions, and reasoning, then deciding whether the environment truly reached the instructed goal rather than whether the agent merely claims so"

A pilot study revealed the problem is real: even the best VLM judges disagree with existing benchmarks' own verifiers on roughly a quarter of desktop verdicts.

### Why Reusing Existing Benchmarks Fails

Reusing off-the-shelf trajectories from existing agent benchmarks would confound judge errors with flaws in the runs themselves. OSReward is instead built on **dedicated cross-platform data infrastructure** designed and operated end-to-end, with:

- Stock environments on web, mobile, Ubuntu, and Windows
- Common and professional applications a real user would have
- Realistic starting states (user profiles, real files, seeded databases, distractor content)
- Coverage of both pure-GUI and GUI+CLI workflows

## Methodology

### Data Infrastructure

Four platforms were prepared with realistic environments:

| Platform | Setup |
|----------|-------|
| **Web** | Chromium-based browser workers, parallel isolated sessions, live websites, self-hosted mirror sites for login tasks |
| **Windows** | ~20 everyday applications (IDEs, media editors, 3D/database tools), CLI utilities, 2K/4K resolution switching |
| **Ubuntu** | ~30 applications, typed pool of real files (~20 types), GUI+CLI action spaces |
| **Mobile** | Android emulator with initialized files, usage records, databases, seeded content and distractors |

### Task Instruction Pipeline

- ~1,500 instruction candidates authored by annotators after exploring environments
- **Peer cross-check** screening removes ambiguous, ungrounded, or unanswerable instructions
- ~800 instructions survive into the next stage
- Instructions include open-ended tasks beyond rule-based verifier capability

### Trajectory Collection and Annotation

1. **Agent execution**: Each instruction rolled out by 1–3 agents from Claude, Gemini, Kimi, and Qwen families, yielding both real successes and real failures
2. **Pre-filtering**: Discards runs with collection problems (anti-bot blocks, network failures, frozen executions)
3. **Human annotation**: Each trajectory labeled by three independent annotators using an in-house annotation website; annotators read full multimodal context (every screenshot, thought, action)
4. **Meta-review**: Disagreements escalate to two senior reviewers who deliberate together (not majority vote)
5. **Multi-axis scoring**: SUCCESS trajectories scored on alignment and efficiency (3-class scales: 0, 0.5, 1); FAIL trajectories tagged with error categories (reasoning-and-planning, action, perception, memory)

Total annotation effort: ~800 human hours.

### Benchmark Variants

- **OSReward (full set)**: 1,019 trajectories, 43% SUCCESS / 57% FAIL
- **OSReward-Hard**: 284 trajectories from annotator-disagreement cases, 30/70 SUCCESS/FAIL split, designed to expose error modes
- **OSReward-Multi**: 440 successful trajectories with alignment and efficiency sub-labels

### Judge Evaluation Protocol

- Judges read the last $N=5$ states with per-step reasoning and action text
- Output: SUCCESS/FAIL verdict + alignment/efficiency ratings
- Metrics: binary accuracy, success recall (sRec), fail recall (fRec), balanced accuracy (BalAcc = mean of sRec and fRec)

## Empirical Validation / Results

### Performance Ceiling on Full Set

Only frontier judges approach the ~90% accuracy needed for training-time reward:

- **Claude-Opus-4-8**: 89.7% accuracy, 90.0% BalAcc
- **GPT-5.5**: 89.5% accuracy, 89.8% BalAcc
- **Claude-Opus-4-6**: 89.5% accuracy, 90.2% BalAcc
- Open-weight judges lag: Kimi-K2.5 (85.9%), Qwen3.5-397B-A17B (85.8%)
- Small open VL models trail far behind: Qwen3-VL-30B at 69.4%

### The Leniency Bias

Analysis of the strict-lenient plane (fRec vs. sRec) reveals two clusters:

- **Lenient cluster** (most models): high sRec but low fRec—accepting almost everything including failures
- **Strict cluster** (GPT-5.2, Claude-Haiku): trades success recall to catch more failures
- **Balanced judges** (few): near the diagonal (sRec ≈ fRec)

The dominant error mode: **over-accepting an incomplete task** makes up two-thirds of all errors, and this is the leading error mode of every single judge (≥48% of each judge's mistakes). Pooled across judges, over-accepts outnumber over-rejects **three to one**.

### OSReward-Hard: The Challenge Set

- Every judge drops 20–43 pp on the hard set
- Best judge (Claude-Opus-4-8): 69.7% (from 89.7%)
- Mean judge: 52% (near chance)
- Windows is the hardest platform to judge; mobile the easiest
- Perception and action failures (which require reading the screen) are harder to catch than planning-and-reasoning failures (legible in text)

### Key Ablation Findings

**Visual inputs barely matter**: Swapping screenshot counts or removing the red click marker moves accuracy by <0.5 pp. However, these changes still flip 5–7% of individual verdicts.

**Text history matters most**: Dropping per-step thought and action text costs 7.2 pp on average (22.7% of verdicts flipped). Dropping only chain-of-thought is milder (1.8 pp, 11.6% flipped). This explains the leniency bias: judges lean on the agent's narrative rather than the screen.

**Decoding noise provides scale**: Re-running the same judge at T=0.7 flips 6–9% of verdicts, so visual settings' churn is at or below sampling noise.

**Ensembling doesn't help**: Top judges agree far more than they disagree (pairwise Cohen's κ ≈ 0.71) and herd on the same hard trajectories. A top-3 majority vote edges the best single judge by ~1 point at several times the cost. However, an oracle (accepting any pooled judge's correct verdict) reaches 99% accuracy—the pool almost always contains a correct verdict, but a vote cannot identify which judge to trust.

### Fine-Grained Grading (OSReward-Multi)

Quality grading is far weaker than outcome judging:

| Judge | Align (macro-recall) | Effic (macro-recall) | Multi | AUC |
|-------|---------------------|---------------------|-------|-----|
| GPT-5.5 | 58.7 | 68.2 | 63.5 | 66.7 |
| Claude-Opus-4-8 | 52.9 | 68.7 | 60.8 | 65.6 |
| OS-Shepherd-35B-A3B | 47.7 | 65.8 | 56.8 | 60.7 |
| OS-Shepherd-9B | 44.1 | 54.0 | 49.0 | 58.5 |

Judges rank levels better than they score them (AUC > macro-recall), indicating miscalibration. Weakness concentrates on alignment, where judges default to the top rating.

### OS-Shepherd Results

| Model | OSReward Acc | sRec | fRec | BalAcc | Hard Acc | Hard sRec | Hard fRec | Hard BalAcc |
|-------|-------------|------|------|--------|----------|-----------|-----------|-------------|
| Qwen3.5-9B (base) | 76.7 | 98.9 | 59.9 | 79.4 | 39.4 | 97.7 | 14.1 | 55.9 |
| **OS-Shepherd-9B** | **86.1** | **86.6** | **86.0** | **86.3** | **60.2** | **66.3** | **57.6** | **61.9** |
| Qwen3.5-35B-A3B (base) | 82.2 | 92.4 | 74.5 | 83.5 | 51.1 | 83.7 | 36.9 | 60.3 |
| **OS-Shepherd-35B-A3B** | **85.6** | **85.0** | **86.2** | **85.6** | **62.7** | **68.6** | **60.1** | **64.3** |

OS-Shepherd-9B moves from the bottom third of the field into the commercial band, catching 57.6% of hard false successes (where cheap lenient judges miss nearly all). Cost: $1.36 to judge the full set vs. $45–100 for frontier judges (30–60× reduction).

### Generalization to Existing Benchmarks

OS-Shepherd's de-biasing transfers to independently built benchmarks:

- **OSWorld and AndroidWorld**: Best open judges, beating every general open model up to Qwen3.5-397B-A17B (~44× the 9B's size)
- **WebArena**: Sits in the frontier cluster; the untuned base model falls below the trivial all-FAIL baseline, while OS-Shepherd is the tightest-calibrated judge in the field
- Agreement with human-written verifiers varies by platform more than by judge: best judges approach ~90% on mobile, come within ~6 pp on web, and fall well short on desktop

## Theoretical and Practical Implications

### For CUA Evaluation

- **Human-written verifiers are non-scalable**: They cost too much to author per task and don't generalize past their own tasks
- **VLM judges are the only practical route** to trajectory-level verification at scale, but their reliability must be measured, not assumed
- The systematic leniency bias means current evaluation pipelines likely overestimate agent capabilities

### For Reward Modeling

- The finding that judges read the agent's narrative more than the screen suggests a recipe for reward-model labeling: **keep the full text history, drop the click marker, and set screenshot count per model**
- The herding behavior of judges argues for **soft-label, confidence-weighted reward modeling** over hard votes
- The false-success problem is the highest-value intervention target since it directly reinforces incorrect agent behavior

### For Training-Scale Economics

A single modest RL bout (200 updates, batch 16, 16 rollouts) issues 51,200 judge calls:

- ~$4,000 with Claude-Opus-4-8
- ~$2,300 with GPT-5.5
- ~$68 with OS-Shepherd-9B (and self-hostable for GPU-time cost only)

### For the Open-Source Community

OS-Shepherd-100K is the first large-scale reasoning-annotated judge corpus for CUA trajectories, with failure types annotated and analyzed, enabling further open research on CUA reward models.

## Conclusion

The paper establishes that:

1. **VLM judges are not yet reliable enough** for CUA trajectory verification, with a dominant leniency bias (accepting false successes) that is shared across all 27 evaluated models
2. **Hard cases expose the gap**: On OSReward-Hard, even the best judges approach chance, and the headroom lies in reading the screen and verifying completion rather than high-level reasoning
3. **Reliability can be bought but not afforded**: The few trustworthy judges cost $45–100 per full-benchmark evaluation, which doesn't scale to training-time reward
4. **Open models can close the gap**: With the OS-Shepherd-100K corpus and two-stage training recipe (SFT + GRPO RL targeting false successes), small open judges (9B, 35B) match commercial judges at 30–60× lower cost, with learned leniency resistance that transfers out of distribution to unseen benchmarks

**Future directions** identified by the authors include:
- Prompting judges to explicitly verify completion (potentially more effective than ensembling)
- Soft-label, confidence-weighted reward modeling
- Scaling the OS-Shepherd approach to further close the frontier gap

> "A judge that is accurate without being too strict or too lenient barely exists yet" — but OS-Shepherd demonstrates that "reliable CUA reward no longer has to be bought at high prices."

---

_Markdown view of https://picx.dev/p/lfAQBy, served by PicX — AI-generated visual whiteboard summaries of research papers._