OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models

Summary (Overview)

• OSReward Benchmark: Introduces the first standardized benchmark for evaluating VLM judges on computer-using agent (CUA) trajectories, built from scratch with 1,019 human-gold trajectories across four platforms (Web, Windows, Ubuntu, Mobile), plus two variants: OSReward-Hard (284 hard cases) and OSReward-Multi (fine-grained quality labels).

• Key Finding – Leniency Bias: The broadest evaluation of 27 VLM judges reveals a systematic leniency bias: judges are easily fooled by false successes (agents claiming completion without actually completing tasks), with over-accepts comprising roughly two-thirds of all errors across every model family.

• Performance Collapse on Hard Cases: On OSReward-Hard, even the best judge (Claude-Opus-4-8) drops to 69.7% accuracy (from 89.7% on the full set), while the mean judge falls to 52% — near chance on a 30/70 SUCCESS/FAIL split.

• Cost-Accuracy Gap: Reliable judges (Claude-Opus-4-8, GPT-5.5) cost $45–100 to judge the full benchmark once, making them unaffordable for training-scale reward signals; affordable open models trail by wide margins.

• OS-Shepherd Models: Release of OS-Shepherd-100K (open corpus of 96.6K reasoning-annotated trajectory judgments) and trained OS-Shepherd-9B and OS-Shepherd-35B reward models that match commercial judges at 30–60× lower cost, with learned leniency resistance that transfers across unseen benchmarks.

Introduction and Theoretical Foundation

Background

Computer-using agents (CUAs) operate across the digital world—web, mobile apps, and desktop software. A CUA trajectory records the agent's interleaved actions, states, and reasoning. Scaling CUA progress through evaluation, data curation, and reinforcement learning requires verifying whether each trajectory fulfilled its task instruction. However:

• Human-written verifiers cover only curated tasks and cannot apply to static corpora or previously collected trajectories
• Human annotation cannot keep pace with the scale of data collection
• VLM judges have become the de-facto practice for trajectory verification

The Core Problem

The paper identifies a fundamental gap: whether VLM judges are reliable for CUA trajectory verification had never been systematically tested. Unlike judging text or general multimodal agents, judging a CUA trajectory requires:

"reading a long, interleaved record of states, actions, and reasoning, then deciding whether the environment truly reached the instructed goal rather than whether the agent merely claims so"

A pilot study revealed the problem is real: even the best VLM judges disagree with existing benchmarks' own verifiers on roughly a quarter of desktop verdicts.

Why Reusing Existing Benchmarks Fails

Reusing off-the-shelf trajectories from existing agent benchmarks would confound judge errors with flaws in the runs themselves. OSReward is instead built on dedicated cross-platform data infrastructure designed and operated end-to-end, with:

• Stock environments on web, mobile, Ubuntu, and Windows
• Common and professional applications a real user would have
• Realistic starting states (user profiles, real files, seeded databases, distractor content)
• Coverage of both pure-GUI and GUI+CLI workflows

Methodology

Data Infrastructure

Four platforms were prepared with realistic environments:

Task Instruction Pipeline

• ~1,500 instruction candidates authored by annotators after exploring environments
• Peer cross-check screening removes ambiguous, ungrounded, or unanswerable instructions
• ~800 instructions survive into the next stage
• Instructions include open-ended tasks beyond rule-based verifier capability

Trajectory Collection and Annotation

1. Agent execution: Each instruction rolled out by 1–3 agents from Claude, Gemini, Kimi, and Qwen families, yielding both real successes and real failures
2. Pre-filtering: Discards runs with collection problems (anti-bot blocks, network failures, frozen executions)
3. Human annotation: Each trajectory labeled by three independent annotators using an in-house annotation website; annotators read full multimodal context (every screenshot, thought, action)
4. Meta-review: Disagreements escalate to two senior reviewers who deliberate together (not majority vote)
5. Multi-axis scoring: SUCCESS trajectories scored on alignment and efficiency (3-class scales: 0, 0.5, 1); FAIL trajectories tagged with error categories (reasoning-and-planning, action, perception, memory)

Total annotation effort: ~800 human hours.

Benchmark Variants

• OSReward (full set): 1,019 trajectories, 43% SUCCESS / 57% FAIL
• OSReward-Hard: 284 trajectories from annotator-disagreement cases, 30/70 SUCCESS/FAIL split, designed to expose error modes
• OSReward-Multi: 440 successful trajectories with alignment and efficiency sub-labels

Judge Evaluation Protocol

• Judges read the last $N=5$ states with per-step reasoning and action text
• Output: SUCCESS/FAIL verdict + alignment/efficiency ratings
• Metrics: binary accuracy, success recall (sRec), fail recall (fRec), balanced accuracy (BalAcc = mean of sRec and fRec)

Empirical Validation / Results

Performance Ceiling on Full Set

Only frontier judges approach the ~90% accuracy needed for training-time reward:

• Claude-Opus-4-8: 89.7% accuracy, 90.0% BalAcc
• GPT-5.5: 89.5% accuracy, 89.8% BalAcc
• Claude-Opus-4-6: 89.5% accuracy, 90.2% BalAcc
• Open-weight judges lag: Kimi-K2.5 (85.9%), Qwen3.5-397B-A17B (85.8%)
• Small open VL models trail far behind: Qwen3-VL-30B at 69.4%

The Leniency Bias

Analysis of the strict-lenient plane (fRec vs. sRec) reveals two clusters:

• Lenient cluster (most models): high sRec but low fRec—accepting almost everything including failures
• Strict cluster (GPT-5.2, Claude-Haiku): trades success recall to catch more failures
• Balanced judges (few): near the diagonal (sRec ≈ fRec)

The dominant error mode: over-accepting an incomplete task makes up two-thirds of all errors, and this is the leading error mode of every single judge (≥48% of each judge's mistakes). Pooled across judges, over-accepts outnumber over-rejects three to one.

OSReward-Hard: The Challenge Set

• Every judge drops 20–43 pp on the hard set
• Best judge (Claude-Opus-4-8): 69.7% (from 89.7%)
• Mean judge: 52% (near chance)
• Windows is the hardest platform to judge; mobile the easiest
• Perception and action failures (which require reading the screen) are harder to catch than planning-and-reasoning failures (legible in text)

Key Ablation Findings

Visual inputs barely matter: Swapping screenshot counts or removing the red click marker moves accuracy by <0.5 pp. However, these changes still flip 5–7% of individual verdicts.

Text history matters most: Dropping per-step thought and action text costs 7.2 pp on average (22.7% of verdicts flipped). Dropping only chain-of-thought is milder (1.8 pp, 11.6% flipped). This explains the leniency bias: judges lean on the agent's narrative rather than the screen.

Decoding noise provides scale: Re-running the same judge at T=0.7 flips 6–9% of verdicts, so visual settings' churn is at or below sampling noise.

Ensembling doesn't help: Top judges agree far more than they disagree (pairwise Cohen's κ ≈ 0.71) and herd on the same hard trajectories. A top-3 majority vote edges the best single judge by ~1 point at several times the cost. However, an oracle (accepting any pooled judge's correct verdict) reaches 99% accuracy—the pool almost always contains a correct verdict, but a vote cannot identify which judge to trust.

Fine-Grained Grading (OSReward-Multi)

Quality grading is far weaker than outcome judging:

Judges rank levels better than they score them (AUC > macro-recall), indicating miscalibration. Weakness concentrates on alignment, where judges default to the top rating.

OS-Shepherd Results

OS-Shepherd-9B moves from the bottom third of the field into the commercial band, catching 57.6% of hard false successes (where cheap lenient judges miss nearly all). Cost: $1.36 to judge the full set vs.$45–100 for frontier judges (30–60× reduction).

Generalization to Existing Benchmarks

OS-Shepherd's de-biasing transfers to independently built benchmarks:

• OSWorld and AndroidWorld: Best open judges, beating every general open model up to Qwen3.5-397B-A17B (~44× the 9B's size)
• WebArena: Sits in the frontier cluster; the untuned base model falls below the trivial all-FAIL baseline, while OS-Shepherd is the tightest-calibrated judge in the field
• Agreement with human-written verifiers varies by platform more than by judge: best judges approach ~90% on mobile, come within ~6 pp on web, and fall well short on desktop

Theoretical and Practical Implications

For CUA Evaluation

• Human-written verifiers are non-scalable: They cost too much to author per task and don't generalize past their own tasks
• VLM judges are the only practical route to trajectory-level verification at scale, but their reliability must be measured, not assumed
• The systematic leniency bias means current evaluation pipelines likely overestimate agent capabilities

For Reward Modeling

• The finding that judges read the agent's narrative more than the screen suggests a recipe for reward-model labeling: keep the full text history, drop the click marker, and set screenshot count per model
• The herding behavior of judges argues for soft-label, confidence-weighted reward modeling over hard votes
• The false-success problem is the highest-value intervention target since it directly reinforces incorrect agent behavior

For Training-Scale Economics

A single modest RL bout (200 updates, batch 16, 16 rollouts) issues 51,200 judge calls:

• ~$4,000 with Claude-Opus-4-8
• ~$2,300 with GPT-5.5
• ~$68 with OS-Shepherd-9B (and self-hostable for GPU-time cost only)

For the Open-Source Community

OS-Shepherd-100K is the first large-scale reasoning-annotated judge corpus for CUA trajectories, with failure types annotated and analyzed, enabling further open research on CUA reward models.

Conclusion

The paper establishes that:

1. VLM judges are not yet reliable enough for CUA trajectory verification, with a dominant leniency bias (accepting false successes) that is shared across all 27 evaluated models
2. Hard cases expose the gap: On OSReward-Hard, even the best judges approach chance, and the headroom lies in reading the screen and verifying completion rather than high-level reasoning
3. Reliability can be bought but not afforded: The few trustworthy judges cost $45–100 per full-benchmark evaluation, which doesn't scale to training-time reward
4. Open models can close the gap: With the OS-Shepherd-100K corpus and two-stage training recipe (SFT + GRPO RL targeting false successes), small open judges (9B, 35B) match commercial judges at 30–60× lower cost, with learned leniency resistance that transfers out of distribution to unseen benchmarks

Future directions identified by the authors include:

• Prompting judges to explicitly verify completion (potentially more effective than ensembling)
• Soft-label, confidence-weighted reward modeling
• Scaling the OS-Shepherd approach to further close the frontier gap

"A judge that is accurate without being too strict or too lenient barely exists yet" — but OS-Shepherd demonstrates that "reliable CUA reward no longer has to be bought at high prices."