GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay

Summary (Overview)

• GameHorizon is a unified data and evaluation suite for measuring gameplay capabilities across multiple temporal horizons (short-horizon operations, medium-horizon goals, long-horizon strategies) in AAA video games.
• The suite comprises three components: GameHorizon-Annotator (an automated annotation pipeline), GameHorizon-Data (5,000 hours of gameplay from 21 games collected by 100 expert players with 6.18M multi-horizon instructions), and GameHorizon-Bench (reproducible offline + stepwise online evaluation).
• The benchmark evaluates 47 models across five families (VLMs, UMMs, GUI agents, coding agents, game agents) through over one million model invocations, revealing a meaningful task-difficulty hierarchy and pronounced capability differences.
• Key finding: future-action planning and goal decomposition are more challenging than current-action perception; multi-horizon instructions improve future-action planning by 7.2 percentage points over vision-only inputs.
• The offline track shows a clear positive association with online gameplay performance, validating offline scores as a proxy for actual gameplay capability.

Introduction and Theoretical Foundation

Background and Motivation

Modern video games provide a measurable testbed for AI models, requiring simultaneous abilities in:

1. Visual understanding – interpreting complex game environments
2. Instruction decomposition – breaking down natural-language goals
3. Goal planning – organizing subgoals across temporal scales
4. Precise action control – executing keyboard/mouse actions

Game objectives span varying temporal scales: collecting an item within seconds, winning a fight lasting minutes, or executing a strategy over an entire session. These objectives are all reflected in the same stream of primitive actions (keystrokes, mouse movements).

Limitations of Existing Work

The paper identifies three critical gaps in prior datasets and benchmarks:

1. Narrow game coverage: GameWorld targets simple mini-games; STEVE-1, MineDojo, and VPT are confined to Minecraft; WildWorld uses only Monster Hunter Wilds. Conclusions from single titles cannot generalize to heterogeneous AAA games.

2. Lack of language instructions: NitroGen and GameVerse omit instructions entirely; Open-P2P provides sparse annotations; Game-TARS relies on costly manual annotation. Instructions are vital for human–agent interaction, including instruction following and goal decomposition.

3. High-variance online evaluation: Lumine reports success rates from only three trials per scene; GameVerse conducts 3–20 rollouts. Small samples lead to low-confidence comparisons, and aggregate success rates conflate distinct failure modes (action misidentification vs. goal inference vs. planning errors).

Key Insight

The paper observes that gameplay capability splits across two model families:

• Dedicated game agents (Open-P2P, JARVIS-VLA) are optimized for acting with lightweight VLA architectures but lack reasoning/planning capacity.
• General-purpose models (Gemini, GPT, Claude) excel at planning but lack systematic action evaluation.

A single standardized yardstick is needed to measure both families across vision, executable actions, and multi-horizon natural-language goals.

Methodology

GameHorizon-Annotator

The annotation pipeline operates bottom-up to construct a three-level instruction pyramid:

1. Action-Aware Video Segmentation:

• Raw keyboard-mouse events are mapped to game-specific action semantics (e.g., Shift = sprinting in Cyberpunk 2077)
• Action streams are scanned chronologically, grouping sustained events and using discrete events (jumping, attacking) as initial boundaries
• A VLM (Gemini) examines video frames to disambiguate action semantics (e.g., left-click as attack vs. item selection) and refine segmentation

2. Bottom-Up Temporal Merging:

• Adjacent $L_1$ clips are merged into $L_2$ clips based on action continuity and semantic coherence
• $L_2$ clips are merged into $L_3$ clips based on goal continuity
• Dynamic programming enforces level-specific durations:
  • $L_1$: 1–5 seconds (short-horizon operations)
  • $L_2$: 1–2 minutes (medium-horizon goals)
  • $L_3$: 5–8 minutes (long-horizon strategies)

3. Multi-Horizon Instruction Annotation:

• $L_1$: VLM receives sampled frames + keyboard-mouse actions → generates action-grounded operation descriptions with coordinates, object descriptions, and spatial relations
• $L_2$: VLM receives frames + actions + constituent $L_1$ instructions → synthesizes medium-horizon goal descriptions
• $L_3$: VLM receives frames + $L_2$ instructions (no actions) → generates high-level strategy descriptions

All prompts include game title, clip duration, frame rate, and game-specific keybinds for action interpretation.

GameHorizon-Data

Key advantage: direct human actions (synchronized keyboard-mouse) rather than IDM-inferred pseudo-labels, ensuring faithful action trajectories.

GameHorizon-Bench

Offline Track – 5,000 standardized MCQs:

• $T_1$ Single-Horizon Action (1,000 questions): Given frames + $L_1$ instruction, determine correct action sequence (evaluates action perception)
• $T_2$ Multi-Horizon Decomposition (1,000 questions): Given $L_2$ goal, identify ordered sequence of $L_1$ operations (evaluates goal decomposition and planning)
• $T_3$ Cross-Horizon Consistency (1,000 questions): Assess consistency across frames, $L_1$–$L_3$ instructions, and actions
• 10 Variant Tasks (200 questions each): Diagnostic probes including:
  • $T_1^*$: Current-action perception vs. future-action planning under three input settings
  • $T_2^*$: Top-down decomposition vs. bottom-up abstraction
  • $T_3^*$: Instruction-only vs. instruction-and-action options

Online Track – Stepwise evaluation in Minecraft (due to game-state accessibility):

• 10 causal tasks (order-dependent subtasks) + 10 thematic tasks (order-flexible subtasks)
• 62 verifiable short-horizon subtasks total
• Environment reset on failure enables failure localization to specific steps
• Long-horizon task passed only when all subtasks succeed

Empirical Validation / Results

Offline Primary Tasks

Overall benchmark statistics:

• Mean accuracy: 64.7% (vs. 25% random baseline)
• Range: 44.6% to 80.2% (35.6-point spread across models)
• Task difficulty hierarchy: $T_1$ = 57.3% < $T_2$ = 65.1% < $T_3$ = 71.6%

Top-performing models (Tier 1):

Key findings:

• Proprietary models dominate (10/11 Tier 1 positions); Kimi-K3 is the only open-weight model in Tier 1
• UMMs (BAGEL-7B-MoT, SenseNova-U1-8B-MoT, Ovis-U1-3B, InternVL-U-4B) all fall in Tiers 3–4, suggesting understanding-generation unification introduces interference
• GUI agents (GELab-Zero-4B-Preview, UI-TARS-1.5-7B) show limited transfer from computer-use to gameplay
• Coding agents (Claude Sonnet 5, Opus 4.8, Fable 5) generalize well, with Fable 5 reaching Tier 2 (71.2%)
• Dedicated game agents perform near random baseline (22.0–28.1%) on unseen AAA games; JARVIS-VLA drops from 54.2% in-domain (Minecraft) to 28.1% zero-shot (AAA)

Effects of thinking:

• Improvement: GLM-5V-Turbo +5.6 pts, Doubao-Seed-2.1-Pro +9.3 pts (gains concentrated on $T_2$: +9.0 and +19.9 pts)
• Degradation: UI-TARS-1.5-7B −8.2 pts, Qwen2.5-VL-7B −5.9 pts (lightweight models hallucinate reasoning)

Offline Variant Tasks

Action perception vs. future planning ($T_1^*$):

Key insight: Short-horizon instructions help current-action perception (+14.1 pts); multi-horizon instructions help future-action planning (+7.0 pts).

Top-down vs. bottom-up ($T_2^*$):

• Bottom-up abstraction: 98.4% accuracy (near ceiling)
• Top-down decomposition: 69.5% accuracy (28.9-point gap)
• Validates bottom-up design for annotation, top-down for evaluation

Action-decoding error ($T_3^*$):

• Adding actions to consistency task: only −0.6 pts (74.0% → 73.4%)
• Primary bottleneck is goal decomposition/planning, not action decoding

Online Track Results

Key findings:

• Clear positive association between offline ranks and online performance (Tiers 1–4 map to online ranks 1–3, 4–6, 7–9, 10–12)
• Long-horizon tasks remain extremely difficult: 10/12 models achieve ≤10% task success
• Stepwise protocol enables failure localization (e.g., distinguishing models that don't know how to craft a cake from those that understand but execute incorrectly)

Theoretical and Practical Implications

Theoretical Contributions

1. Multi-horizon instruction pyramid: The paper demonstrates that gameplay capability must be measured across temporal scales, with distinct instruction types serving different purposes—short-horizon for action grounding, medium-horizon for goal decomposition, long-horizon for strategy formulation.

2. Bottom-up annotation vs. top-down evaluation: The 28.9-point gap between abstraction (98.4%) and decomposition (69.5%) reveals an information asymmetry: concrete actions determine goals more reliably than abstract goals determine actions. This justifies the design principle of bottom-up data annotation and top-down benchmark evaluation.

3. Bottleneck localization: The finding that action decoding contributes only −0.6 pts while goal decomposition/planning dominates errors identifies the primary capability gap in current models.

4. Understanding-generation trade-off: UMMs underperforming their understanding backbones (InternVL-U-4B vs. InternVL3.5-2B, −1.4 pts) suggests interference in unified architectures.

Practical Implications

1. Standardized yardstick: GameHorizon enables unified comparison across model families (VLMs, UMMs, GUI agents, coding agents, game agents) that previously used incompatible evaluation setups.

2. Reproducible evaluation: The offline MCQ format eliminates environment/harness variance, while the stepwise online protocol provides failure localization beyond aggregate success rates.

3. Data foundation: The first public dataset aligning videos, direct human actions, and dense multi-horizon instructions across 21 AAA titles, exceeding prior corpora in scale (5,000 vs. 300–500 hours).

4. Instruction value: The 7.2-point improvement in future-action planning from multi-horizon instructions validates the utility of dense textual annotations for model training and evaluation.

Conclusion

GameHorizon provides a comprehensive, unified framework for evaluating gameplay capabilities across temporal horizons and model families. The suite's three components—annotator, data, and benchmark—address critical gaps in prior work: narrow game coverage, missing instruction annotations, and non-reproducible online evaluations.

Key Takeaways

1. Task difficulty hierarchy: $T_1$ (action) < $T_2$ (decomposition) < $T_3$ (consistency), with action prediction being the most challenging due to strict correctness criteria.

2. Model capability gaps: Proprietary VLMs lead; open-weight models are catching up; UMMs, GUI agents, and dedicated game agents lag significantly, particularly in generalization to unseen games.

3. Planning bottleneck: Future-action planning and goal decomposition are the primary bottlenecks for current models, not action decoding.

4. Offline-online validity: Offline scores serve as a reliable proxy for actual gameplay capability, with tier-based alignment confirmed in online testing.

Future Directions

The authors will release the dataset, annotator, and benchmark to facilitate future research. The work opens avenues for:

• Training models with multi-horizon instruction supervision
• Developing better goal-decomposition and planning mechanisms
• Extending stepwise online evaluation to more AAA games as game-state access improves
• Investigating the understanding-generation interference in unified multimodal models

The paper positions GameHorizon as "a data foundation and unified yardstick for systematically examining gameplay performance across model families and temporal horizons."