# GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay

> GameHorizon provides a unified multi-horizon benchmark revealing that goal decomposition and future-action planning, not action decoding, are the primary bottlenecks in current gameplay AI models.

- **Source:** [arXiv](https://arxiv.org/abs/2609.25001)
- **Published:** 2026-09-23
- **Permalink:** https://picx.dev/p/rtCkaH
- **Whiteboard:** https://picx.dev/p/rtCkaH/image

## Summary

# GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay

## Summary (Overview)

- **GameHorizon** is a unified data and evaluation suite for measuring gameplay capabilities across multiple temporal horizons (short-horizon operations, medium-horizon goals, long-horizon strategies) in AAA video games.
- The suite comprises three components: **GameHorizon-Annotator** (an automated annotation pipeline), **GameHorizon-Data** (5,000 hours of gameplay from 21 games collected by 100 expert players with 6.18M multi-horizon instructions), and **GameHorizon-Bench** (reproducible offline + stepwise online evaluation).
- The benchmark evaluates **47 models** across five families (VLMs, UMMs, GUI agents, coding agents, game agents) through **over one million model invocations**, revealing a meaningful task-difficulty hierarchy and pronounced capability differences.
- Key finding: future-action planning and goal decomposition are more challenging than current-action perception; multi-horizon instructions improve future-action planning by **7.2 percentage points** over vision-only inputs.
- The offline track shows a clear positive association with online gameplay performance, validating offline scores as a proxy for actual gameplay capability.

## Introduction and Theoretical Foundation

### Background and Motivation

Modern video games provide a measurable testbed for AI models, requiring simultaneous abilities in:
1. **Visual understanding** – interpreting complex game environments
2. **Instruction decomposition** – breaking down natural-language goals
3. **Goal planning** – organizing subgoals across temporal scales
4. **Precise action control** – executing keyboard/mouse actions

Game objectives span varying temporal scales: collecting an item within seconds, winning a fight lasting minutes, or executing a strategy over an entire session. These objectives are all reflected in the same stream of primitive actions (keystrokes, mouse movements).

### Limitations of Existing Work

The paper identifies three critical gaps in prior datasets and benchmarks:

1. **Narrow game coverage**: GameWorld targets simple mini-games; STEVE-1, MineDojo, and VPT are confined to Minecraft; WildWorld uses only Monster Hunter Wilds. Conclusions from single titles cannot generalize to heterogeneous AAA games.

2. **Lack of language instructions**: NitroGen and GameVerse omit instructions entirely; Open-P2P provides sparse annotations; Game-TARS relies on costly manual annotation. Instructions are vital for human–agent interaction, including instruction following and goal decomposition.

3. **High-variance online evaluation**: Lumine reports success rates from only three trials per scene; GameVerse conducts 3–20 rollouts. Small samples lead to low-confidence comparisons, and aggregate success rates conflate distinct failure modes (action misidentification vs. goal inference vs. planning errors).

### Key Insight

The paper observes that gameplay capability splits across two model families:
- **Dedicated game agents** (Open-P2P, JARVIS-VLA) are optimized for acting with lightweight VLA architectures but lack reasoning/planning capacity.
- **General-purpose models** (Gemini, GPT, Claude) excel at planning but lack systematic action evaluation.

A single standardized yardstick is needed to measure both families across vision, executable actions, and multi-horizon natural-language goals.

## Methodology

### GameHorizon-Annotator

The annotation pipeline operates **bottom-up** to construct a three-level instruction pyramid:

**1. Action-Aware Video Segmentation:**
- Raw keyboard-mouse events are mapped to game-specific action semantics (e.g., Shift = sprinting in Cyberpunk 2077)
- Action streams are scanned chronologically, grouping sustained events and using discrete events (jumping, attacking) as initial boundaries
- A VLM (Gemini) examines video frames to disambiguate action semantics (e.g., left-click as attack vs. item selection) and refine segmentation

**2. Bottom-Up Temporal Merging:**
- Adjacent $L_1$ clips are merged into $L_2$ clips based on action continuity and semantic coherence
- $L_2$ clips are merged into $L_3$ clips based on goal continuity
- Dynamic programming enforces level-specific durations:
  - $L_1$: 1–5 seconds (short-horizon operations)
  - $L_2$: 1–2 minutes (medium-horizon goals)
  - $L_3$: 5–8 minutes (long-horizon strategies)

**3. Multi-Horizon Instruction Annotation:**
- **$L_1$**: VLM receives sampled frames + keyboard-mouse actions → generates action-grounded operation descriptions with coordinates, object descriptions, and spatial relations
- **$L_2$**: VLM receives frames + actions + constituent $L_1$ instructions → synthesizes medium-horizon goal descriptions
- **$L_3$**: VLM receives frames + $L_2$ instructions (no actions) → generates high-level strategy descriptions

All prompts include game title, clip duration, frame rate, and game-specific keybinds for action interpretation.

### GameHorizon-Data

| Property | Value |
|----------|-------|
| Total recordings | 5,000 hours (4,341 valid after filtering) |
| Game titles | 21 (open-world, ARPG, shooter, sandbox, creature-collecting) |
| Players | 100 experienced human experts |
| Videos | 4,571 at 60 fps, 2K resolution |
| Action events | 411.03 million (22.84 events/second average) |
| Total instructions | 6,184,036 |
| $L_1$ instructions | 5,947,588 (avg span 2.63s) |
| $L_2$ instructions | 189,158 (avg span 82.6s) |
| $L_3$ instructions | 47,290 (avg span 330.4s) |

Key advantage: **direct human actions** (synchronized keyboard-mouse) rather than IDM-inferred pseudo-labels, ensuring faithful action trajectories.

### GameHorizon-Bench

**Offline Track** – 5,000 standardized MCQs:
- **$T_1$ Single-Horizon Action** (1,000 questions): Given frames + $L_1$ instruction, determine correct action sequence (evaluates action perception)
- **$T_2$ Multi-Horizon Decomposition** (1,000 questions): Given $L_2$ goal, identify ordered sequence of $L_1$ operations (evaluates goal decomposition and planning)
- **$T_3$ Cross-Horizon Consistency** (1,000 questions): Assess consistency across frames, $L_1$–$L_3$ instructions, and actions
- **10 Variant Tasks** (200 questions each): Diagnostic probes including:
  - $T_1^*$: Current-action perception vs. future-action planning under three input settings
  - $T_2^*$: Top-down decomposition vs. bottom-up abstraction
  - $T_3^*$: Instruction-only vs. instruction-and-action options

**Online Track** – Stepwise evaluation in Minecraft (due to game-state accessibility):
- 10 causal tasks (order-dependent subtasks) + 10 thematic tasks (order-flexible subtasks)
- 62 verifiable short-horizon subtasks total
- Environment reset on failure enables **failure localization** to specific steps
- Long-horizon task passed only when all subtasks succeed

## Empirical Validation / Results

### Offline Primary Tasks

**Overall benchmark statistics:**
- Mean accuracy: **64.7%** (vs. 25% random baseline)
- Range: 44.6% to 80.2% (35.6-point spread across models)
- Task difficulty hierarchy: $T_1$ = 57.3% < $T_2$ = 65.1% < $T_3$ = 71.6%

**Top-performing models (Tier 1):**

| Rank | Model | $T_1$ | $T_2$ | $T_3$ | Overall |
|------|-------|-------|-------|-------|---------|
| 1 | GPT-6-Astra | 69.4 | 79.6 | 91.5 | **80.2** |
| 2 | Gemini 3.8 Flash | 65.9 | 81.2 | 84.8 | 77.3 |
| 3 | Gemini 3.7 Flash | 66.2 | 80.1 | 83.9 | 76.7 |
| 4 | Gemini 3.6 Flash | 62.3 | 79.1 | 84.6 | 75.3 |
| 5 | GPT-5.6 Sol | 65.3 | 77.5 | 81.5 | 74.8 |

**Key findings:**
- Proprietary models dominate (10/11 Tier 1 positions); Kimi-K3 is the only open-weight model in Tier 1
- **UMMs** (BAGEL-7B-MoT, SenseNova-U1-8B-MoT, Ovis-U1-3B, InternVL-U-4B) all fall in Tiers 3–4, suggesting understanding-generation unification introduces interference
- **GUI agents** (GELab-Zero-4B-Preview, UI-TARS-1.5-7B) show limited transfer from computer-use to gameplay
- **Coding agents** (Claude Sonnet 5, Opus 4.8, Fable 5) generalize well, with Fable 5 reaching Tier 2 (71.2%)
- **Dedicated game agents** perform near random baseline (22.0–28.1%) on unseen AAA games; JARVIS-VLA drops from 54.2% in-domain (Minecraft) to 28.1% zero-shot (AAA)

**Effects of thinking:**
- Improvement: GLM-5V-Turbo +5.6 pts, Doubao-Seed-2.1-Pro +9.3 pts (gains concentrated on $T_2$: +9.0 and +19.9 pts)
- Degradation: UI-TARS-1.5-7B −8.2 pts, Qwen2.5-VL-7B −5.9 pts (lightweight models hallucinate reasoning)

### Offline Variant Tasks

**Action perception vs. future planning ($T_1^*$):**

| Setting | Perception | Planning | Gap |
|---------|-----------|----------|-----|
| Frames only | 44.5% | 34.6% | 9.9 |
| + Short instruction | 58.6% (+14.1) | 34.8% (+0.2) | 23.8 |
| + Multi-horizon | 60.0% (+1.4) | 41.8% (+7.0) | 18.2 |

**Key insight:** Short-horizon instructions help current-action perception (+14.1 pts); multi-horizon instructions help future-action planning (+7.0 pts).

**Top-down vs. bottom-up ($T_2^*$):**
- Bottom-up abstraction: **98.4%** accuracy (near ceiling)
- Top-down decomposition: **69.5%** accuracy (28.9-point gap)
- Validates bottom-up design for annotation, top-down for evaluation

**Action-decoding error ($T_3^*$):**
- Adding actions to consistency task: only −0.6 pts (74.0% → 73.4%)
- Primary bottleneck is goal decomposition/planning, not action decoding

### Online Track Results

| Offline Tier | Online Rank | Model | Subtask Success | Task Success |
|-------------|-------------|-------|-----------------|--------------|
| 1 | 1 | GPT-6-Astra | 66.1% (41/62) | **45.0%** (9/20) |
| 1 | 2 | Gemini 3.6 Flash | 56.5% (35/62) | 30.0% (6/20) |
| 1 | 3 | Kimi-K3 | 46.8% (29/62) | 10.0% (2/20) |
| 2 | 4–6 | GPT-5.6 Terra, GPT-5.6 Luna, MiniMax-M3 | 29.0–37.1% | 10.0% |
| 3 | 7–9 | GLM-5V-Turbo, Qwen3.5-397B, Step3-VL-10B | 14.5–27.4% | 5.0% |
| 4 | 10–12 | Qwen3.6-35B, InternVL3.5-8B, UI-TARS-1.5-7B | 1.6–11.3% | 0–5.0% |

**Key findings:**
- Clear positive association between offline ranks and online performance (Tiers 1–4 map to online ranks 1–3, 4–6, 7–9, 10–12)
- Long-horizon tasks remain extremely difficult: 10/12 models achieve ≤10% task success
- Stepwise protocol enables failure localization (e.g., distinguishing models that don't know how to craft a cake from those that understand but execute incorrectly)

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Multi-horizon instruction pyramid**: The paper demonstrates that gameplay capability must be measured across temporal scales, with distinct instruction types serving different purposes—short-horizon for action grounding, medium-horizon for goal decomposition, long-horizon for strategy formulation.

2. **Bottom-up annotation vs. top-down evaluation**: The 28.9-point gap between abstraction (98.4%) and decomposition (69.5%) reveals an information asymmetry: concrete actions determine goals more reliably than abstract goals determine actions. This justifies the design principle of bottom-up data annotation and top-down benchmark evaluation.

3. **Bottleneck localization**: The finding that action decoding contributes only −0.6 pts while goal decomposition/planning dominates errors identifies the primary capability gap in current models.

4. **Understanding-generation trade-off**: UMMs underperforming their understanding backbones (InternVL-U-4B vs. InternVL3.5-2B, −1.4 pts) suggests interference in unified architectures.

### Practical Implications

1. **Standardized yardstick**: GameHorizon enables unified comparison across model families (VLMs, UMMs, GUI agents, coding agents, game agents) that previously used incompatible evaluation setups.

2. **Reproducible evaluation**: The offline MCQ format eliminates environment/harness variance, while the stepwise online protocol provides failure localization beyond aggregate success rates.

3. **Data foundation**: The first public dataset aligning videos, direct human actions, and dense multi-horizon instructions across 21 AAA titles, exceeding prior corpora in scale (5,000 vs. 300–500 hours).

4. **Instruction value**: The 7.2-point improvement in future-action planning from multi-horizon instructions validates the utility of dense textual annotations for model training and evaluation.

## Conclusion

GameHorizon provides a comprehensive, unified framework for evaluating gameplay capabilities across temporal horizons and model families. The suite's three components—annotator, data, and benchmark—address critical gaps in prior work: narrow game coverage, missing instruction annotations, and non-reproducible online evaluations.

### Key Takeaways

1. **Task difficulty hierarchy**: $T_1$ (action) < $T_2$ (decomposition) < $T_3$ (consistency), with action prediction being the most challenging due to strict correctness criteria.

2. **Model capability gaps**: Proprietary VLMs lead; open-weight models are catching up; UMMs, GUI agents, and dedicated game agents lag significantly, particularly in generalization to unseen games.

3. **Planning bottleneck**: Future-action planning and goal decomposition are the primary bottlenecks for current models, not action decoding.

4. **Offline-online validity**: Offline scores serve as a reliable proxy for actual gameplay capability, with tier-based alignment confirmed in online testing.

### Future Directions

The authors will release the dataset, annotator, and benchmark to facilitate future research. The work opens avenues for:
- Training models with multi-horizon instruction supervision
- Developing better goal-decomposition and planning mechanisms
- Extending stepwise online evaluation to more AAA games as game-state access improves
- Investigating the understanding-generation interference in unified multimodal models

The paper positions GameHorizon as "a data foundation and unified yardstick for systematically examining gameplay performance across model families and temporal horizons."

---

_Markdown view of https://picx.dev/p/rtCkaH, served by PicX — AI-generated visual whiteboard summaries of research papers._