Summary (Overview)

• Core Problem Identification: The paper identifies Sequential Distraction as a fundamental challenge in multi-step omni-modal GUI interaction, where transient and sparse relevance in observations causes compounding errors that lead to irreversible trajectory drift in GUI agents.

• Key Concept Formulation: The authors formalize Focused Perception — a representation requirement that is step-adaptive, action-sufficient, and non-intrusive — distinguishing between what evidence an agent is given versus how its decoder reads it.

• Proposed Solution: FocusGUI, a lightweight adapter that dynamically prioritizes multimodal channels, compresses relevant evidence into a compact latent, and injects it via a zero-initialized residual at the action boundary of a frozen backbone, requiring only action supervision for training.

• Empirical Results: Across four omni-modal backbones, FocusGUI improves Type Match by +33.2 points and Exact Match by 3.0× over frozen baselines on average, and outperforms the strongest existing enhancement method (GUI-Actor) by +30.1 Type Match points.

• Key Insight: The gains grow with interaction depth — while the frozen backbone loses half its Type Match accuracy from first to fourth step, FocusGUI retains 86.8%, confirming the mechanism targets error accumulation rather than serving as a uniformly stronger predictor.

---

Introduction and Theoretical Foundation

Background and Motivation

GUI agents are shifting from static screenshot parsing to acting on dynamic, streaming observations across multiple modalities (screenshots, video, audio, interaction history, device states). The default paradigm exposes complete multimodal observations at every step, assuming the decoder can implicitly determine what matters. However, this assumption fails because:

1. Relevance is transient: The dominant modality switches across 45.3% of adjacent steps, with 94.0% of episodes containing at least one switch
2. Relevance is sparse: The target element occupies a median of only 0.87% of screen area; video content changes by a median of just 7.41% within a step
3. Interference is pervasive: 90.1% of steps contain at least one interfering modality — a modality whose presence reduces the model's likelihood of predicting the correct action

Theoretical Foundation

The paper distinguishes between two failure modes:

• Naive subtraction (input reduction): Existing methods crop, parse, or zoom into screen regions. This fails because (a) decisive signals can come from audio, video, or device states beyond spatial cropping's scope, and (b) cropped-away background often contains essential contextual evidence for grounding coordinates and text parameters.

• Focused perception (attention reallocation): The problem is not that the decoder is given too much, but that it lacks guidance on how to prioritize what it receives. The solution should change how the decoder reads observations, not what it is given.

Sequential Distraction arises because executed actions feed back into context — when the agent misreads relevance at one step, its erroneous action corrupts the observation for all subsequent steps, creating error compounding that drives irreversible trajectory drift.

---

Methodology

Problem Formulation

At interaction step $t$, the agent receives an omni-modal observation:

$$O_t = \{g, \bar{I}_t, V_t, A_t, H_t, S_t\}$$

consisting of user goal $g$, current interface $I_t$, temporal visual observation $V_t$, acoustic observation $A_t$, interaction history $H_t$, and device state $S_t$. The decoder operates on:

$$\widetilde{E}_t = E_t + \mathcal{S}_{J_t}(\gamma R_t), \qquad p_{\Theta_0,\phi}(y_t \mid O_t) = p_{\Theta_0}(y_t \mid \widetilde{E}_t)\tag{1}$$

where the residual $R_t$ is placed at positions $J_t$ and is zero elsewhere, separating observation preservation from decision guidance.

Step-Adaptive Evidence Organization

FocusGUI exposes the observation as separable perceptual channels $X_t = \{x_t^{(m)}\}_{m \in \mathcal{M}}$, projects them into shared space, and scores each against a decision reference:

$$\begin{array}{l} s_t^{(m)} = v^\top \tanh \Big(W_e(e_t^{(m)} + r_m) + W_q q_t\Big), \\ \alpha_t^{(m)} = \frac{\exp(s_t^{(m)}/\tau)}{\sum_{j \in \mathcal{M}} \exp(s_t^{(j)}/\tau)}, \qquad c_t = \sum_{m \in \mathcal{M}} \alpha_t^{(m)} e_t^{(m)}, \end{array}\tag{2}$$

where $r_m$ is a learned channel-identity embedding, $\tau$ is softmax temperature, and $q_t$ is the decision reference derived from the integrated representation of the complete observation.

Compact Focus and Residual Alignment

The prioritized representation is compressed into an action-sufficient latent and aligned with the decoder's native embedding space:

$$R_t = \mathrm{Reshape}(W_p z_t + b_p) \in \mathbb{R}^{K \times d_{model}}, \qquad \widetilde{E}_{t,J_t} = E_{t,J_t} + \gamma R_t.\tag{3}$$

Key design properties:

• Zero-initialized: $W_p$ and $b_p$ start at zero, so FocusGUI begins from the original frozen-model input
• Non-intrusive: The residual is added to existing prompt positions — no text hints appended, no multimodal input removed
• Action-sufficient: The latent need not reconstruct the observation since all evidence remains available through the unmodified pathway

Training Objective

FocusGUI trains from action supervision alone (no modality-relevance labels) via average action-token negative log-likelihood:

$$\mathcal{L}_{act}(\phi) = -\frac{1}{|\mathcal{D}_{tr}|} \sum_{t \in \mathcal{D}_{tr}} \frac{1}{L_t} \sum_{k=1}^{L_t} \log p_{\Theta_0,\phi}\left(y_{t,k} \mid y_{t,<k}, O_t\right).\tag{4}$$

All components — channel projectors, prioritization, compression, and residual alignment — are learned jointly while the backbone, decoder, and LM head remain frozen.

---

Empirical Validation / Results

Experimental Setup

• Dataset: OmniGUI benchmark — 708 episodes, 2,572 action steps across 29 applications
• Backbones: Qwen2.5-Omni, VITA-1.5, MiniCPM-o-4.5, Baichuan-Omni-1.5
• Metrics: Type Match (TM), Exact Match (EM), Goal Progress (GP)
• Baselines: DRS-GUI, OmniParser, RegionFocus (input refinement), GUI-Actor (learned action head)

Main Results (Table 1 Summary)

Average across all four backbones:

Key findings:

• FocusGUI holds the best TM and EM in all 24 backbone-category cells
• Screen refinement baselines (DRS-GUI, OmniParser, RegionFocus) are nearly neutral on TM (+0.9 avg) yet lower EM by 28% and GP by 35% — they make primitives more salient but discard grounding context
• TM gains are largest on Temporal tasks (+37.3 points) where the decisive signal persists across frames, and smallest on Instant tasks (+27.8 points)

Performance Across Interaction Depth

The frozen backbone Qwen2.5-Omni falls from 64.79 TM at step 1 to 32.79 beyond step 3 (retaining only half its accuracy), while FocusGUI retains 86.8% of its initial accuracy. Exact Match rises from 38.73 to 46.72. The EM margin over the backbone increases monotonically from +22.54 to +35.25.

Negative control: On Instant tasks at early steps (little accumulated history), FocusGUI is statistically indistinguishable from the backbone (both 90% confidence intervals include zero). The separation only emerges at deeper steps once interaction history accumulates — the expected signature of a mechanism targeting accumulation.

Component Ablation (Table 2 Summary)

Critical finding: Relative to their effect on single-step accuracy, every component contributes 4–5× more to episode progress than to action type, confirming the design primarily prevents trajectory unravelling.

Learned Priorities and Capacity Sensitivity

• Channel priorities reorder dynamically: Interaction state receives 30.9% of priority mass at opening steps, falling to 7.7% by closing phases; audio and video rise from ~13% and ~11% to ~19% and ~18%, respectively. The screen is never dominant, staying between 10.9% and 12.8% across all phases.
• Capacity sensitivity: Varying K (boundary window) across a 16-fold range changes Overall TM by only 2.6 points. The compact dimension $d_z$ matters more — at $d_z = 32$, EM drops 6.9 points, but from $d_z = 64$ upward the method is stable.

---

Theoretical and Practical Implications

Theoretical Contributions

1. Empirical account of Sequential Distraction: The paper provides the first systematic evidence that relevance in omni-modal GUI interaction is a step-level property — transient across steps and sparse within them — and that per-step misreads compound through action feedback into trajectory drift.

2. Distinction between evidence and reading: The paper articulates why spatial refinement cannot satisfy both action selection and argument grounding simultaneously, and derives from first principles that the solution must take the form of a conditioning signal that reallocates attention over untouched evidence.

3. Action-supervision-only training: Demonstrates that step-adaptive modality prioritization can be learned without modality-relevance labels, purely from next-action supervision — the ordering emerges as a byproduct of optimizing action likelihood.

Practical Implications

• Backbone-agnostic enhancement: FocusGUI introduces only 1.15M–1.45M trainable parameters per backbone and requires no per-backbone hyperparameter tuning, making it practical for deployment across different omni-modal model families.
• Frozen-model preservation: The approach never modifies or fine-tunes the backbone, preserving its native capabilities while adding adaptive focus — valuable in production settings where backbone updates are costly or risky.
• Guiding failure diagnosis: The finding that screen-refinement baselines systematically reduce EM and GP provides a diagnostic framework: methods that make action primitives more salient without preserving grounding context will consistently underperform on exact-match metrics.

---

Conclusion

The paper identifies Sequential Distraction as a primary challenge in multi-step omni-modal GUI interaction, where unfiltered observations cause compounding errors due to transient and sparse relevance. It formalizes Focused Perception as the requirement that representations be step-adaptive, action-sufficient, and non-intrusive, and realizes this in FocusGUI — a lightweight adapter that:

1. Decomposes observations into perceptual channels with source-specific projectors
2. Scores each channel against a step-specific decision reference
3. Compresses prioritized evidence into a compact latent
4. Injects a zero-initialized residual at the action boundary

Trained from action supervision alone (Eq. 4), FocusGUI substantially improves Type Match (+33.2 on average over frozen backbones) and maintains high accuracy at deeper interaction steps where historical context typically degrades. The learned channel priorities reorder dynamically across interaction phases — showing the agent discovers that streaming modalities (audio, video) matter more as tasks progress, while screen content alone is never dominant.

Future direction: Extending the prioritization mechanism to anticipate channel relevance rather than derive it only from the current observation could help mitigate trajectory drift earlier in the interaction sequence.