Summary of "Why Can't I Open My Drawer? Mitigating Object-Driven Shortcuts in Zero-Shot Compositional Action Recognition"

Summary (Overview)

• Core Problem: Zero-Shot Compositional Action Recognition (ZS-CAR) models exhibit object-driven shortcuts—predicting verbs based on the labeled object class rather than temporal evidence—which severely limits generalization to unseen verb–object compositions.
• Root Causes Identified: The authors identify two intertwined factors: (1) sparse and skewed compositional supervision creating strong co-occurrence priors, and (2) asymmetric learning difficulty where objects are easier to learn than verbs (single-frame vs. multi-frame reasoning).
• Diagnostic Metrics Introduced: New metrics—False Seen Prediction (FSP) and False Co-occurrence Prediction (FCP)—quantify how often models collapse unseen inputs to seen (especially frequent) training compositions.
• Proposed Solution (RCORE): A two-component framework—Co-occurrence Prior Regularization (CPR) for synthesizing novel compositions and suppressing frequent co-occurrence priors, and Temporal Order Regularization for Composition (TORC) for enforcing temporal-order sensitivity in verb representations.
• Results: RCORE achieves consistent improvements on Sth-com and a newly curated EK100-com benchmark across multiple backbones (CLIP and InternVideo2), reducing FSP/FCP and improving unseen compositional generalization.

---

Introduction and Theoretical Foundation

Background

Human actions can be decomposed into two semantic primitives—verbs and objects—and robust video understanding requires recognizing each component and reasoning over their interaction as a composition. ZS-CAR formalizes this: a model must recognize unseen verb–object pairs while keeping verb and object vocabularies shared across splits.

The Object-Driven Shortcut Problem

The paper argues that ZS-CAR models often predict verbs via object-driven shortcuts rather than temporal evidence, stemming from two inherent issues:

1. Sparse and skewed compositional supervision: Datasets cover only a small fraction of the combinatorial verb–object space, and training labels concentrate on a limited set of frequent pairs. This induces a strong co-occurrence prior—the empirical dominance of frequent verb–object pairs—which models amplify into shortcuts.

2. Verbs are harder to learn than objects: An object often becomes recognizable from a single frame, whereas a verb requires multi-frame temporal reasoning. Prior work on shortcut learning suggests models tend to rely on easier-to-learn cues, making object cues an attractive shortcut for verb prediction.

Theoretical Basis

The paper draws on shortcut learning theory (Geirhos et al., 2018; Nam et al., 2020), which posits that models favor easier-to-fit spurious correlations over intended evidence. In ZS-CAR, the verb–object difficulty gap makes object cues a particularly appealing shortcut.

ZS-CAR Formal Definition

Let $Y^V$ and $Y^O$ denote the sets of verb and object labels, and each composition label be a pair $y^C = (y^V, y^O) \in Y^V \times Y^O$. The training set is $D_{train} = \{(X_i, y^C_i)\}_{i=1}^N$, and its set of observed compositions is $Y_{seen} = \{y^C_i | (X_i, y^C_i) \in D_{train}\}$. The space of unseen compositions is $Y_{unseen} = (Y^V \times Y^O) \setminus Y_{seen}$.

---

Methodology

Diagnostic Metrics

Training-Bias Metrics

Two misclassification ratios are defined on the unseen subset of an evaluation split:

False Seen Prediction (FSP): The fraction of misclassified unseen composition samples whose predictions $\hat{y}^C$ fall into seen composition categories $Y_{seen}$:

$$FSP = \frac{1}{|E_{unseen}|} \sum_{i \in E_{unseen}} \mathbb{I}(\hat{y}^C_i \in Y_{seen})$$

False Co-occurrence Prediction (FCP): The fraction whose predictions fall into frequent compositions $Y_{freq}$, defined as:

$$Y_{freq} = \{y^C \in Y_{seen} | f(y^C) > \mu_f + \sigma_f\}$$

where $\mu_f$ and $\sigma_f$ are the mean and standard deviation of training frequencies.

Component-wise decomposition: FSP/FCP are decomposed into three cases:

• Verb-collapse: predicted object correct, verb incorrect
• Object-collapse: predicted verb correct, object incorrect
• Dual-collapse: both incorrect

Compositional Gap

$$\Delta_{CG} = Acc_C - (Acc_V \times Acc_O)$$

This measures whether the model benefits from predicting the joint verb–object composition beyond getting the two components correct independently.

RCORE Framework

Feature Extraction

Given a video $X$ with $T$ frames, the backbone encoder outputs frame-level features $F \in \mathbb{R}^{T \times D}$. These are transformed into verb features $F^V \in \mathbb{R}^{T \times D}$ and object features $f^O \in \mathbb{R}^D$ using dedicated encoders.

Co-occurrence Prior Regularization (CPR)

Synthesized composition supervision: Given a training sample $X_i$ with composition label $y^C_i = (y^V_i, y^O_i)$, a new video $\tilde{X}_i$ is constructed by injecting the static object cue from another video $X_j$ (with $y^O_j \neq y^O_i$) into high-motion regions:

$$\tilde{X}^k_i = (1 - \lambda M^k_i) \odot X^k_i + \lambda M^k_i \odot X^{\lfloor T/2 \rfloor}_j$$

where $\lambda \in [0,1]$ controls injection strength and $M^k_i \in \{0,1\}^{H \times W}$ is the high-motion region mask. Soft labels are used:

$$\tilde{y}^O_i = (1-\lambda) y^O_i + \lambda y^O_j$$ $$\tilde{y}^C_i = (1-\lambda) \phi(y^V_i, y^O_i) + \lambda \phi(y^V_i, y^O_j)$$

Co-occurrence prior regularization loss: A margin-based regularizer penalizes frequent seen hard negatives:

$$\mathcal{L}_{CPR} = \sum_{y^C \in Y^{pen}} \max(0, s(y^C) - s(y^C_{i,new}) + m)$$

where $Y^{pen} = Y^{hard,i} \cap Y^{CPR}_{freq}$, with $Y^{hard,i}$ defined as:

$$Y^{hard,i} = \{(y^V, y^O) | ((y^V = y^V_i) \oplus (y^O = y^O_j)) \wedge (y^V, y^O) \neq (y^V_i, y^O_i)\}$$

Batch-adaptive label-space expansion: For each mini-batch, the label set is expanded as $Y^{exp} = Y^{seen} \cup Y^{new}$, avoiding the instability of full open-world optimization while injecting supervision for new compositions.

Temporal Order Regularization for Composition (TORC)

Temporal perturbation: Given frame features $F = (f_1, \ldots, f_T)$, two views are created:

• Reversed sequence $F^{rev} = (f_T, \ldots, f_1)$
• Shuffled sequence $F^{shuffled} = \pi(F)$ where $\pi$ is a random permutation

TORC loss: Two components:

1. Cosine similarity loss pushing forward and reversed features apart:

$$\mathcal{L}_{cos} = \frac{f^V \cdot f^V_{rev}}{\|f^V\|\|f^V_{rev}\|}$$

2. Negative entropy loss penalizing confident predictions on shuffled inputs:

$$\mathcal{L}_{ent} = \sum_{m=1}^{|Y^V|} p_m \log p_m$$

where $p_m = \frac{\exp(\cos(f^V_{shuffled}, e^V_m)/\tau)}{\sum_{n=1}^{|Y^V|} \exp(\cos(f^V_{shuffled}, e^V_n)/\tau)}$

Total TORC loss: $\mathcal{L}_{TORC} = \mathcal{L}_{cos} + \mathcal{L}_{ent}$

Total Loss

$$\mathcal{L}_{total} = \alpha \mathcal{L}_{com} + \beta \mathcal{L}_{comp} + \gamma \mathcal{L}_{TORC} + \delta \mathcal{L}_{CPR}$$

---

Empirical Validation / Results

Datasets

• Sth-com: Derived from Something-Something V2 and Something-Else, 79K videos with 161 verbs and 248 objects. Label coverage ratio: 12.8%
• EK100-com (newly introduced): Repurposed from EPIC-KITCHENS-100, 71K egocentric videos, 81 verbs, 216 objects. Label coverage ratio: 7.5% (more severe compositional sparsity)

Main Results on Sth-com (Open-World Setting)

CLIP Backbone:

InternVideo2 Backbone:

Main Results on EK100-com

CLIP Backbone:

InternVideo2 Backbone:

Diagnostic Analysis Results

• RCORE suppresses FSP/FCP growth during training (baseline: 53%→63% FSP; RCORE: 47%→44%)
• RCORE reduces the seen–unseen accuracy gap (9 points vs. 17 points for baseline)
• On temporal shuffling, RCORE shows a larger performance drop, indicating stronger temporal grounding
• Verb confusion matrices show RCORE distinguishes opposite verbs (e.g., unfolding vs. folding) better than baseline

Ablation Studies

Combining both components yields the best overall results, with complementary benefits.

---

Theoretical and Practical Implications

Theoretical Contributions

• Diagnostic framework: FSP/FCP and $\Delta_{CG}$ provide a systematic way to quantify shortcut reliance in compositional video understanding, applicable beyond ZS-CAR
• Understanding of failure modes: The work reveals that even strong video-pretrained backbones (InternVideo2) exhibit persistent co-occurrence-driven shortcuts, showing that stronger pretraining alone is insufficient
• The role of asymmetric difficulty: Demonstrates that the verb–object difficulty gap is a structural issue inherent to ZS-CAR, not merely a data artifact

Practical Contributions

• RCORE framework: A training-time regularization approach that is backbone-agnostic, transferable across different VLM architectures
• New benchmark (EK100-com): A more challenging dataset with severe compositional sparsity (7.5% coverage), enabling future research
• Open-world evaluation protocol: Adopts a more realistic evaluation setting that spans the full compositional space without test-set-tuned bias calibration

---

Conclusion

This work identifies object-driven shortcuts as a key failure mode in Zero-Shot Compositional Action Recognition, attributing them to compositional sparsity and asymmetric verb–object learning difficulty. Using diagnostic metrics (FSP/FCP, $\Delta_{CG}$), the authors demonstrate that existing models overfit to co-occurrence priors, degrading verb learning and generalization to unseen compositions.

The proposed RCORE framework combines:

• CPR: Expanding supervision over synthesized compositions and suppressing frequent co-occurrence priors through margin-based hard-negative penalties
• TORC: Enforcing temporal-order sensitivity to learn temporally grounded verb representations

Across two benchmarks (Sth-com and EK100-com) under an open-world evaluation protocol, RCORE consistently improves unseen composition performance, achieving positive compositional gaps on unseen compositions where all baselines remain negative.

Future directions include:

• Exploring decoder-based VLM interfaces that may require different regularization strategies
• Investigating whether the diagnostic framework extends to other compositional video understanding tasks
• Scaling RCORE to larger video foundation models (e.g., InternVideo2-1B)