# Why Can't I Open My Drawer? Mitigating Object-Driven Shortcuts in Zero-Shot Compositional Action Recognition

> RCORE's co-occurrence and temporal-order regularizations eliminate object-driven shortcuts in zero-shot compositional action recognition, improving unseen verb-object generalization across CLIP and InternVideo2 backbones.

- **Source:** [arXiv](https://arxiv.org/abs/2601.16211)
- **Published:** 2026-07-11
- **Permalink:** https://picx.dev/p/HK96dk

## Summary

# Summary of "Why Can't I Open My Drawer? Mitigating Object-Driven Shortcuts in Zero-Shot Compositional Action Recognition"

## Summary (Overview)

- **Core Problem**: Zero-Shot Compositional Action Recognition (ZS-CAR) models exhibit **object-driven shortcuts**—predicting verbs based on the labeled object class rather than temporal evidence—which severely limits generalization to unseen verb–object compositions.
- **Root Causes Identified**: The authors identify two intertwined factors: (1) **sparse and skewed compositional supervision** creating strong co-occurrence priors, and (2) **asymmetric learning difficulty** where objects are easier to learn than verbs (single-frame vs. multi-frame reasoning).
- **Diagnostic Metrics Introduced**: New metrics—**False Seen Prediction (FSP)** and **False Co-occurrence Prediction (FCP)**—quantify how often models collapse unseen inputs to seen (especially frequent) training compositions.
- **Proposed Solution (RCORE)**: A two-component framework—**Co-occurrence Prior Regularization (CPR)** for synthesizing novel compositions and suppressing frequent co-occurrence priors, and **Temporal Order Regularization for Composition (TORC)** for enforcing temporal-order sensitivity in verb representations.
- **Results**: RCORE achieves consistent improvements on Sth-com and a newly curated **EK100-com** benchmark across multiple backbones (CLIP and InternVideo2), reducing FSP/FCP and improving unseen compositional generalization.

---

## Introduction and Theoretical Foundation

### Background
Human actions can be decomposed into two semantic primitives—**verbs** and **objects**—and robust video understanding requires recognizing each component and reasoning over their interaction as a composition. ZS-CAR formalizes this: a model must recognize unseen verb–object pairs while keeping verb and object vocabularies shared across splits.

### The Object-Driven Shortcut Problem
The paper argues that ZS-CAR models often predict verbs via **object-driven shortcuts** rather than temporal evidence, stemming from two inherent issues:

1. **Sparse and skewed compositional supervision**: Datasets cover only a small fraction of the combinatorial verb–object space, and training labels concentrate on a limited set of frequent pairs. This induces a strong **co-occurrence prior**—the empirical dominance of frequent verb–object pairs—which models amplify into shortcuts.

2. **Verbs are harder to learn than objects**: An object often becomes recognizable from a single frame, whereas a verb requires multi-frame temporal reasoning. Prior work on shortcut learning suggests models tend to rely on easier-to-learn cues, making object cues an attractive shortcut for verb prediction.

### Theoretical Basis
The paper draws on **shortcut learning theory** (Geirhos et al., 2018; Nam et al., 2020), which posits that models favor easier-to-fit spurious correlations over intended evidence. In ZS-CAR, the verb–object difficulty gap makes object cues a particularly appealing shortcut.

### ZS-CAR Formal Definition
Let $Y^V$ and $Y^O$ denote the sets of verb and object labels, and each composition label be a pair $y^C = (y^V, y^O) \in Y^V \times Y^O$. The training set is $D_{train} = \{(X_i, y^C_i)\}_{i=1}^N$, and its set of observed compositions is $Y_{seen} = \{y^C_i | (X_i, y^C_i) \in D_{train}\}$. The space of unseen compositions is $Y_{unseen} = (Y^V \times Y^O) \setminus Y_{seen}$.

---

## Methodology

### Diagnostic Metrics

#### Training-Bias Metrics
Two misclassification ratios are defined on the unseen subset of an evaluation split:

**False Seen Prediction (FSP)**: The fraction of misclassified unseen composition samples whose predictions $\hat{y}^C$ fall into seen composition categories $Y_{seen}$:

$$FSP = \frac{1}{|E_{unseen}|} \sum_{i \in E_{unseen}} \mathbb{I}(\hat{y}^C_i \in Y_{seen})$$

**False Co-occurrence Prediction (FCP)**: The fraction whose predictions fall into *frequent* compositions $Y_{freq}$, defined as:

$$Y_{freq} = \{y^C \in Y_{seen} | f(y^C) > \mu_f + \sigma_f\}$$

where $\mu_f$ and $\sigma_f$ are the mean and standard deviation of training frequencies.

**Component-wise decomposition**: FSP/FCP are decomposed into three cases:
- *Verb-collapse*: predicted object correct, verb incorrect
- *Object-collapse*: predicted verb correct, object incorrect
- *Dual-collapse*: both incorrect

#### Compositional Gap
$$\Delta_{CG} = Acc_C - (Acc_V \times Acc_O)$$

This measures whether the model benefits from predicting the joint verb–object composition beyond getting the two components correct independently.

### RCORE Framework

#### Feature Extraction
Given a video $X$ with $T$ frames, the backbone encoder outputs frame-level features $F \in \mathbb{R}^{T \times D}$. These are transformed into verb features $F^V \in \mathbb{R}^{T \times D}$ and object features $f^O \in \mathbb{R}^D$ using dedicated encoders.

#### Co-occurrence Prior Regularization (CPR)

**Synthesized composition supervision**: Given a training sample $X_i$ with composition label $y^C_i = (y^V_i, y^O_i)$, a new video $\tilde{X}_i$ is constructed by injecting the static object cue from another video $X_j$ (with $y^O_j \neq y^O_i$) into high-motion regions:

$$\tilde{X}^k_i = (1 - \lambda M^k_i) \odot X^k_i + \lambda M^k_i \odot X^{\lfloor T/2 \rfloor}_j$$

where $\lambda \in [0,1]$ controls injection strength and $M^k_i \in \{0,1\}^{H \times W}$ is the high-motion region mask. Soft labels are used:

$$\tilde{y}^O_i = (1-\lambda) y^O_i + \lambda y^O_j$$
$$\tilde{y}^C_i = (1-\lambda) \phi(y^V_i, y^O_i) + \lambda \phi(y^V_i, y^O_j)$$

**Co-occurrence prior regularization loss**: A margin-based regularizer penalizes frequent seen hard negatives:

$$\mathcal{L}_{CPR} = \sum_{y^C \in Y^{pen}} \max(0, s(y^C) - s(y^C_{i,new}) + m)$$

where $Y^{pen} = Y^{hard,i} \cap Y^{CPR}_{freq}$, with $Y^{hard,i}$ defined as:

$$Y^{hard,i} = \{(y^V, y^O) | ((y^V = y^V_i) \oplus (y^O = y^O_j)) \wedge (y^V, y^O) \neq (y^V_i, y^O_i)\}$$

**Batch-adaptive label-space expansion**: For each mini-batch, the label set is expanded as $Y^{exp} = Y^{seen} \cup Y^{new}$, avoiding the instability of full open-world optimization while injecting supervision for new compositions.

#### Temporal Order Regularization for Composition (TORC)

**Temporal perturbation**: Given frame features $F = (f_1, \ldots, f_T)$, two views are created:
- Reversed sequence $F^{rev} = (f_T, \ldots, f_1)$
- Shuffled sequence $F^{shuffled} = \pi(F)$ where $\pi$ is a random permutation

**TORC loss**: Two components:
1. **Cosine similarity loss** pushing forward and reversed features apart:
$$\mathcal{L}_{cos} = \frac{f^V \cdot f^V_{rev}}{\|f^V\|\|f^V_{rev}\|}$$

2. **Negative entropy loss** penalizing confident predictions on shuffled inputs:
$$\mathcal{L}_{ent} = \sum_{m=1}^{|Y^V|} p_m \log p_m$$

where $p_m = \frac{\exp(\cos(f^V_{shuffled}, e^V_m)/\tau)}{\sum_{n=1}^{|Y^V|} \exp(\cos(f^V_{shuffled}, e^V_n)/\tau)}$

Total TORC loss: $\mathcal{L}_{TORC} = \mathcal{L}_{cos} + \mathcal{L}_{ent}$

#### Total Loss
$$\mathcal{L}_{total} = \alpha \mathcal{L}_{com} + \beta \mathcal{L}_{comp} + \gamma \mathcal{L}_{TORC} + \delta \mathcal{L}_{CPR}$$

---

## Empirical Validation / Results

### Datasets
- **Sth-com**: Derived from Something-Something V2 and Something-Else, 79K videos with 161 verbs and 248 objects. Label coverage ratio: **12.8%**
- **EK100-com** (newly introduced): Repurposed from EPIC-KITCHENS-100, 71K egocentric videos, 81 verbs, 216 objects. Label coverage ratio: **7.5%** (more severe compositional sparsity)

### Main Results on Sth-com (Open-World Setting)

**CLIP Backbone**:

| Method | Verb@Unseen | Object@Unseen | Composition Unseen | Composition H.M. |
|--------|-------------|---------------|--------------------|-------------------|
| C2C [19] | 54.36 | 56.10 | 30.08 (−0.42) | 36.47 |
| RCORE (Ours) | **59.00** | 56.34 | **33.90 (+0.66)** | **38.67** |

**InternVideo2 Backbone**:

| Method | Verb@Unseen | Object@Unseen | Composition Unseen | Composition H.M. |
|--------|-------------|---------------|--------------------|------------------|
| C2C [19] | 63.29 | 63.44 | 39.53 (−0.63) | 44.73 |
| RCORE | **66.65** | 64.56 | **43.98 (+0.95)** | **46.88** |

### Main Results on EK100-com

**CLIP Backbone**:

| Method | Verb@Unseen | Object@Unseen | Composition Unseen | Composition H.M. |
|--------|-------------|---------------|--------------------|------------------|
| C2C [19] | 49.57 | 46.99 | 21.56 (−1.73) | 28.65 |
| RCORE | **54.31** | 49.88 | **28.41 (+1.32)** | **33.12** |

**InternVideo2 Backbone**:

| Method | Verb@Unseen | Object@Unseen | Composition Unseen | Composition H.M. |
|--------|-------------|---------------|--------------------|------------------|
| C2C [19] | 57.01 | 52.31 | 30.33 (+0.51) | 36.96 |
| RCORE | **59.78** | 56.00 | **37.31 (+3.83)** | **39.08** |

### Diagnostic Analysis Results

- RCORE suppresses FSP/FCP growth during training (baseline: 53%→63% FSP; RCORE: 47%→44%)
- RCORE reduces the seen–unseen accuracy gap (9 points vs. 17 points for baseline)
- On temporal shuffling, RCORE shows a larger performance drop, indicating stronger temporal grounding
- Verb confusion matrices show RCORE distinguishes opposite verbs (e.g., unfolding vs. folding) better than baseline

### Ablation Studies

| CPR | TORC | Verb@Unseen | Composition Unseen | H.M. |
|-----|------|-------------|--------------------|------|
| ✗ | ✗ | 54.02 | 30.14 | 36.83 |
| ✓ | ✗ | 54.05 | 33.22 | 37.51 |
| ✗ | ✓ | 57.49 | 33.92 | 39.15 |
| ✓ | ✓ | **58.13** | **35.16** | **39.48** |

Combining both components yields the best overall results, with complementary benefits.

---

## Theoretical and Practical Implications

### Theoretical Contributions
- **Diagnostic framework**: FSP/FCP and $\Delta_{CG}$ provide a systematic way to quantify shortcut reliance in compositional video understanding, applicable beyond ZS-CAR
- **Understanding of failure modes**: The work reveals that even strong video-pretrained backbones (InternVideo2) exhibit persistent co-occurrence-driven shortcuts, showing that stronger pretraining alone is insufficient
- **The role of asymmetric difficulty**: Demonstrates that the verb–object difficulty gap is a structural issue inherent to ZS-CAR, not merely a data artifact

### Practical Contributions
- **RCORE framework**: A training-time regularization approach that is **backbone-agnostic**, transferable across different VLM architectures
- **New benchmark (EK100-com)**: A more challenging dataset with severe compositional sparsity (7.5% coverage), enabling future research
- **Open-world evaluation protocol**: Adopts a more realistic evaluation setting that spans the full compositional space without test-set-tuned bias calibration

---

## Conclusion

This work identifies **object-driven shortcuts** as a key failure mode in Zero-Shot Compositional Action Recognition, attributing them to compositional sparsity and asymmetric verb–object learning difficulty. Using diagnostic metrics (FSP/FCP, $\Delta_{CG}$), the authors demonstrate that existing models overfit to co-occurrence priors, degrading verb learning and generalization to unseen compositions.

The proposed **RCORE** framework combines:
- **CPR**: Expanding supervision over synthesized compositions and suppressing frequent co-occurrence priors through margin-based hard-negative penalties
- **TORC**: Enforcing temporal-order sensitivity to learn temporally grounded verb representations

Across two benchmarks (Sth-com and EK100-com) under an open-world evaluation protocol, RCORE consistently improves unseen composition performance, achieving positive compositional gaps on unseen compositions where all baselines remain negative.

**Future directions** include:
- Exploring decoder-based VLM interfaces that may require different regularization strategies
- Investigating whether the diagnostic framework extends to other compositional video understanding tasks
- Scaling RCORE to larger video foundation models (e.g., InternVideo2-1B)

---

_Markdown view of https://picx.dev/p/HK96dk, served by PicX — AI-generated visual whiteboard summaries of research papers._