Summary (Overview)

  • MPIE-Bench introduces a 2,500-sample benchmark of video-mined editing triplets spanning 405 scenes, 14 interaction categories, and four contact densities (C0–C3), designed to evaluate multi-person interaction editing with anatomically plausible body coherence.
  • MPIE-Eval proposes two novel evaluation axes—Anatomy (whether every human-like mass is explained by complete reconstructed bodies) and Interaction (whether penetration and surface distance between bodies match the requested contact)—derived from frozen multi-person mesh reconstruction rather than VLM opinion.
  • Across ten editors, mesh Anatomy tops out at 0.65 and mesh Interaction at 0.72, while VLM checklists rate the same images above 0.95, demonstrating that existing evaluations saturate on interaction while missing obvious anatomical failures.
  • A five-rater human consistency study confirms that both mesh axes track human judgment more closely than a zero-shot VLM judge, and the ranking holds under ablation of every weight and threshold.

Introduction and Theoretical Foundation

Text-to-image and personalized editing models now synthesize high-fidelity single-subject images with ease. However, placing multiple named people into shared contact actions such as embrace, carry, or grapple still exposes major failures: fused limbs, invented extremities, and interpenetrating bodies. Existing evaluations largely overlook these anatomical and geometric issues.

Current benchmarks for multi-person and personalized generation ask how many people appear, whether their identities match the references, whether the requested action is present, and how pleasing the result looks. An image whose bodies are anatomically impossible satisfies all four criteria. The suites that do ask about interaction often delegate that judgment to a vision-language model (VLM), which in experiments rates the strongest closed-source models between 0.98 and 0.99 on interaction quality for images whose fused limbs are obvious to human readers.

The authors identify a critical gap: whether the requested edit happened and whether the bodies survived it are different questions, and only the first is being asked. This motivates the need for MPIE-Bench and MPIE-Eval.

Methodology

Data Construction (MPIE-Bench)

The benchmark is constructed through a licensing-aware mining pipeline:

  1. Sources: Multi-person interaction videos from three open sources: Pexels stock footage, Harmony4D, and CHI3D.

  2. Input–output selection: Contact-sparse frames yield clean per-person references; contact-dense frames yield interaction targets. For each video, a contact-density curve ϕ(t)\phi(t) is computed from person-box IoU, pose proximity, optical flow, and occlusion cues, with refine mix 0.35/0.30/0.20/0.150.35 / 0.30 / 0.20 / 0.15 after within-video percentile norm.

  3. Prompt reverse writing: Given valley references {Rk}\{R_k\} and the peak target, a VLM reverse-captions an edit instruction that names roles, the shared action, and contact intent, without revealing target pixels at test time.

  4. Human review: Automatic mining yields ~20,000 candidates. Adult reviewers remove unsafe or off-task clips, check that each reverse-captioned prompt names the visible interaction, then de-duplicate and cap per scene to freeze 2,500 targets across 405 scenes.

Evaluation Protocol (MPIE-Eval)

MPIE-Eval reports six axes separately and never blends them into one scalar:

  • Count: Binary person-count correctness from a shared VLM checklist.
  • Identity: Occlusion-aware face matching to frozen reference embeddings.
  • Instruction: A separate pipeline with frozen atomic QA sets authored from edit text alone.
  • Quality: Standard human-preference aesthetic score.
  • Anatomy: Whether every human-like region in the image is explained by a complete set of people.
  • Interaction: Whether reconstructed contact geometry matches the prompt's contact intent.

Anatomy Scoring

Failures are grouped into four families:

  • Extra/fused limbs: mass glued onto a person that should not be there.
  • Floating parts: detached leftover blobs that belong to no body.
  • Body structure: broken scale, part ownership, or self-collision on the meshes.
  • Residual mass: leftover human-like foreground not covered above.

The leftover fraction is computed as:

ℓ=∣Mfg∩R∖Mex∣∣Mfg∩R∣\ell = \frac{|M_{fg} \cap R \setminus M_{ex}|}{|M_{fg} \cap R|}

Each diagnostic maps through the same soft ramp:

s(x;a,b)=clip(b−xb−a,0,1)s(x; a, b) = \text{clip}\left(\frac{b - x}{b - a}, 0, 1\right)

The final Anatomy score is:

Sanatmesh=0.40sattach+0.20sorphan+0.25sstruct+0.15sresidS_{\text{anat}}^{\text{mesh}} = 0.40 s_{\text{attach}} + 0.20 s_{\text{orphan}} + 0.25 s_{\text{struct}} + 0.15 s_{\text{resid}}

Interaction Scoring

For every kept mesh pair, two measures are computed:

  • Convex-hull penetration proxy: Vp=rin⋅min⁡(Volhull(A),Volhull(B))V_p = r_{in} \cdot \min(\text{Vol}_{\text{hull}}(A), \text{Vol}_{\text{hull}}(B))
  • Surface gap: ds=min⁡u∈A,v∈B∥u−v∥2d_s = \min_{u \in A, v \in B} \|u - v\|_2

where rinr_{in} is the symmetrized in-hull vertex fraction. The soft map yields penetration and proximity scores:

spen=min⁡(s(Vp;0.05,0.15),s(rin;0.20,0.50))s_{\text{pen}} = \min(s(V_p; 0.05, 0.15), s(r_{in}; 0.20, 0.50)) sprox=s(ds;0.05,0.40)s_{\text{prox}} = s(d_s; 0.05, 0.40) sclear=1−max⁡(sprox,1−spen,1−squal)s_{\text{clear}} = 1 - \max(s_{\text{prox}}, 1 - s_{\text{pen}}, 1 - s_{\text{qual}})

The final Interaction score depends on prompt intent:

0.45 s_{\text{pen}} + 0.35 s_{\text{prox}} + 0.20 s_{\text{qual}} & \text{req.} \\ 0.55 s_{\text{pen}} + 0.45 s_{\text{clear}} & \text{forb.} \\ 0.85 s_{\text{pen}} + 0.15 s_{\text{qual}} & \text{unspec.} \end{cases}$$ ## Empirical Validation / Results ### Main Results Ten editors (7 open, 3 closed) share one multi-reference + instruction interface. Key results from Track A (vendor-native resolution): | Model | Anat | Inter | Count | ID | Instr | Qual | |---|---|---|---|---|---|---| | GPT-Image-2 | 0.58 | 0.61 | 0.74 | 0.58 | 0.93 | 0.29 | | Gemini-3-Pro-Image | 0.65 | 0.68 | 0.71 | 0.50 | 0.93 | 0.27 | | Seedream-5-Pro | 0.62 | 0.72 | 0.72 | 0.49 | 0.93 | 0.29 | | FLUX.1-Kontext | 0.57 | 0.57 | 0.69 | 0.21 | 0.71 | 0.30 | | DreamO | 0.56 | 0.57 | 0.69 | 0.20 | 0.72 | 0.30 | | OmniGen2 | 0.55 | 0.53 | 0.69 | 0.17 | 0.77 | 0.29 | | UNO | 0.57 | 0.53 | 0.68 | 0.05 | 0.65 | 0.28 | | ACE++ | 0.54 | 0.45 | 0.49 | 0.02 | 0.54 | 0.29 | | BAGEL | 0.62 | 0.59 | 0.68 | 0.15 | 0.75 | 0.26 | | FireRed-Image-Edit | 0.63 | 0.60 | 0.72 | 0.17 | 0.86 | 0.29 | ### VLM vs. Mesh Comparison | Model | V-Anat | V-Inter | M-Anat | M-Inter | |---|---|---|---|---| | GPT-Image-2 | 0.95 | 0.98 | 0.58 | 0.61 | | Gemini-3-Pro-Image | 0.95 | 0.99 | 0.65 | 0.68 | | Seedream-5-Pro | 0.96 | 0.98 | 0.62 | 0.72 | | FLUX.1-Kontext | 0.64 | 0.95 | 0.57 | 0.57 | | DreamO | 0.71 | 0.96 | 0.56 | 0.57 | | OmniGen2 | 0.75 | 0.96 | 0.55 | 0.53 | | UNO | 0.71 | 0.93 | 0.57 | 0.53 | | ACE++ | 0.56 | 0.88 | 0.54 | 0.45 | | BAGEL | 0.71 | 0.94 | 0.62 | 0.59 | | FireRed-Image-Edit | 0.89 | 0.97 | 0.63 | 0.60 | ### Human Agreement Study On a shared Anat/Inter checklist (N=170 hard units), the mesh-based mapping (M) beats VLM (V) on 9/10 items: | Item | $\rho$(H, M) | $\rho$(H, V) | |---|---|---| | I0 | 0.41 | 0.77 | | I1 | 0.39 | 0.30 | | Ic | 0.54 | 0.53 | | I3† | 0.43 | 0.09 | | Ir† | 0.46 | 0.17 | | A1 | 0.34 | 0.20 | | A2 | 0.24 | 0.18 | | A3 | 0.38 | 0.21 | | A4 | 0.38 | 0.33 | | A5 | 0.26 | 0.19 | †Steiger p<0.05 for M > V ### Contact-Density Stress Anatomy softens from C2→C3 for Gemini, Seedream, and FireRed, while several open systems stay nearly flat. For high-contact comparisons, absolute Inter under C3 ranges from ≈0.51–0.69. ## Theoretical and Practical Implications **Theoretical implications**: The paper demonstrates that VLM-based evaluation of interaction quality saturates near ceiling (0.98–0.99) while missing obvious anatomical failures. This reveals a fundamental limitation of language-model-based judges for geometric reasoning about human bodies in contact. The mesh-based Anatomy and Interaction axes provide a principled alternative grounded in 3D reconstruction. **Practical implications**: - The benchmark reveals that no single editor is strong on both Anatomy and Interaction, with leaders diverging (Gemini Anat 0.65, Seedream Inter 0.72). - Identity shows the steepest closed/open gap ($S_{id} \approx 0.49–0.58$ vs. 0.02–0.21), indicating that open-source models struggle with face preservation under contact. - The protocol is reproducible: frozen dumps allow audit, and alternate frontend tracks can be added without redefining the editing task. - The benchmark can guide training interventions: the paper sketches a future training recipe with auxiliary skeleton streams and contact-density curriculum. ## Conclusion MPIE-Bench contributes a licensing-aware, scene-split test set of 2,500 video-mined triplets over 405 scenes, stratified by 14 interaction categories and C0–C3 contact density, with held-out targets so editors cannot copy pixels. MPIE-Eval contributes a six-axis protocol that keeps Count/ID/Instruction/Quality for task success while scoring Anatomy/Interaction from a frozen, dump-auditable Multi-HMR frontend under dual public tracks. Across ten editors, the mesh axes do work the checklists do not: closed V-Inter saturates near ceiling while M-Inter retains dynamic range; identity remains the steepest closed/open gap; Anat and Inter leaders need not coincide; and density, corruption, human-checklist, and ablation checks support model-level ranking under the frozen frontend rather than a single blended score. The authors release the redistributable subset, evaluation code, per-sample Multi-HMR dumps, and the calibration recipe so the community can audit rankings and add alternate-frontend tracks without redefining the editing task.

Related papers