Apple-π: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence

Summary (Overview)

First law-grounded benchmark for video models: Apple-π is the first benchmark that explicitly anchors video-model evaluation in physical laws, decomposing reasoning into Perception, Formulation, and Deduction stages rather than merely evaluating output-level physical plausibility.
Orchard dataset: A curated collection of 400 videos covering ten canonical classical mechanics tasks, organized into single-law tasks (for confounder-free diagnosis) and multi-law tasks (for probing compositional generalization), sourced from simulations, self-recorded real-world videos, and internet footage.
Hybrid evaluation suite: Combines MLLM-based subjective scoring with physics-law-grounded objective measures (mask IoU, spatiotemporal overlap, velocity error), enabling stage-resolved diagnosis of where models fail, not just whether they fail.
Key findings: The best video model (Seedance 2.0) scores only 0.473, while unified understanding-generation models (GPT Image 2: 0.704, Nano Banana 2: 0.699) substantially outperform video models, revealing a Perception→Formulation→Deduction bottleneck and weak multi-law state transfer.
Diagnostic insights: Video models exhibit useful physical priors from large-scale training but fail at law formulation, temporal deduction, state transfer across law transitions, and real-world generalization (Sim-to-Real gap).

Introduction and Theoretical Foundation

Background and Motivation

The paper draws an analogy between the history of physics and the evolution of video models:

"Aristotle attributed a falling apple to its natural tendency to seek the earth. Newton, gazing at the same fall, abstracted a compact law, and from it deduced the motion of the moon, the tides, and the wandering planets."

The authors argue that modern video models appear to be "reenacting this trajectory at scale" by implicitly absorbing geometry, motion, and causality from massive video corpora. However, existing benchmarks suffer from a critical limitation:

"All evaluate only what a model outputs, never how it got there. When a model succeeds, we cannot tell whether it truly invoked a physical law or merely produced something that looks right; when it fails, we cannot tell whether it misread the scene, misidentified the law, or stumbled in the deduction."

Theoretical Foundation

The benchmark is built on three conceptual pillars:

Newton-style scientific reasoning as an auditable protocol: Models must perceive physical quantities, formulate governing laws, and deduce law-consistent future dynamics.
Chain-of-frames reasoning: Building on recent work formalizing frame-by-frame video generation as analogous to step-by-step textual reasoning, Apple-π uses chain-of-frames prompts to elicit visible, frame-level reasoning traces.
Law-first dataset design: Rather than starting from arbitrary videos and retrospectively assigning physical labels, each case begins with an explicit governing principle in classical mechanics, specified initial conditions, and law-predicted future trajectories.

Comparison with Existing Benchmarks

Benchmark	Eval Target	Data Source	# Cases	Physics Scope	Law-Grounded	Multi-Stage	Chain-of-Frames	Objective Phys. Metric
PhysBench	VLM	R+S	10K	Multi-domain	✗	✗	✗	✗
QuantiPhy	VLM	R	3.3K	Kinematics	★	✗	✗	✓
VideoPhy-2	T2V	R	200	Material interaction	✗	✗	✗	✗
PhyWorldBench	T2V	R+S	700	Mechanics	★	✗	✗	✗
PhysicsMind	VLM+I2V	R+S	687	Mechanics	★	✗	✗	✓
Physics-IQ	I2V	R	396	Multi-domain	✗	✗	✗	✓
Apple-π (Ours)	I2V	R+S	400	Mechanics	✓	✓	✓	✓

Methodology

3.1 Orchard Dataset

Data sources (400 total cases):

Simulated videos (243 cases): Generated with NVIDIA Isaac Sim, providing exact physical parameters and pixel-accurate trajectories
Self-recorded real-world videos (121 cases): Captured under controlled laboratory conditions at 30 fps
Internet-sourced real-world videos (36 cases): Curated from physics-education YouTube channels

Controlled object vocabulary: Four primitive solids (sphere, cube, cylinder, cone) to control object-specific semantic priors and ensure consistent annotation.

Task taxonomy (two-level):

Task	Pillar	Representative Law
Free fall	Universal gravitation	$v(t) = v_0 + gt$ , $h(t) = h_0 + v_0t + \frac{1}{2}gt^2$
Projectile motion	Universal gravitation	$\vec{r}(t) = \vec{r}_0 + \vec{v}_0t + \frac{1}{2}\vec{g}t^2$
Inclined plane	Universal gravitation	$a = g(\sin\theta - \mu\cos\theta)$
Circular motion	Universal gravitation	$mg = mv^2/r$
Perfectly elastic collision	Conservation of momentum	$e = 1$
Perfectly inelastic collision	Conservation of momentum	$e = 0$
Inelastic collision	Conservation of momentum	$0 < e < 1$
At rest	Newton's first law	$\sum\vec{F} = 0$ , $v = 0$
Uniform linear motion	Newton's first law	$\sum\vec{F} = 0$ , $v(t) = v_0$
Composition	Multi-law	Sequential composition of the above

3.2 Benchmark Protocol

Input format: Infographic-style annotated first frame with physical parameters overlaid next to their visual referents (e.g., mass adjacent to objects, global labels for gravitational acceleration).

Output format: Chain-of-frames video, taking one of three forms:

Full generated sequence (for trajectory evaluation)
Final-frame artifact with fade-to-white (for discrete answers like labels or formulas)
Final-frame artifact without fade-to-white (for scene-grounded answers like object positions)

Five subtracks:

Subtrack	Question	Expected Output	Main Evaluation Target
P-T (Perception-Text)	Read physical quantities	White-background annotation artifact	Text/OCR, annotation placement, readability
P-G (Perception-Graphic)	Ground physical objects	White-background object-only artifact	Object localization and mask IoU
F-T (Formulation-Text)	Select governing law	Three-line formula answer (option, formula, substitution)	Option, formula, substitution
F-G (Formulation-Graphic)	Predict state at $t^\star$	Scene at target time with velocity arrows/labels	Target-state mask IoU, arrow direction, speed label
Deduction	Generate full dynamics	Full trajectory video or timestamped keyframes	Law-consistent temporal evolution

Formulation-Text distractors designed to expose specific failure modes:

A confusing real law sharing symbols with annotations
An unrelated real law with no symbol overlap
A fabricated formula that does not exist in physics

3.3 Evaluation Suite

MLLM-based subjective scoring: Uses Gemini 3 Flash as judge with track-specific rubrics organized into semantically coherent groups, weighted by importance per subtrack.

Physics-law-grounded objective measures:

Perception-Graphic & Formulation-Graphic: Segmentation IoU against ground-truth masks
Deduction: Pixel-level fidelity, spatiotemporal mask overlap, and 3D velocity error against law-predicted ground-truth dynamics

Empirical Validation / Results

4.1 Experimental Setup

Models evaluated (11 total):

Video generation models (5): Wan2.2, HunyuanVideo-1.5, VBVR-Wan2.2, Seedance 2.0, Veo 3.1
Unified understanding-generation models (6): BAGEL, OmniGen2, SenseNova-U1-8B-MoT(-Think), GPT Image 2, Nano Banana 2

Each model tested on 400 cases × 5 subtracks × 3 rollouts = 6,000 responses per model.

4.2 Main Results

Model	Avg.	P-T	P-G	F-T	F-G	Ded.	Grav.	Mom.	N1	Multi	Sim.	Real
Video Models
Wan2.2	0.267	0.645	0.257	0.009	0.275	0.149	0.245	0.274	0.344	0.200	0.310	0.224
HunyuanVideo-1.5	0.177	0.230	0.292	0.027	0.180	0.155	0.128	0.205	0.272	0.183	0.208	0.146
VBVR-Wan2.2	0.373	0.923	0.502	0.001	0.239	0.201	0.347	0.375	0.459	0.387	0.394	0.352
Seedance 2.0	0.473	0.597	0.490	0.478	0.485	0.315	0.450	0.510	0.495	0.389	0.487	0.459
Veo 3.1	0.313	0.408	0.357	0.325	0.316	0.160	0.288	0.314	0.420	0.235	0.356	0.270
Unified Models
BAGEL	0.218	0.218	0.383	0.091	0.186	0.213	0.192	0.242	0.275	0.162	0.246	0.191
OmniGen2	0.214	0.168	0.442	0.061	0.213	0.184	0.168	0.232	0.313	0.208	0.242	0.185
SenseNova-U1-8B-MoT	0.362	0.515	0.492	0.157	0.339	0.304	0.326	0.389	0.454	0.267	0.393	0.330
SenseNova-U1-8B-MoT-Think	0.359	0.505	0.485	0.154	0.345	0.307	0.322	0.390	0.449	0.268	0.390	0.328
GPT Image 2	0.704	0.921	0.719	0.824	0.651	0.406	0.695	0.711	0.742	0.618	0.740	0.668
Nano Banana 2	0.699	0.934	0.667	0.841	0.650	0.405	0.684	0.723	0.733	0.567	0.743	0.656

Key Findings

Takeaway 1: Large-scale video training provides useful physical priors, but reliable law-grounded physical intelligence requires explicit understanding modules, richer physical-reasoning data, and reasoning-oriented post-training.

Takeaway 2: Video models show a progressive reasoning bottleneck: they can often perceive physical cues and partially formulate laws, but these intermediate successes do not reliably transfer to law-consistent dynamics over time.

Takeaway 3: Video models show limited generalization across both laws and sources: they struggle to carry physical states across law transitions in multi-law cases and to apply the same laws under real-world visual variation.

4.5 Qualitative Failure Analysis

Common failure modes include:

Annotation semantics failure: Model preserves text and tracks objects but fails to bind annotations to physical meaning (e.g., initial-velocity arrow assigned wrong direction)
Format compliance failures: Blurred annotation text, failure to isolate target objects, unreadable formulas, missing velocity arrows

4.6 Protocol Ablations

Model	Avg. Δ	P-T	P-G	F-T	F-G	Ded.
Text-Parameter
Veo 3.1	+0.008	—	+0.020	+0.038	-0.024	+0.000
Nano Banana 2	+0.020	—	+0.044	-0.006	+0.011	+0.030
Concise-Prompt
Veo 3.1	-0.018	+0.015	-0.068	+0.051	-0.096	+0.006
Nano Banana 2	-0.037	-0.016	-0.085	+0.051	-0.089	-0.045

Ablation conclusions:

Text-Parameter (providing quantities in structured text) is nearly neutral, confirming the benchmark is not bottlenecked by reading infographic annotations
Concise-Prompt hurts graphic-output tracks (P-G, F-G) but improves F-T (+0.051), showing the full prompt mainly standardizes visual answer format rather than injecting physics solutions

Theoretical and Practical Implications

Diagnostic Value

Apple-π's stage-resolved design enables three levels of diagnosis:

Stage-level: Identifies the Perception→Formulation→Deduction bottleneck as the primary failure mode
Pillar-level: Reveals weak multi-law state transfer (Multi scores consistently lower than single-law pillars)
Source-level: Quantifies the Sim-to-Real gap, showing models struggle to apply the same laws under realistic visual conditions

Architectural Insights

The gap between video models and unified understanding-generation models suggests:

Explicit understanding matters: GPT Image 2 and Nano Banana 2 substantially outperform video models, particularly on Perception and Formulation
The advantage is not architectural alone: Open-source unified models remain much weaker, indicating the benefit also depends on stronger multimodal foundations, broader training data, and mature post-training
Deduction remains the universal bottleneck: Even the strongest unified models score only ~0.40 on Deduction, indicating law-consistent temporal dynamics is the central challenge

Practical Implications for Model Development

The findings suggest future video world models will require:

Explicit understanding modules for binding quantities to visual referents
Richer physical-reasoning data targeting law formulation and deduction
Reasoning-oriented post-training (as demonstrated by VBVR-Wan2.2's improved perception skills)
Mechanisms for state transfer across law transitions in compositional scenarios
Robust domain generalization to close the Sim-to-Real gap

Conclusion

Apple-π establishes the first benchmark that anchors video-model evaluation explicitly in physical laws, decomposing the reasoning process into Perception, Formulation, and Deduction stages. The benchmark reveals that:

Current video models acquire useful physical priors from large-scale training but remain far from reliable law-grounded simulators
There exists a progressive reasoning bottleneck: perception is easiest, formulation is harder, and deduction is hardest
Explicit understanding-generation models outperform video models, suggesting that understanding-centric design improves law-grounded physical intelligence
Multi-law composition and real-world generalization remain significant challenges

Future directions implied by this work include:

Developing video models with explicit physical understanding modules
Creating richer physical-reasoning training data targeting law formulation and temporal deduction
Investigating reasoning-oriented post-training strategies
Exploring mechanisms for robust state transfer across law transitions and visual domains

The authors position Apple-π as "a diagnostic foundation for guiding future video models toward world models with law-grounded physical intelligence," emphasizing that the path forward requires not only better temporal generation but also explicit physical reasoning capabilities.