Summary (Overview)

• Mental World Modeling (MWM) is proposed as a novel theoretical framework that extends traditional world models by explicitly incorporating mental and social variables (beliefs, goals, intentions, emotions, norms) alongside physical state variables, enabling AI systems to predict human decisions in social contexts.
• M ENTIS, a training-free, fully inspectable baseline implementation, operationalizes MWM through a six-stage pipeline: state parsing, observation generation, action decomposition, coupled physical-mental transition simulation, value evaluation, and deterministic decision.
• Empirical validation on a manually constructed, quality-controlled dataset (Menti-Bench) of 448 records spanning text, image, and sounding-video modalities demonstrates that explicit mental world modeling is necessary for predicting human decisions, with full MWM achieving 87.9% average F1 across 8 LLM-based world models versus 63.3% for direct answering.
• Bottleneck analysis reveals that transition simulation (predicting how the coupled world changes) is the largest single source of error, recovering 45% of the human gap when replaced with gold annotations, setting the agenda for future improvements.
• Modality-general gains: MWM closes the performance gap across text, image, and video inputs by converting all evidence into a common structured state representation, with the largest gains on interpersonal scenes (+26.4 F1 points) where hidden mental variables govern decisions.

Introduction and Theoretical Foundation

The paper argues that existing world models—whether representation-based (Dreamer, JEPA), video-generative (Sora, Genie), or 3D interactive (Marble)—share a fundamental limitation: they model only the physical substrate of the world (objects, positions, geometry, motion). Human behavior, however, is driven by hidden mental states: what a person believes, wants, intends, feels, and considers socially permissible.

Theoretical grounding draws from three established premises:

1. Theory-of-Mind premise (Premack & Woodruff, 1978): Mental states (beliefs, desires, intentions, knowledge, affect) are causal state variables that help produce behavior. Two agents in the same physical room with different beliefs will take different actions.

2. Embodied cognition premise (Varela, Thompson, & Rosch, 1991): Mental state is situated and depends on the agent's interaction with the world. Disappointment depends on what was expected and what occurred; politeness depends on role, place, and norm.

3. Affordance premise (Gibson, 1979): Observation is already interpreted—a chair can be sit-able, offer-able, blocked, or socially reserved. The same physical scene renders different observations for different agents.

Formal foundation: MWM builds on Partially Observable Markov Decision Processes (POMDPs), maintaining a joint state $s_t = (s_t^{phy}, s_t^{ment})$, rendering a target-specific observation $o_t^\epsilon$, decomposing action into physical and mental aspects $a_t = (a_t^{phy}, a_t^{ment})$, and defining an action-conditioned transition over the next joint state.

Core modeling claim: "The minimal object of simulation for social decision-making is not a physical trajectory alone and not a theory-of-mind answer alone. It is an action-conditioned transition over a coupled state: physical variables constrain what can happen, while mental variables determine what the same happening means to the agents involved."

Property 3.1 (Why a joint state is necessary): Let $p^\star(a_t^\epsilon | s_t)$ be the true target-action distribution over the joint state $s_t = (s_t^{phy}, s_t^{ment})$. A physical-only representation is insufficient whenever there exist mental components $s_t^{ment} \neq \tilde{s}_t^{ment}$ such that:

$$p^\star(a_t^\epsilon | s_t^{phy}, s_t^{ment}) \neq p^\star(a_t^\epsilon | s_t^{phy}, \tilde{s}_t^{ment})$$

A mental-only representation is insufficient when symmetric conditions hold for physical components.

Methodology

Theoretical Framework

MWM defines three coupled functions at each time step:

Observation generation function:

$$o_t^\epsilon = \Omega^\epsilon(s_t) = (\Omega_{phy}^\epsilon(s_t^{phy}; \kappa_t^\epsilon), \Omega_{ment}^\epsilon(s_t^{phy}, s_t^{ment}; \rho_t^\epsilon))$$

where $\kappa_t^\epsilon$ summarizes perceptual access and $\rho_t^\epsilon$ summarizes social-cognitive perspective.

Action proposal function:

$$a_t^\epsilon \sim \Pi^\epsilon(\cdot | o_t^\epsilon), \quad \mathcal{A}_t^\epsilon = \Pi_K^\epsilon(o_t^\epsilon) = \{a_{t,1}^\epsilon, \ldots, a_{t,K}^\epsilon\}$$

State transition function:

$$T_\theta(s_{t+1} | s_t, a_t^\epsilon) = T_{phy}^\theta(s_{t+1}^{phy} | s_t^{phy}, s_t^{ment}, a_t^{\epsilon,phy}) \cdot T_{ment}^\theta(s_{t+1}^{ment} | s_t^{phy}, s_t^{ment}, a_t^{\epsilon,phy}, a_t^{\epsilon,ment})$$

State Taxonomy

Physical state $s_t^{phy} = (O_t, C_t, R_t^{phy}, E_t)$ includes:

• Objects (intrinsic properties, contextual attributes)
• Characters (intrinsic properties, facial expression, pose, gesture, gaze, motion)
• Relations (spatial, contact)
• Environment (time, place, ambient conditions)

Mental state $s_t^{ment} = (\{m_t^i\}_{i \in \mathcal{I}}, \{m_t^G\}_{G \in \mathcal{G}}, R_t^{ment}, \alpha_t)$ includes:

• Individual mental fields: identity $id_t^i$, beliefs $b_t^i$, attention $q_t^i$, goals $g_t^i$, intentions $\iota_t^i$, emotions $e_t^i$, dispositions $d_t^i$, norms $n_t^i$, behavioral constraints $c_t^i$
• Group mental states (same structure as individuals)
• Relations (attitudes, role relations)
• Atmosphere $\alpha_t$ (scene-level social mood)

M ENTIS Implementation

M ENTIS follows a six-stage pipeline:

1. State parsing: Converts raw scene into structured JSON state following the taxonomy
2. Observation generation: Renders target's first-person observation, filtering inaccessible information
3. Action decomposition: Parses each option into physical carrier and mental content
4. Branch simulation: Predicts successor states for each candidate action (physical and mental channels in parallel)
5. Value evaluation: Scores each branch on three criteria (mental consistency, physical plausibility, social appropriateness)
6. Decision: Deterministic rule selects highest-value branch

Empirical Validation / Results

Dataset: Menti-Bench

Records stratified over four scene categories: interpersonal (47.5%), object/resource (28.1%), spatial/perceptual (12.9%), risk/norm (11.4%). 78% of scenes involve at least two characters.

Necessity Ladder Results

Key findings:

• F1 increases monotonically at every rung for all 8 models
• Removing mental channel costs 12.1 points; removing physical channel costs 16.5 points
• Decoupled transition costs 6.4 points, confirming the necessity of joint physical-mental transition
• Stronger base models benefit less from MWM (+21.1 for gpt-5.6-sol vs. +28.0 for gpt-4.1)

Bottleneck Localization (gpt-5.6-sol)

Transition simulation is the largest single bottleneck, recovering 45% of the human gap. Oracle gains are sub-additive (+8.7 sum vs. +6.3 combined), indicating overlapping, propagating stage errors.

Scenario Study (gpt-5.6-sol)

Gains are largest on interpersonal scenes (where hidden mental variables govern decisions) and flat across domains (19.1–22.8), confirming the gains are modeling gains rather than surface-pattern gains.

Modality Analysis (gpt-5.6-sol)

The text-video gap shrinks from 6.0 (S1) to 0.3 (S6). Channel interventions confirm that media evidence is genuinely used: replacing images with captions costs S6 6.4 points; removing video audio costs S6 6.1 points.

Theoretical and Practical Implications

Theoretical implications:

• MWM provides a formal mathematical framework unifying physical and mental dynamics within a single world-modeling view, formalized through augmented POMDPs
• The framework establishes that mental variables are not merely labels to infer but state variables that evolve as physical events, speech acts, observations, and social consequences unfold
• The coupled transition formulation reveals that physical dynamics constrain what can happen while mental dynamics determine what the same happening means to agents involved

Practical implications:

• Embodied collaboration: MWM enables robots to reason about whether a person has noticed them, whether an object is socially available, whether an interruption is acceptable, and whether an action will be read as help or pressure
• Care and support: MWM maintains explicit hypotheses about comprehension, trust, anxiety, obligation, and dependency—variables a physical record cannot expose yet determine whether an intervention helps or harms
• Education and training: MWM clarifies that an explanation is an action with a linguistic carrier and a mental transition target, updating not only knowledge but also confidence, motivation, and trust
• Interactive agents and social worlds: MWM provides a transition interface for digital characters to remember promises, hide information, react differently to friends and strangers, and maintain coherent emotional and normative trajectories

Diagnostic value: M ENTIS enables component-level failure analysis—if a prediction fails, the failure can be localized to state parsing, observation generation, action decomposition, transition simulation, or value evaluation, rather than being hidden behind final-answer accuracy.

Conclusion

This paper introduces Mental World Modeling (MWM) as a framework for world models that reason about people rather than only objects. The central move is treating physical and mental states as one coupled world state, rendering a target-specific partial observation, and simulating how candidate actions change both the material scene and the mental-social configuration.

Key contributions:

1. A formal mathematical framework unifying physical and mental dynamics within a single world-modeling view
2. M ENTIS, a training-free, fully inspectable baseline implementation
3. Systematic empirical validation confirming explicit mental world modeling is necessary for predicting human decisions

Main findings:

• Full MWM is the best configuration for all 8 LLM-based world models tested
• Both physical and mental channels, plus their coupled transition, are required
• Gains are largest on interpersonal scenes (+26.4 F1 points)
• Transition simulation is the binding bottleneck, recovering 45% of the human gap when replaced with gold annotations

Future directions: The paper identifies transition simulation as the priority for improvement, with state parsing as the secondary bottleneck. The larger claim is that "human-centered AI requires world models whose latent variables include beliefs, goals, intentions, emotions, norms, relationships, and social atmosphere. Without these variables, many apparently simple human actions remain unpredictable for the wrong reason: the model looks at the world, but not at the world as the target agent understands it."