# Do Enterprise Systems Need Learned World Models? The Importance of Context to Infer Dynamics

> CascadeBench shows offline world models degrade sharply under deployment shift, while enterprise discovery agents that retrieve business rules at inference time remain robust.

- **Source:** [arXiv](https://arxiv.org/abs/2605.12178)
- **Published:** 2026-05-14
- **Permalink:** https://picx.dev/p/3yg2yo

## Summary

## Summary (Overview)

- **Core question**: The paper investigates whether enterprise systems need learned world models, or whether agents should instead *discover* transition dynamics at runtime by reading the system's configuration, since enterprise dynamics are tenant-specific, configurable, and evolve over time.
- **Key contribution**: Introduces **CascadeBench**, a reasoning-focused benchmark for enterprise cascade prediction built on synthetic schemas, designed to isolate rule-based reasoning from parametric memorization and retrieval noise.
- **Main finding**: Offline-trained world models perform well in-distribution (e.g., Gemma-4-31B reaches 91.6 IoU in-distribution) but degrade significantly under deployment shift (dropping to ~40-41 IoU), whereas **enterprise discovery agents** that retrieve business rules at inference time remain more robust.
- **Proposed approach**: **Enterprise discovery agents** use a retrieve-then-reason strategy, querying the live instance for business rules, schemas, and records at inference time rather than relying solely on internalized dynamics.
- **Empirical evidence**: Discovery agents recover a large fraction of the oracle signal (reaching mid-20s to low-30s IoU vs. ~10 for prompted baselines without rules) and outperform prompted baselines across all rollout horizons on the World of Workflows benchmark.

---

## Introduction and Theoretical Foundation

### Background

Large Language Model (LLM) agents are increasingly deployed in enterprise environments with complex dynamics. To plan and act effectively over long horizons, agents must understand how their actions affect the environment, enabling accurate anticipation of downstream state changes. Enterprise systems differ from traditional environments because their dynamics are partly specified by **tenant-specific configuration artifacts**—business rules, workflows, approval policies, and SLA definitions—that vary across deployments and evolve over time.

### Theoretical Foundation

The authors formalize enterprise dynamics as a **contextual transition model**. Let $s_t$ denote the observable platform state at step $t$, $a_t$ the action taken, and $c$ the instance configuration (the collection of all business rules, workflow definitions, approval policies, SLA definitions, and access control lists deployed on a particular customer's instance):

$$s_{t+1} \sim P(s_{t+1} | s_t, a_t, c)$$

Enterprise systems differ from standard world model settings in two ways:
1. **$c$ is not fixed** — administrators continuously modify rules, so dynamics shift without changes to the underlying platform.
2. **$c$ is explicit and readable** — rules, workflows, and policies are stored as inspectable records with defined conditions and actions.

The paper also introduces a **sparse transition view**, modeling a state delta $\Delta s_t \approx s_{t+1} - s_t$ (the subset of fields whose values change after action $a_t$), rather than the full environment state.

### Complexity Tiers

The authors distinguish three levels of transition complexity:

| Tier | Description | Example |
|------|-------------|---------|
| **Tier 1: Schema-Deterministic** | Transitions fully determined by the data dictionary (field types, defaults, constraints); no rule inspection required | Creating a user sets: `active=true`, `notification=2`, `locked_out=false` |
| **Tier 2: Rule-Composable** | Transitions determined by composing business rules; predictable given full rule knowledge, but requiring tracing of cascading execution | Setting priority to P1 triggers auto-assignment, starts an SLA timer, sends a notification, and creates an escalation record |
| **Tier 3: Execution-Inferred** | Transitions depend on undocumented engine behavior, timing, or emergent interactions not recoverable from configuration alone | Race conditions between synchronous and asynchronous rules produce outcomes only observable through execution |

---

## Methodology

### Enterprise Gym

The authors build a **world** as $W = (E, T)$, where $E$ specifies the environment (organizational structure, configuration database, business rules, initial records) and $T$ is the transition function induced by $E$ on the platform. **$T$ is not simulated**: environments are deployed to a live ServiceNow platform instance so that when an agent acts, the real engine executes server-side scripts and the resulting state is the actual database state.

Key characteristics:
- **Diversity at scale**: Worlds generated from a catalog of 1,596 business rule patterns spanning 6 industries (financial services, government, healthcare, manufacturing, retail, technology) and 11 operational domains, with 3 organizational sizes (small, midmarket, enterprise)
- **Data collection**: 27,243 verified transition samples spanning 64 worlds, collected by firing tool calls against live worlds and recording cascades through platform audit logs
- Each retained sample is a tuple $(s_t, a_t, s_{t+1}, \pi)$ where $\pi$ is the cascade path (ordered sequence of tables touched and business rules attributed to each transition)

### CascadeBench Benchmark

CascadeBench adopts the evaluation methodology of World of Workflows (WoW) but differs along three axes:
1. **Synthetic schemas** that do not appear in real platform deployments (preventing memorization)
2. **Controlled context availability** — relevant table schemas, business rules, and seed records are provided, enabling both fully contextualized evaluation and context-limited settings
3. **Audit-log ground truth restricted to content fields**, removing engine-internal metadata

### Three Approaches Compared

1. **Prompted Baseline**: Frozen language model predicting effects of an action from provided context alone
2. **Learned Enterprise World Model**: Fine-tuned (LoRA) on $(s_t, a_t, s_{t+1})$ tuples to internalize dynamics
3. **Enterprise Discovery Agent**: Queries live instance configuration $c$ at inference time using a retrieve-then-reason strategy:

$$\hat{s}_{t+1} = f_{LLM}(s_t, a_t, \tilde{c}, \hat{s}_{1:t})$$

where $f_{LLM}$ is a frozen language model, $\tilde{c} \subseteq c$ is the task-relevant retrieved subset, and $\hat{s}_{1:t}$ denotes prior predictions in a multi-step rollout.

### Models and Metrics

- **Models**: Qwen-3.5-27B, Qwen-3.6-27B, Gemma-4-31B-it (fine-tuned with LoRA, rank 16, $\alpha=32$); frontier models: Claude Sonnet 4.6, Claude Opus 4.6, GPT-5, Gemini 3 Pro
- **Metrics**: Two IoU variants — **IoU(T+F)** credits predictions identifying the affected (table, field) pair; **Strict IoU** additionally requires predicted value match

---

## Empirical Validation / Results

### Rung 1: Prompting Alone Struggles When Rules Are Hidden

When business rules are not provided, prompted models perform poorly on CascadeBench (9–16 IoU(T+F) range). SFT improves this no-BR setting modestly (~2–3 IoU points on CascadeBench, ~10 points on WoW), but with rules in context, SFT is not uniformly beneficial.

**Table 1: Main transition-prediction results (excerpt)**

| Type | Model | CascadeBench w/ BR (IoU) | CascadeBench w/o BR (IoU) | WoW w/o BR (IoU) |
|------|-------|--------------------------|---------------------------|-------------------|
| Frontier | GPT-5 | 41.78 | 9.82 | 29.34 |
| Frontier | Opus 4.6 | 40.46 | 10.91 | 41.32 |
| Frontier | Gemini 3 Pro | 41.36 | 10.38 | 35.59 |
| Finetuned | Qwen-3.5-27B-LoRA | **50.90** | 10.60 | 31.21 |
| Finetuned | Gemma-4-31B-LoRA | 41.33 | 12.19 | 31.73 |

### Rung 2: SFT Is Strong In-Distribution but Degrades Under Shift

Fine-tuning strongly internalizes training dynamics (in-distribution test IoU rises to 91.6 for Gemma-4-31B and 82.0 for Qwen-3.6-27B), but this advantage largely collapses on CascadeBench, where models face unseen synthetic schemas and configurations (both fall to roughly 40–41 IoU).

### Rung 3: Runtime Discovery Recovers Cross-Instance Accuracy

The discovery agent improves over the matched prompted baseline for every model and horizon on WoW, including at $k=1$ where Opus 4.6 rises from 0.40 to 0.45 and Sonnet 4.6 from 0.32 to 0.44.

**Table 2: Discovery agent vs. oracle and prompted models on CascadeBench**

| Type | Model | Oracle (w/ BR) | DA (w/o BR) | Prompted (w/o BR) |
|------|-------|----------------|-------------|-------------------|
| Frontier | GPT-5 | 41.78 | 32.1 | 9.82 |
| Frontier | Claude Opus 4.6 | 40.46 | 32.0 | 10.91 |
| Frontier | Claude Sonnet 4.6 | 38.15 | 29.7 | 10.30 |
| Finetuned | Qwen-3.5-27B-LoRA | 50.90 | 21.5 | 10.60 |
| Finetuned | Gemma-4-31B-LoRA | 41.33 | 12.5 | 12.19 |

### Tier-Stratified Results

**Table 4: Tier-stratified IoU on CascadeBench ($k=1$)**

| Setting | Model | ALL | T1 | T2 | T3 |
|---------|-------|-----|-----|-----|-----|
| Direct (no rules) | Opus 4.6 | 0.175 | 0.562 | 0.043 | 0.000 |
| Discovery Agent | Opus 4.6 | 0.638 | 0.633 | 0.633 | 0.545 |
| Oracle (rules in prompt) | Opus 4.6 | 0.643 | 0.632 | 0.637 | 0.564 |

Key findings:
- **T1** is predictable from action and schema alone (Direct IoU: 0.56–0.60 across all models)
- **T2 and T3** drop to 0.00 uniformly under Direct prompting, identifying business rules as the load-bearing signal
- Discovery reaches **Oracle parity on T1 and T2** (mean DA − Oracle deltas: +0.014 on T1, +0.001 on T2) but falls short on T3 (−0.046), where outcomes depend on execution-order resolution not exposed in configuration

---

## Theoretical and Practical Implications

### The Role of Business Rules

Removing business rules from the prompt produces a **uniform collapse** on CascadeBench: with BR, every model class sits in a 38–51 IoU band; without BR, the same models fall to 7–12. This demonstrates that business rules carry the dynamics that CascadeBench measures—the benchmark probes rule-grounded reasoning rather than what models already know from pretraining.

### Discovery vs. Internalization

The discovery agent's advantage persists across rollout horizons despite compounding errors. The likely mechanism is **repeated grounding**: unlike the prompted baseline, the Discovery Agent can re-query the live instance at each step, refreshing its view of the deployed configuration rather than relying only on its evolving prediction state.

### Practical Implications

- **Deployment-dependent choice**: Frontier APIs favor discovery agents; constrained open-weight deployments may favor fine-tuning due to tool-use reliability issues in the 27–31B range
- **Access controls matter**: If business rules are not readable (due to access controls), runtime discovery degenerates to the prompted baseline
- **Retrieval is not a replacement for reasoning**: Failure mode analysis reveals that even with perfect retrieval (oracle condition), models struggle with insert/creation blindness (P1), cascade fade-out after 1–2 hops (P2), and single-record assumptions (P3)

---

## Conclusion

The paper addresses a question not previously explored in the world-models literature: **when the rules can be read at inference time, does an agent still need to learn them?** The answer is nuanced:

1. **Offline-trained world models** perform well in-distribution but degrade on held-out configurations due to deployment shift.
2. **Enterprise discovery agents**, which retrieve relevant rules at inference time, remain more robust under shift and avoid some error compounding observed in purely internalized models.
3. **Discovery agents are not a replacement for learned world models** — the next step is to combine learned priors with runtime retrieval and reasoning, training agents that learn *when, what, and how* to retrieve.

### Limitations

- **Inspectability assumption**: Discovery requires readable business rules on the live instance
- **Tool-use capability bounds discovery**: On open-weight models in the 27–31B range, the retrieval loop can be unreliable
- **Single-platform evaluation**: Experiments use ServiceNow; other platforms may differ
- **Tier 3 dynamics**: Only multi-rule conflicts detectable from audit logs are stratified; broader execution-order dynamics remain out of scope

### Future Directions

The paper motivates training **discovery agents** that learn to retrieve and compose active rules more effectively, closing the gap between current discovery performance and the oracle ceiling (which represents perfect retrieval). The failure mode analysis suggests that even with perfect context, models need training to compose multi-step rule cascades—pointing toward a hybrid approach combining learned priors with runtime retrieval.

---

_Markdown view of https://picx.dev/p/3yg2yo, served by PicX — AI-generated visual whiteboard summaries of research papers._