Summary (Overview)

• OR-License is an end-to-end benchmark for evaluating LLM agents as real-world Operations Research (OR) practitioners, built from 31 instances across 10 industrial scenarios sourced from paid OR engagements totaling $3.68 million in project value.
• The benchmark preserves original client materials (requirements, input data, deliverable formats, information boundaries) after de-identification, addressing three gaps in prior work: task complexity, information acquisition, and grounding in professional delivery.
• ORSIM, a controlled client simulator, allows agents to acquire client-held business facts throughout formulation, solving, and refinement without supplying OR expertise, maintaining a strict information boundary.
• Across 11 frontier LLMs, several exceed expert-level conditional objective quality on feasible instances, yet none matches the expert reference in Deployment-Adjusted Quality (DAQ); the strongest configuration (Grok 4.6) achieves 88.39% strict feasibility and DAQ of 86.55%.
• Trajectory analysis reveals that client question-asking is heavily front-loaded (86.7%–98.3% in stage S1), yet later-stage questions (S4–S5) show higher confirmed issue-resolution proportions, indicating a gap between continuous information access and its sustained use.

---

Introduction and Theoretical Foundation

Operations research supports high-value decisions across manufacturing, logistics, healthcare, and retail. Deploying an OR system requires experts to transform incomplete business materials into executable decision processes through:

• Requirement elicitation
• Problem formulation
• Solution strategy selection
• Solver implementation

Gurobi's 2024 report notes that 82% of survey respondents hold a master's or doctoral degree, underscoring the professional expertise required. The paper asks: Can LLM agents carry out this work?

Three gaps motivate the benchmark:

1. Gap 1: Task Complexity and Practical Relevance – Benchmarks from academic/synthetic tasks may not capture real client engagement complexity.
2. Gap 2: Information Acquisition – Evaluations with full task context omit the work required to establish that context.
3. Gap 3: Limited Grounding in Professional Delivery – Professional capability requires a reference grounded in client-accepted expert solutions.

Three research questions guide the study:

• RQ1: How do LLM agents compare with OR experts in feasibility and objective quality?
• RQ2: When do agents acquire client information across the OR workflow?
• RQ3: What failure patterns emerge in translating business requirements into executable OR solutions?

---

Methodology

Task Definition

Each instance $i$ in scenario $s$ defines a decision space $\mathcal{V}_{i,c}$ and objective $f_{i,c}: \mathcal{V}_{i,c} \to \mathbb{R}$ with hidden verifier $\text{verifier}_{i,c}$:

$$\max_{y \in \mathcal{Y}_{i,c}} \sigma_i f_{i,c}(y) \quad \text{subject to} \quad \operatorname{verifier}_{i,c}(y) = \text{pass}, \qquad \sigma_i \in \{+1, -1\}$$

where $\sigma_i = +1$ for maximization and $\sigma_i = -1$ for minimization.

ORSIM Client Simulator

ORSIM exposes scenario-grounded client facts throughout the episode while withholding OR expertise:

• Each query discloses at most three non-brief facts
• A simulator-side LLM selects relevant entries from a closed fact catalog
• A deterministic controller enforces ownership, visibility, and per-query disclosure rules
• A response-only LLM renders natural-language answers

Key Metrics

Strict Feasibility:

$$F_i = \mathbf{1}[\forall c \in \mathcal{C}_i, \text{ verifier}_{i,c}(\hat{y}_{i,c}) = \text{pass}], \qquad \text{Feas} = \frac{1}{N} \sum_{i=1}^N F_i$$

Expert-Relative Quality (for feasible instances):

$$Q_i = \begin{cases} \frac{\tilde{J}_i^{\text{expert}}}{\tilde{J}_i^{\text{agent}}}, & \text{if minimization} \\ \frac{\tilde{J}_i^{\text{agent}}}{\tilde{J}_i^{\text{expert}}}, & \text{if maximization} \end{cases}$$

Deployment-Adjusted Quality (DAQ) — the primary metric, gating quality by feasibility:

$$\text{DAQ} = \frac{1}{N} \sum_{i=1}^N Z_i = \frac{1}{N} \sum_{i \in \mathcal{F}} Q_i = \text{Feas} \cdot Q_{\text{feas}}$$

Fully infeasible deliveries receive zero credit. The expert reference has DAQ = 1.

Reliability–Quality Profile (power-mean family, penalizing low scores):

$$\text{DAQ}_\alpha = \left(\frac{1}{N} \sum_{i=1}^N Z_i^{1/\alpha}\right)^\alpha, \quad \alpha \in \{1.0, 1.1, \ldots, 3.0\}$$

Trajectory Analysis

Three annotated dimensions:

1. Client information acquisition across six workflow stages (S1–S6)
2. Scientific inquiry via HCAEU chains (Hypothesis → Explanation → Experiment → Evidence interpretation → Update)
3. Error analysis across eight categories (UND, DATA, DEC, CON, DYN, UNC, OBJ, IMP)

---

Empirical Validation / Results

RQ1: Cross-Model Comparison (Claude Code harness, 5 runs per model–task pair)

Table 1: Main results on 31 tasks (%)

Key findings:

• Feasibility spans 43.23 percentage points (45.16%–88.39%), while conditional quality spans only 4.09 points (96.92%–101.01%)
• Several models exceed 100% expert-relative quality on feasible instances, but none matches expert DAQ = 1
• Grok 4.6's 13.45-point DAQ gap decomposes into 11.61 points from infeasible deliveries and only 1.84 points from objective shortfall — infeasibility dominates
• The reliability–quality profile shows stable rankings across all 21 α values; no model closes the expert gap

Research-chain (HCAEU) results:

• Task-equalized complete-chain rates range from 24% (Muse Spark 1.3) to 45% (Grok 4.6)
• GPT-5.6 Sol forms no testable hypothesis in 41% of opportunities
• Qwen3.8-Max stops at HC/HCAE in 21% each — evidence interpreted without linked update
• GPT-5.6 Sol has high DAQ (85.02%) but low conditional stage quality; Qwen3.8-Max has highest stage quality but lower DAQ (77.97%)

RQ2: Client Information Acquisition

• Questioning is strongly front-loaded: S1 accounts for 86.7%–98.3% of questions across models
• Later-stage questions contribute to confirmed issue resolution:
  • S1 confirmed-resolution: 10.6%–37.0%
  • S4 (DeepSeek-V4.1-Flash): ~56% confirmed resolution
  • S5 (Hy4 Preview): 71% confirmed resolution
• Agents have continuous access to client information but do not consistently revisit assumptions as solutions evolve

RQ3: Failure Patterns

From Figure 5 (right panel), error categories vary substantially by model:

• CON (static constraints): 42.7% of Claude Opus 5's errors
• DYN (dynamic/state transitions): 28.7% (Hy4 Preview), 25.5% (GLM-5.3-Flash)
• OBJ + UNC (objective + uncertainty): 69.0% (Grok 4.6), 70.5% (Muse Spark 1.3)
• IMP (implementation/delivery): largest category only for GPT-5.6 Sol (33.3%) and Hy3 (33.1%)

The key insight: an executable program may faithfully implement a formulation that misrepresents the required objectives, state transitions, or information structure — improving execution alone would not fix that mismatch.

---

Theoretical and Practical Implications

Theoretical implications:

• Feasibility-gated quality (DAQ) provides a principled hierarchical evaluation: solution quality has operational meaning only after feasibility is established
• Conditional quality vs. sustained research capability are distinct dimensions of expertise — models can demonstrate strong evidence-based reasoning occasionally but lack reliability across long-horizon tasks
• The power-mean family (DAQ_α) offers a generalizable framework for examining how rankings change under increasing reliability penalties

Practical implications:

• Client information acquisition should be a continuous, feedback-responsive process, not just initial requirement gathering
• Agents need to recognize when intermediate results or execution feedback warrant revisiting business assumptions
• The dominant gap is incomplete feasible delivery, not modest objective shortfalls — reliability matters more than marginal optimization improvement
• Failure profiles suggest the need for agents that combine experimental evidence and client facts to revisit task interpretation and revise formulations

---

Conclusion

OR-License evaluates LLM agents on 31 de-identified OR instances from 10 industrial scenarios using:

• Original client materials
• ORSIM-mediated interaction
• Hidden feasibility checks
• Client-accepted expert references

Key takeaways:

1. Several LLMs exceed expert-normalized objective quality on feasible instances, yet none matches the expert DAQ reference
2. Information acquisition extends beyond initial clarification into solution development and validation; later-stage questions show higher confirmed issue-resolution proportions
3. Major capability gaps remain in:
   • Requirement interpretation
   • Faithful problem formulation
   • Sustained information-gap resolution
   • Evidence-linked self-improvement

Future directions include developing agents with consistent, feedback-responsive processes for identifying and resolving information gaps as solutions evolve, and improving formulation fidelity from business semantics through implementation and validation.