# OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners

> OR-License shows frontier LLMs match expert objective quality on feasible instances, yet none achieve expert-level deployment-adjusted quality, with infeasibility—not objective shortfall—dominating the performance gap.

- **Source:** [arXiv](https://arxiv.org/abs/2610.24087)
- **Published:** 2026-10-21
- **Permalink:** https://picx.dev/p/F8LlCF
- **Whiteboard:** https://picx.dev/p/F8LlCF/image

## Summary

## Summary (Overview)

- **OR-License** is an end-to-end benchmark for evaluating LLM agents as real-world Operations Research (OR) practitioners, built from 31 instances across 10 industrial scenarios sourced from paid OR engagements totaling $3.68 million in project value.
- The benchmark preserves original client materials (requirements, input data, deliverable formats, information boundaries) after de-identification, addressing three gaps in prior work: task complexity, information acquisition, and grounding in professional delivery.
- **ORSIM**, a controlled client simulator, allows agents to acquire client-held business facts throughout formulation, solving, and refinement without supplying OR expertise, maintaining a strict information boundary.
- Across 11 frontier LLMs, several exceed expert-level conditional objective quality on feasible instances, yet **none matches the expert reference in Deployment-Adjusted Quality (DAQ)**; the strongest configuration (Grok 4.6) achieves 88.39% strict feasibility and DAQ of 86.55%.
- Trajectory analysis reveals that client question-asking is heavily front-loaded (86.7%–98.3% in stage S1), yet later-stage questions (S4–S5) show higher confirmed issue-resolution proportions, indicating a gap between continuous information access and its sustained use.

---

## Introduction and Theoretical Foundation

Operations research supports high-value decisions across manufacturing, logistics, healthcare, and retail. Deploying an OR system requires experts to transform incomplete business materials into executable decision processes through:
- Requirement elicitation
- Problem formulation
- Solution strategy selection
- Solver implementation

Gurobi's 2024 report notes that 82% of survey respondents hold a master's or doctoral degree, underscoring the professional expertise required. The paper asks: **Can LLM agents carry out this work?**

Three gaps motivate the benchmark:

1. **Gap 1: Task Complexity and Practical Relevance** – Benchmarks from academic/synthetic tasks may not capture real client engagement complexity.
2. **Gap 2: Information Acquisition** – Evaluations with full task context omit the work required to establish that context.
3. **Gap 3: Limited Grounding in Professional Delivery** – Professional capability requires a reference grounded in client-accepted expert solutions.

Three research questions guide the study:

- **RQ1:** How do LLM agents compare with OR experts in feasibility and objective quality?
- **RQ2:** When do agents acquire client information across the OR workflow?
- **RQ3:** What failure patterns emerge in translating business requirements into executable OR solutions?

---

## Methodology

### Task Definition

Each instance $i$ in scenario $s$ defines a decision space $\mathcal{V}_{i,c}$ and objective $f_{i,c}: \mathcal{V}_{i,c} \to \mathbb{R}$ with hidden verifier $\text{verifier}_{i,c}$:

$$\max_{y \in \mathcal{Y}_{i,c}} \sigma_i f_{i,c}(y) \quad \text{subject to} \quad \operatorname{verifier}_{i,c}(y) = \text{pass}, \qquad \sigma_i \in \{+1, -1\}$$

where $\sigma_i = +1$ for maximization and $\sigma_i = -1$ for minimization.

### ORSIM Client Simulator

ORSIM exposes scenario-grounded client facts throughout the episode while withholding OR expertise:
- Each query discloses at most **three non-brief facts**
- A simulator-side LLM selects relevant entries from a closed fact catalog
- A deterministic controller enforces ownership, visibility, and per-query disclosure rules
- A response-only LLM renders natural-language answers

### Key Metrics

**Strict Feasibility:**
$$F_i = \mathbf{1}[\forall c \in \mathcal{C}_i, \text{ verifier}_{i,c}(\hat{y}_{i,c}) = \text{pass}], \qquad \text{Feas} = \frac{1}{N} \sum_{i=1}^N F_i$$

**Expert-Relative Quality** (for feasible instances):
$$Q_i = \begin{cases} \frac{\tilde{J}_i^{\text{expert}}}{\tilde{J}_i^{\text{agent}}}, & \text{if minimization} \\ \frac{\tilde{J}_i^{\text{agent}}}{\tilde{J}_i^{\text{expert}}}, & \text{if maximization} \end{cases}$$

**Deployment-Adjusted Quality (DAQ)** — the primary metric, gating quality by feasibility:
$$\text{DAQ} = \frac{1}{N} \sum_{i=1}^N Z_i = \frac{1}{N} \sum_{i \in \mathcal{F}} Q_i = \text{Feas} \cdot Q_{\text{feas}}$$

Fully infeasible deliveries receive **zero credit**. The expert reference has DAQ = 1.

**Reliability–Quality Profile** (power-mean family, penalizing low scores):
$$\text{DAQ}_\alpha = \left(\frac{1}{N} \sum_{i=1}^N Z_i^{1/\alpha}\right)^\alpha, \quad \alpha \in \{1.0, 1.1, \ldots, 3.0\}$$

### Trajectory Analysis

Three annotated dimensions:
1. **Client information acquisition** across six workflow stages (S1–S6)
2. **Scientific inquiry** via HCAEU chains (Hypothesis → Explanation → Experiment → Evidence interpretation → Update)
3. **Error analysis** across eight categories (UND, DATA, DEC, CON, DYN, UNC, OBJ, IMP)

---

## Empirical Validation / Results

### RQ1: Cross-Model Comparison (Claude Code harness, 5 runs per model–task pair)

**Table 1: Main results on 31 tasks (%)**

| Model | Feasibility (%) | Quality (%) | DAQ (%) |
|-------|----------------|-------------|---------|
| Grok 4.6 (high) | **88.39 ± 10.28** | 97.92 ± 14.67 | **86.55 ± 12.97** |
| GPT-5.6 Sol (xhigh) | 84.52 ± 17.02 | 100.59 ± 21.04 | 85.02 ± 17.78 |
| Gemini 3.8 Flash (high) | 82.58 ± 17.49 | 100.43 ± 23.07 | 82.94 ± 19.06 |
| Qwen3.8-Max (xhigh) | 78.71 ± 23.11 | 99.06 ± 30.13 | 77.97 ± 23.71 |
| Claude Opus 5 (xhigh) | 77.42 ± 4.98 | 100.03 ± 9.38 | 77.44 ± 7.26 |
| Muse Spark 1.3 (high) | 74.19 ± 12.04 | 98.21 ± 16.63 | 72.86 ± 12.34 |
| DeepSeek-V4.1-Flash (high) | 69.68 ± 14.60 | 99.44 ± 22.86 | 69.28 ± 15.93 |
| Hy4 preview (high) | 66.45 ± 9.95 | 100.61 ± 15.62 | 66.85 ± 10.38 |
| Doubao-Seed-2.1-Pro (high) | 65.16 ± 17.81 | 99.47 ± 28.34 | 64.82 ± 18.47 |
| GLM-5.3-Flash (max) | 62.58 ± 25.35 | 101.01 ± 41.04 | 63.21 ± 25.68 |
| Hy3 (high) | 45.16 ± 19.58 | 96.92 ± 43.18 | 43.77 ± 19.50 |

**Key findings:**
- Feasibility spans **43.23 percentage points** (45.16%–88.39%), while conditional quality spans only **4.09 points** (96.92%–101.01%)
- Several models **exceed 100% expert-relative quality** on feasible instances, but **none matches expert DAQ = 1**
- Grok 4.6's 13.45-point DAQ gap decomposes into **11.61 points from infeasible deliveries** and only 1.84 points from objective shortfall — infeasibility dominates
- The reliability–quality profile shows stable rankings across all 21 α values; no model closes the expert gap

**Research-chain (HCAEU) results:**
- Task-equalized complete-chain rates range from **24% (Muse Spark 1.3) to 45% (Grok 4.6)**
- GPT-5.6 Sol forms no testable hypothesis in **41% of opportunities**
- Qwen3.8-Max stops at HC/HCAE in 21% each — evidence interpreted without linked update
- GPT-5.6 Sol has high DAQ (85.02%) but low conditional stage quality; Qwen3.8-Max has highest stage quality but lower DAQ (77.97%)

### RQ2: Client Information Acquisition

- Questioning is **strongly front-loaded**: S1 accounts for **86.7%–98.3% of questions** across models
- Later-stage questions contribute to confirmed issue resolution:
  - S1 confirmed-resolution: 10.6%–37.0%
  - S4 (DeepSeek-V4.1-Flash): ~56% confirmed resolution
  - S5 (Hy4 Preview): **71% confirmed resolution**
- Agents have continuous access to client information but **do not consistently revisit assumptions** as solutions evolve

### RQ3: Failure Patterns

From Figure 5 (right panel), error categories vary substantially by model:

- **CON (static constraints):** 42.7% of Claude Opus 5's errors
- **DYN (dynamic/state transitions):** 28.7% (Hy4 Preview), 25.5% (GLM-5.3-Flash)
- **OBJ + UNC (objective + uncertainty):** 69.0% (Grok 4.6), 70.5% (Muse Spark 1.3)
- **IMP (implementation/delivery):** largest category only for GPT-5.6 Sol (33.3%) and Hy3 (33.1%)

The key insight: an executable program may faithfully implement a formulation that **misrepresents the required objectives, state transitions, or information structure** — improving execution alone would not fix that mismatch.

---

## Theoretical and Practical Implications

**Theoretical implications:**
- **Feasibility-gated quality** (DAQ) provides a principled hierarchical evaluation: solution quality has operational meaning only after feasibility is established
- **Conditional quality vs. sustained research capability are distinct dimensions** of expertise — models can demonstrate strong evidence-based reasoning occasionally but lack reliability across long-horizon tasks
- The power-mean family (DAQ_α) offers a generalizable framework for examining how rankings change under increasing reliability penalties

**Practical implications:**
- Client information acquisition should be a **continuous, feedback-responsive process**, not just initial requirement gathering
- Agents need to recognize when intermediate results or execution feedback warrant **revisiting business assumptions**
- The dominant gap is **incomplete feasible delivery**, not modest objective shortfalls — reliability matters more than marginal optimization improvement
- Failure profiles suggest the need for agents that combine **experimental evidence and client facts** to revisit task interpretation and revise formulations

---

## Conclusion

OR-License evaluates LLM agents on 31 de-identified OR instances from 10 industrial scenarios using:
- Original client materials
- ORSIM-mediated interaction
- Hidden feasibility checks
- Client-accepted expert references

**Key takeaways:**

1. Several LLMs exceed expert-normalized objective quality on feasible instances, yet **none matches the expert DAQ reference**
2. Information acquisition extends beyond initial clarification into solution development and validation; later-stage questions show higher confirmed issue-resolution proportions
3. Major capability gaps remain in:
   - **Requirement interpretation**
   - **Faithful problem formulation**
   - **Sustained information-gap resolution**
   - **Evidence-linked self-improvement**

**Future directions** include developing agents with consistent, feedback-responsive processes for identifying and resolving information gaps as solutions evolve, and improving formulation fidelity from business semantics through implementation and validation.

---

_Markdown view of https://picx.dev/p/F8LlCF, served by PicX — AI-generated visual whiteboard summaries of research papers._