Summary (Overview)

• SoCRATES introduces a unified benchmark for evaluating proactive LLM mediators across 8 conflict domains and 5 socio-cognitive axes, overcoming limitations of prior testbeds that rely on few expert-authored domains and conflate multiple variation axes.
• The framework features agentic scenario curation (a three-stage pipeline that automatically constructs realistic scenarios from real public disputes), socio-cognitive probing (independent perturbation of strategic posture, party composition, history length, emotional reactivity, and cultural identity), and topic-localized evaluation (scoring each topic only at turns that advance it).
• The topic-localized evaluator achieves 0.82 Pearson correlation with human experts on trajectory-level judgments, more than doubling the performance of prior per-turn baselines (ProMediate and Non-expert).
• Benchmarking 8 frontier LLMs (proprietary and open-source) reveals that even the strongest mediator closes only about one-third of the unmediated consensus gap under diverse and realistic conditions.
• Performance varies sharply by socio-cognitive axis, indicating that mediation progress lies in social adaptation to diverse conditions rather than in general reasoning capability.

---

Introduction and Theoretical Foundation

Social conflict imposes heavy societal costs, yet skilled human mediators remain scarce. This has motivated efforts to deploy LLMs as automated mediators, but current frontier models—despite reaching near-expert performance on olympiad and research-level mathematics—close only a modest fraction of the unmediated consensus gap and collapse under the variations real conflicts exhibit. The bottleneck is evaluation rather than modeling, because mediation has no single correct answer and must be judged on a real-time trajectory shaped by disputants’ shifting emotions, intentions, and evolving context.

Building such an evaluation framework poses three core challenges:

1. Scenario coverage does not scale. Real disputes carry privacy and legal sensitivity, confining existing testbeds to a few expert-authored domains (e.g., bargaining, legal disputes).
2. Real-world complexity must be reproduced along multiple independent axes. Conflicts vary in emotion, culture, and history, yet prior testbeds vary only strategic posture, conflating these axes and obscuring which one a mediator fails on.
3. Evaluation must be both trajectory-aware and noise-resilient. Mediation quality emerges across turns, but protocols like ProMediate score every topic at every turn with an LLM judge, letting off-topic content distort scores and compound errors along the trajectory.

SoCRATES addresses these challenges through a unified, automated pipeline that integrates agentic scenario curation, socio-cognitive probing, and topic-localized evaluation.

---

Methodology

Task Formulation

Following the Harvard conflict simulation framework, social conflict is cast as the negotiation of a fixed topic set by parties with divergent positions. A conflict scenario is a tuple:

$$s = (B, P, T, W)$$

• $B$: Background (past histories, prior commitments, strategic posture)
• $P = \{p_1, \dots, p_n\}$: Disputants, with $n \geq 2$
• $T = \{T_1, \dots, T_k\}$: Topics of conflict, each with a discrete option set
• $W = \{w_1, \dots, w_n\}$: Preference set, where each $w_i$ is a weight vector over topics summing to 100

Disputing Parties: Each party $p_i$ is an LLM agent that speaks on its turn, conditioned on shared inputs (background, topics, dialogue so far) and private inputs (profile: objective, fallback, per-topic starting stance, persona $\pi_i$, preferences $W$). Socio-cognitive conditions perturb one scenario component at a time.

Mediator: A third-party LLM observes the exchange and may speak between party turns. It sees only shared inputs—never any party’s persona, stance, or preferences—making mediation a test of social cognition.

Agentic Scenario Curation

A three-step pipeline that scales scenario construction without human authoring:

1. Seed Scenario Search: A Searcher agent (o4-mini-deep-research) gathers real conflict cases from the web across 8 domains (transactional, healthcare, environmental, business-to-business, public-policy, international, legal, intra-organizational).
2. Scenario Recast: A Scenario Writer agent (GPT-5.4) recasts each raw seed into the structured scenario tuple $(B, P, T, W)$.
3. Simulation-based Filtering: Each candidate is enacted in a multi-turn unmediated simulation (with DeepSeek-V3.2 role-playing agents). Scenarios are retained only when 3 independent replays all end in impasse (no consensus within 100 turns). This yields 40 hard scenarios, five per domain, forming the general condition.

Socio-Cognitive Probing

Starting from the 40 general-condition scenarios, SoCRATES applies 5 socio-cognitive axes independently (not stacked) to fresh copies, ensuring any performance shift is attributable to a single axis:

Together with the general condition, this yields 15 conditions (40 scenarios × 15 conditions = 600 runs per mediator).

Topic-Localized Evaluation

Benchmark Metrics

For each topic $T_j$, the evaluator outputs a 1–5 agreement rating (remapped to $[0,1]$) and averages across topics to produce a Consensus Score $S_{\leq t} \in [0,1]$ at every turn $t$. Each scenario yields two matched trajectories: $\{S^{\text{unmed}}_{\leq t}\}$ and $\{S^{\text{med}}_{\leq t}\}$. Three metrics are computed:

Intervention Timeliness rewards prompt response when consensus drops. Let $t_{\text{drop}}$ be a turn when $S^{\text{med}}_{\leq t}$ falls by at least $\tau = 0.1$, and $t_s$ the first intervention within $W=10$ turns:

$$\text{Intervention Timeliness} = \left(1 - \frac{t_s - t_{\text{drop}}}{W}\right) \times 100$$

Intervention Effectiveness measures the consensus lift produced by a mediator utterance over the following 5 turns:

$$\text{Intervention Effectiveness} = \frac{S^{\text{med}}_{\leq i+5} - S^{\text{med}}_{\leq i-1}}{1 - S^{\text{med}}_{\leq i-1}} \times 100$$

Consensus Gain measures the fraction of the unmediated consensus gap closed at the end state:

$$\text{Consensus Gain} = \frac{S^{\text{med}} - S^{\text{unmed}}}{1 - S^{\text{unmed}}} \times 100$$

where $S^{\text{med}}$ and $S^{\text{unmed}}$ are terminal Consensus Scores. When $S^{\text{unmed}} = 1$, the raw change $S^{\text{med}} - S^{\text{unmed}}$ is reported.

Automatic Evaluation

Rather than scoring every topic at every turn (which injects noise from irrelevant content), the topic-localized evaluator reads the dialogue once, locates the turns where each topic $T_j$ is actively in play (discussed or a party shifts position), records an agreement score and each party’s stance at those turns, and inherits the prior score for turns that do not touch $T_j$. DeepSeek-V3.2 serves as the backbone.

---

Empirical Validation / Results

Simulation Fidelity

To verify that the float-valued persona intensity steers agent behavior as intended, 7 simulators were tested on emotional reactivity at 4 scalar levels $\{0, 0.33, 0.66, 1\}$. Human annotators selected the more reactive side in A/B pairs (160 pairs per simulator, Krippendorff’s $\alpha = 0.75$).

Table 1: Simulation fidelity – DeepSeek-V3.2 achieves the highest accuracy, indicating reliable float-valued intensity control.

Topic-Localized Evaluation

The evaluator was validated against two expert annotators on 1,844 snippets from 144 mediator trajectories (inter-annotator agreement $\alpha = 0.86$). Alignment with average expert score measured via Pearson correlation $r$:

Table 2: Evaluator alignment with experts – SoCRATES more than doubles both baselines on trajectory-level correlation.

Benchmarking LLM Mediators

8 LLMs were benchmarked (2 proprietary, 6 open-source) across all 15 conditions (600 runs per mediator, 4,800 total). Table 3 reports consensus gain by domain:

Table 3: Conflict resolution performance across eight domains – Average consensus gain caps at 34.4, with domain means ranging from 41.3 (Transactional) down to 16.6 (Intra-organizational). Cell color intensity increases within each column for higher scores.

Key findings:

• No mediator clears half the unmediated gap in any domain; average consensus gain is only 25.9.
• Proprietary mediators lead (GPT-5.4-mini: 34.4, Gemini-3.1-FL: 33.0) but scale alone does not order the field (Nemotron-3-120B trails smaller Gemma-4-26B on some domains).
• Timeliness without effectiveness (Solar-Pro-3 and Qwen3-30B: highest timeliness, lowest consensus gain) shows that good timing is insufficient without effective content.
• Domain coverage shapes the verdict: restricting evaluation to transactional domains (where most prior testbeds concentrate) would overstate mediation ability.

Socio-Cognitive Adaptation Analysis

Figure 2 (profiles across axes) and Figure 3 (shift from general condition for strategy, emotion, culture) reveal:

• Strategic posture is the sharpest stress test: competing and accommodating postures cause large drops (18.9–64.1 and 13.8–66.8 respectively). Qwen3-235B suffers the largest drops despite high overall ranking.
• Emotional reactivity produces consistent degradation: composed pairs hold scores, reactive pairs degrade every mediator.
• Cultural identity produces systematic shifts: scores decline as cultural distance from U.S. norms grows, indicating weaker performance on East Asian cultural dynamics.

Intervention timing adaptation (Figure 4): The best intervention window varies by condition. For Strategy and Emotion, effectiveness rises early and falls off. For Multi-state and Long-context, effectiveness grows toward later turns. Stronger mediators (GPT-5.4-mini, Qwen3-235B) peak near each condition’s window; weaker ones trace flatter curves.

---

Theoretical and Practical Implications

• Mediation competence comprises distinct socio-cognitive abilities, not a single capability frontier. Even top-tier mediators show uneven profiles (e.g., GPT-5.4-mini and DeepSeek-V3.2 lose far more under Multi-state Tracking than Gemini-3.1-FL and Qwen3-235B). This calls for diagnostic, per-axis evaluation rather than aggregate scoring.
• General reasoning capability does not directly translate to mediation success. Frontier models that excel on math and coding benchmarks still close only a third of the consensus gap. The bottleneck is social adaptation to diverse conditions—varying strategy, emotion, culture, and context.
• Topic-localized evaluation is essential for reliable trajectory scoring. Per-turn baselines introduce noise from off-topic content that distorts consensus trajectories and compounds errors over time. The topic-localized approach achieves 0.82 expert alignment, enabling fine-grained, real-time metrics.
• Benchmark design matters: a benchmark restricted to transactional conflict overstates mediation ability. SoCRATES’s multi-domain, multi-axis design reveals that progress lies in adapting intervention strategy (timing and content) to the socio-cognitive demands of each conflict.

---

Conclusion

SoCRATES provides a unified benchmark for evaluating proactive LLM mediators across eight domains and five socio-cognitive axes, built on automatic scenario construction and a topic-localized evaluator that achieves high expert alignment (0.82 Pearson). Benchmarking eight frontier LLMs shows that even the strongest mediator closes only about one-third of the unmediated consensus gap, with performance varying sharply by socio-cognitive condition. Effective mediation hinges on adaptation to diverse contexts and party compositions, not on uniform capabilities. SoCRATES offers the controlled testbed to study this adaptation.

Limitations: (1) All conversations are in English, even for non-US cultural identities—extending to multilingual settings would reveal language effects. (2) SoCRATES focuses on consensus as the primary outcome; additional dimensions like party satisfaction, procedural fairness, and emotional repair remain for future work.

Ethical Considerations: Scenarios are synthesized from deep-research seeds with anonymization; no real people are involved as disputants. Crowd-sourced annotators and expert examiners were compensated above federal minimum wage and $35/hour respectively.