Summary (Overview)

• New benchmark for Long-Term Coherence: MerchantBench evaluates LLM agents as autonomous e-commerce store operators over 365 simulated days, requiring sustained decision-making across four interdependent components: Product Sourcing, Listing and Pricing Control, Cash-Flow Management, and Mixed-Latency Feedback Adaptation.
• Real-world grounding: The environment is built on 98,843 real product records from the 1688 wholesale marketplace, with calibrated demand patterns, supplier events, and order-level risk profiles.
• Order-level simulation with delayed feedback: Actions produce immediate cash commitments, while fulfillment failures, refunds, and bad reviews become observable only after variable delays, forcing agents to associate later outcomes with earlier decisions.
• Large performance gap: The best LLM configuration (Qwen3.7-Max under Hermes) achieves only 27.3% of the mean final net assets achieved by human participants. GPT-5.6 Sol under Hermes reaches 24.3% of human performance.
• Identified failure modes: Agents suffer from Operational Coherence decay (declining activity over time) and Strategic Coherence failures (goal abandonment, ineffective policy revision despite accumulating evidence).

Introduction and Theoretical Foundation

Large language model agents are increasingly deployed as autonomous tool users, yet most benchmarks focus on bounded tasks with immediate success criteria. Real-world deployments require Long-Term Coherence – the capacity to preserve purposeful behavior across extended horizons while adapting decisions to accumulated evidence. Evaluating this requires a persistent environment where actions constrain future choices, feedback arrives at heterogeneous delays, and incoherent behavior produces measurable cumulative effects.

Seller-side e-commerce provides a suitable testbed because it involves:

• Recurrent and interdependent decisions over product sourcing, pricing, cash management, and order fulfillment.
• Feedback generated through individual order lifecycles: listing/pricing decisions commit cash immediately, while fulfillment failures and after-sales outcomes become observable only later (see Figure 1).
• A non-stationary product catalog with full-year demand trajectories, requiring agents to continually identify new opportunities and revise their portfolio.

MerchantBench formulates store operation as a partially observable Markov decision process (POMDP):

$$\mathcal {M} = \langle \mathcal {S}, \mathcal {A}, P, \mathcal {O}, Z, R, \mu_ {0}, H _ {c} \rangle .$$

The horizon is $H_c = 8,760$ hourly steps over 365 days, with agent decision windows every 12 steps. The objective is expected terminal net assets:

$$\begin{aligned} J (\pi) &= \mathbb{E}_{\pi} [ R (s_{T}) ], \\ R (s_{T}) &= B_{T} + D_{T} + I_{T} + Q_{T}, \end{aligned}$$

where $B_T$, $D_T$, $I_T$, and $Q_T$ are cash balance, security deposit, funds in transit, and receivables.

Methodology

Environment Design

MerchantBench is grounded in 98,843 real product records from 1688 (Alibaba's wholesale marketplace) with 365-day demand histories (June 2025 – May 2026) across 10 product categories. The environment couples:

• Upstream Supplier Simulation: Time-varying procurement prices, inventory, and three event types (Price Change, Product Delisting, Shipment Delay) with calibrated probabilities.
• Downstream Order Simulation: Converts daily demand traces into individual orders. Hourly arrival intensity for product $i$ listed by merchant $m$ at time $t$ is:

$$\lambda_{m,i,t} = D_{i,d(t)} \, w_{c(i),h(t)} \, r_{m,t} \, \ell_{m,i,t} \, \left( \frac{p_{m,i,t}}{p_i^{\text{ref}}} \right)^{-\epsilon_i},$$

where $D_{i,d(t)}$ is real daily demand, $w$ distributes demand across hours, $r_{m,t}$ is store rating multiplier, $\ell_{m,i,t}$ is listing exposure (cold-start ramp then exponential decay), and $\epsilon_i$ is price elasticity.

• Order Lifecycle: Follows a drop-shipping model. Orders progress through Ordered → Shipped → Delivered → Settled. Six abnormal outcomes (Cancellation, Stockout, Late Shipment, Returnless Refund, Return and Refund, Bad Review) incur fines and affect store rating.
• Store Rating: Continuous rating $R_{m,d}$ updated daily using discounted evidence:

$$R_{m,d} = \frac{\alpha R_0 + \sum_{j \in \mathcal{C}_{m,d}} \gamma^{d-1-d_j} v_j u_j}{\alpha + \sum_{j \in \mathcal{C}_{m,d}} \gamma^{d-1-d_j} v_j},$$

with prior $R_0=4.0$, $\alpha=20$, $\gamma = 2^{-1/30}$, and outcome-specific experience scores $u_j$ and weights $v_j$. The rating maps to a star level (1–5) that multiplies subsequent demand.

Agent Interface

26 merchant tools are exposed across four categories:

• Product Sourcing: daily market reports, catalog search, product/supplier detail queries.
• Listing and Pricing Control: list, delist, adjust price, review listings, query performance.
• Cash-Flow Management: balance queries, store snapshot, platform rules, cash pipeline.
• Supplier and Order Monitoring: supply chain anomalies, order queries, order updates.

Experimental Setup

• 8 LLMs evaluated: GPT-5.6 Sol, Claude Opus 4.8, Qwen3.7-Max/Plus, GLM-5.2, DeepSeek-V4-Pro/Flash, Kimi K2.6.
• 2 agent frameworks: ReAct (minimal controller) and Hermes (with built-in code execution, memory, planning, skills).
• 3 runs per LLM×framework = 48 total runs, each starting with RMB 2,000 cash + RMB 1,000 deposit, 50 listing slots.
• Baselines: Rule-based daily checks + 3 human participants (no prior e-commerce experience).
• Evaluation metrics: Business Performance (Final Net Assets, GMV, Profit Margin, Orders), Store Reliability (Fines, Avg. Store Rating, Anomaly Rate), Long-Horizon Activity (Avg. Active Listings, Sustained Window Rate (SWR), Tool Calls).

Empirical Validation / Results

Main Performance (Table 1)

Table 1: Business performance, store reliability, and long-horizon activity after 365 simulated days. Values are means over three runs. Final net assets and GMV in thousands of RMB, fines in RMB, rates in percent. SWR = Sustained Window Rate.

• Best LLM: Qwen3.7-Max under Hermes achieves 59.46k RMB net assets (27.3% of human 217.61k). GPT-5.6 Sol under Hermes achieves 52.93k (24.3%).
• Framework effect: Hermes produces 53.3% higher net assets on average than ReAct, but benefits vary widely across models (from +11.5% for Claude Opus to +187.8% for Qwen3.7-Max; Kimi K2.6 is 4.1% lower).
• Performance variability: GPT-5.6 Sol under ReAct is most stable (CV 3.3%); Qwen3.7-Max under Hermes has highest mean but CV 55.1%.

Long-Term Coherence Analysis

Operational Coherence: Many LLM agents progressively reduce activity. SWR ranges from 10.6% (Kimi K2.6 ReAct) to 99.4% (GPT-5.6 Sol ReAct), while humans maintain 100%. Qwen3.7-Max under Hermes drops from 64% effective window rate in June to 31% in May (Figure 5).

Strategic Coherence:

• Goal Consistency failures: Control-Loop Narrowing (sourcing collapses into reactive supplier handling) and Premature Abandonment (agent concludes store is unrecoverable, e.g., Kimi K2.6 Hermes run took no action in 355 of 523 remaining windows after Day 104).
• Evidence-Calibrated Adaptation failures: Agents fail to revise policies as liquidity changes, seasonal demand shifts, or evidence accumulates. Human operators broaden procurement price ranges over time (from RMB 43-53 in first 3 months to RMB 59-91 in last 3 months), while many LLM agents keep flat price trajectories.
• Memory errors: Claude Opus 4.8 falsely inferred that delisting weak products concentrates traffic; Qwen3.7-Max misremembered the simulation end date and stopped sourcing 83 days early.

Order-Level Risk Propagation

• Product-Level Risk Response: GPT-5.6 Sol and Kimi K2.6 attributed adverse outcomes to specific listings and replaced them; Qwen3.7-Plus retained risky products; DeepSeek-V4-Pro did not revise affected listings after refunds. Humans performed full chains (delist risky product → search similar keyword → replace).
• Store-Level Rating Adaptation: GPT-5.6 Sol lowered prices on proven products to recover rating after anomalies; Qwen3.7-Max repriced 40 listings after a bad review dropped rating to 3 stars.

Hermes Case Study

• Code Use: GPT-5.6 Sol dominated native code execution (13–15 calls per run), computing launch prices as $p = \max(1.70c, c+6)$ rounded to 0.9 endings. Other models rarely used code tools.
• Skill Evolution: 17 of 24 Hermes runs created a "RealShop" skill. GPT-5.6 Sol created skills with 7–15 patches and 10–17 uses per run, framing the task as repeated portfolio allocation.

Theoretical and Practical Implications

• Theoretical: MerchantBench formalizes Long-Term Coherence as a distinct evaluation dimension beyond isolated task completion. The POMDP formulation with mixed-latency feedback and non-stationary demand provides a rigorous testbed for studying how agents maintain goal-directed behavior over extended horizons.
• Practical: The substantial gap between LLMs and humans (best LLM achieves ~27% of human net assets) highlights critical weaknesses in current agents: activity decay, premature abandonment, failure to revise strategies based on accumulated evidence, and inability to perform demand-preserving product replacement. These findings suggest that scaling model size alone is insufficient; agent frameworks must incorporate robust memory, self-monitoring, and evidence-calibrated policy revision mechanisms.
• Benchmark design: The 365-day horizon, real-world data grounding, and order-level simulation with delayed feedback provide a realistic and challenging setting that reveals failure modes not captured by shorter or static benchmarks.

Conclusion

MerchantBench introduces the first benchmark for evaluating Long-Term Coherence through persistent seller-side e-commerce operation. Grounded in 98,843 real product records over 365 simulated days, the environment couples immediate supplier feedback with delayed order outcomes, requiring agents to follow individual order lifecycles and revisit earlier decisions. Across 48 runs with eight LLMs and two frameworks, the best LLM configuration achieves only 27.3% of human mean final net assets. Trace analyses reveal two primary failure modes: Operational Coherence decay (declining activity) and Strategic Coherence failures (goal abandonment, strategy drift uncalibrated to evidence). These results demonstrate that current LLM agents lack the sustained, evidence-calibrated decision-making required for real-world long-horizon deployments. Future work should explore improved memory architectures, self-monitoring mechanisms, and reinforcement learning approaches to bridge this gap.