LiveMACE: Process-Aware Evaluation of LLM Agent Capabilities in Evolving Markets

Summary (Overview)

• Process-aware evaluation framework: LiveMACEBench introduces a benchmark that evaluates LLM agents not just by outcomes (e.g., returns) but by the process through which outcomes are produced, distinguishing between mechanism access, mechanism use, and realized task benefit.
• Live financial markets as testbed: Five frontier LLMs (GPT-5.4, DeepSeek-V3.2, Gemini-3.1-Pro, Grok-4.20, Qwen3-Max) operate in live cryptocurrency and U.S. equity markets over 30 days, with continuously evolving external conditions and persistent, non-resettable agent state.
• Controlled agent configurations: The benchmark evaluates four capability-specific variants (Tool Use, Persistent Memory, Rule Following, Multi-Agent Collaboration) on top of a common ReAct scaffold, enabling matched comparisons.
• Outcome–capability gap identified: Realized returns often diverge from capability-specific measurements; similar outcomes arise from markedly different patterns of mechanism use (e.g., 21 pairwise return-rank reversals vs. 0 reversals for tool-use rankings).
• Distinct capability bottlenecks exposed: Trace-level diagnostics reveal stage-specific failures—e.g., high information coverage masks inefficient tool invocation; memory accumulation does not guarantee effective reuse; rule execution can diverge from auditability.

Introduction and Theoretical Foundation

The paper addresses a fundamental distinction in agent evaluation: evaluating agent success versus evaluating agent capability. Outcome metrics reveal whether an agent succeeds but fail to identify the capabilities that produced that success. This attribution problem intensifies when:

1. Agents operate over long horizons with tools, persistent memory, explicit policies, and multi-agent architectures.
2. The environment itself evolves—external conditions change independently of the agent while consequences of earlier actions persist.

The authors argue that observed performance reflects not only the agent and its mechanisms, but also accumulated state, environmental variation, and their interaction. Therefore, evaluating capability in evolving settings requires examining the process that produces an outcome, not only the outcome itself.

Live financial markets provide a natural testbed: prices and information continuously evolve under external forces, continually eliciting information gathering, experience reuse, constraint following, and coordination. The theoretical foundation builds on prior work moving from outcome-only toward process-aware evaluation (AgentBoard, AgentQuest) and extends it to persistent trajectories in environments whose dynamics are not task-scripted.

The core theoretical contribution is separating three quantities that outcome-only evaluation conflates:

• Access to an agentic mechanism
• Effective use of that mechanism
• Realized task benefit

Methodology

Environment Design

• Live markets: Cryptocurrency (Hyperliquid) and U.S. equities (Alpaca SIP feed), with six cryptocurrencies and seventeen equities in the tradable universe.
• Decision rounds: Scheduled every four hours on a shared decision-time grid; each round is one complete agent invocation (reasoning, tool interactions, evidence gathering, trading actions).
• Persistent state: Account positions carry across rounds; complete decision traces record observations, reasoning, mechanism interactions, and actions.

Controlled Agent Configurations

Capability-Specific Evaluation Metrics

Tool Use:

• Judge-based: Tool Relevance (TR), Invocation Efficiency (IE), Information Coverage (IC), Evidence Faithfulness (EF)
• Objective: Routing Quality (RQ), Valid Call Rate (VCR), Hallucination-Free Rate (HFR)
• Combined: Tool Call Score (TCS)

Persistent Memory:

• Judge-based: Memory Content Quality (MCQ), Memory Usage Quality (MUQ)
• Objective (downstream effects relative to memory-free config): Change in Maximum Drawdown ($\Delta MDD$), Change in Tail Loss ($\Delta TL$), $\Delta PnL$
• Combined: Memory Score (MS)

Rule Following:

• Objective: Hard-rule Pass Rate (HPR), Soft-rule Satisfaction (SRS)
• Combined: Rule Satisfaction Score (RSS), with hard-rule compliance gating soft-rule satisfaction
• Judge-based: Rule Coverage (RC), Conflict Handling (CH)
• Combined: Rule Auditability Score (RAS)

Multi-Agent Collaboration:

• Objective: Role Diversity (RD), Evidence Integration (EI)
• Combined: Observable Collaboration Score (OCS)

Empirical Validation / Results

Outcome–Capability Gap

• Return rankings fluctuate: 21, 19, 19, and 29 pairwise reversals for tool use, memory, rule following, and collaboration configurations, respectively.
• Capability rankings are stable: Only 0, 5, 8, and 2 reversals, respectively.
• Capability leaders differ from return leaders: E.g., Grok-4.20 achieves highest return in tool-use config despite lowest TCS; Gemini-3.1-Pro leads rule satisfaction while DeepSeek-V3.2 leads rule-condition returns.

Capability Scores (Full Period)

Tool Use Findings

• High coverage masks inefficiency: All models achieve Information Coverage > 0.8, yet Invocation Efficiency is consistently weakest (0.581–0.731).
• Stage-specific failure modes: Qwen3-Max has lowest Routing Quality (RQ = 0.736) but near-perfect Valid Call Rate (VCR = 0.978); Grok-4.20 shows the converse (RQ = 0.805, VCR = 0.666); GPT-5.4 is strong at both (RQ = 0.920, VCR = 0.998).

Persistent Memory Findings

Table 1: Persistent-memory lifecycle across backbones

Key findings:

• Persistent access ≠ persistent learning: GPT performs 332 retrievals but stores only 1 memory; Gemini accumulates 93 entries.
• Accumulation, content quality, and effective use are distinct: Gemini has largest store with high MCQ (0.938) but lowest MUQ (0.497); Grok stores only 12 entries with lower MCQ (0.760) but highest MUQ (0.855).
• Consistent downside-risk improvement: All five memory-augmented agents reduce tail loss; effects evolve over time (Gemini shifts from negative to positive late-phase differential).

Rule Following Findings

Table 2: Trading outcomes and rule-following metrics

Key findings:

• Execution and auditability diverge: DeepSeek achieves strong RSS (0.847) but low RAS (0.623); Gemini is strong on both (RSS = 0.872, RAS = 0.882).
• Return does not identify rule-following capability: DeepSeek and GPT achieve nearly identical returns (+5.765% vs. +5.738%) but differ substantially in HPR (0.877 vs. 0.682).
• Recurring bottleneck: R1-02 (single-asset concentration limit) violations arise from incorrectly combining existing exposure, incremental orders, and price movements.

Multi-Agent Collaboration Findings

Table 3: Paired changes from ReAct to Multi-Agent

Key findings:

• Model-dependent gains with common risk cost: Multi-Agent improves returns for three models (up to +3.72 points) but reduces Qwen3-Max by −4.39 points; all models show larger drawdowns and more negative tail losses.
• Same harness, distinct patterns: DeepSeek shows most balanced behavior (RD = 0.760, EI = 0.852); Gemini is evidence-centric (RD = 0.492, EI = 0.984); GPT nearly collapses to single-role workflow (RD = 0.012, EI = 0.518).
• Observable collaboration ≠ realized performance: GPT improves return despite almost no role diversity; Qwen retains non-trivial evidence integration but underperforms ReAct.

Theoretical and Practical Implications

Theoretical Implications

1. Outcome–capability gap formalized: The paper provides empirical evidence that realized outcomes and capability-specific measurements capture distinct dimensions of agent evaluation, challenging the assumption that performance metrics alone suffice for agent assessment.

2. Mechanism access ≠ effective use ≠ realized benefit: The controlled configuration design demonstrates that these three quantities must be measured separately—an agent may have access to a mechanism, use it poorly, and still achieve good outcomes (or vice versa).

3. Process-aware evaluation as a paradigm: The work extends the shift from outcome-only evaluation toward process-aware measurement, showing that live evolving environments can serve as diagnostic substrates rather than mere leaderboards.

4. Stage-specific capability decomposition: The trace-level diagnostics reveal that capabilities themselves decompose into separable stages (e.g., tool discovery vs. tool execution; memory construction vs. retrieval vs. application), each with distinct failure modes.

Practical Implications

1. Benchmark design: LiveMACEBench provides a template for evaluating persistent LLM agents in naturally evolving environments, with implications for any domain where external conditions change independently of the agent.

2. Agent development: The diagnostic metrics identify specific bottlenecks (redundant tool interactions, ineffective memory reuse, unauditable rule execution, limited evidence integration) that can guide targeted improvements in agent design.

3. Risk assessment: The finding that memory consistently reduces downside risk while having heterogeneous return effects suggests that capability evaluation should consider risk-adjusted and process-level metrics, not just returns.

4. Multi-agent system design: Evidence that the same collaboration harness induces markedly different collaboration patterns across models highlights the need for model-specific evaluation of multi-agent architectures.

Conclusion

LiveMACEBench demonstrates that evaluating agents in evolving environments requires measuring the mechanisms through which outcomes arise, not just ranking outcomes. Across five frontier LLMs and approximately 30 days of live evaluation, the benchmark establishes:

• A pronounced outcome–capability gap: realized performance varies substantially over time and does not reliably recover how effectively agents use available mechanisms.
• Distinct capability bottlenecks across tool use, persistent memory, rule following, and multi-agent collaboration, showing that similar outcomes can mask different capabilities while mechanism access need not translate into effective use or improved performance.

The paper's contributions are threefold: (1) a process-aware evaluation framework distinguishing realized outcomes from producing mechanisms; (2) controlled diagnosis of agentic mechanisms under matched conditions; (3) a naturally evolving testbed for persistent evaluation in live financial markets.

Future directions include extending process-aware evaluation across longer horizons, broader market regimes, and other continuously evolving environments beyond financial markets.