Summary (Overview)

• Recursive Self-Improvement Framework: AREX introduces a bi‑level architecture that alternates between an inner research loop (evidence gathering and provisional answer construction) and an outer self‑improvement loop (constraint‑wise audit and targeted follow‑up), motivated by the discovery–verification asymmetry in deep research.
• Autonomous Context Updating (ACU): The model learns to invoke a dedicated update_context tool that compresses the growing interaction history into a compact improvement state, preserving verified evidence, unresolved constraints, rejected candidates, and next‑step plans without relying on an external model.
• Step‑Aware Training: A multi‑stage pipeline combines supervised agentic mid‑training (progressive capability acquisition, key‑step focused supervision) with step‑aware reinforcement learning (hierarchical step‑level GRPO and key‑step reward shaping) to handle sparse credit assignment in long‑horizon trajectories.
• Strong Empirical Results: AREX is instantiated as a dense 4B model (Turbo) and a 122B‑A10B MoE model (Base). On six benchmarks (BrowseComp, GAIA, DeepSearchQA, WideSearch, HLE, xbench‑2510), AREX‑Base substantially outperforms comparable‑scale open‑source baselines and remains competitive with frontier closed‑source models using significantly more activated parameters.

Introduction and Theoretical Foundation

Deep research requires discovering answers that simultaneously satisfy multiple coupled constraints. The paper identifies a fundamental discovery–verification asymmetry:

• Discovery is costly because the search space is large and sparsely informative.
• Verification of a candidate can often be decomposed into tractable constraint‑wise checks, revealing which claims are supported, unresolved, or conflicting.

Existing systems often extend a single search trajectory with more reasoning, tool calls, or context, but this does not guarantee systematic progress—early errors may persist, exhausted directions may be revisited, and partially valid candidates may be accepted prematurely. The key insight of AREX is that verification should not be a final filter but an active control signal that recursively refines the research state and solution. By converting a provisional answer into a partially verified state, the agent preserves established progress, isolates remaining uncertainty, and formulates a more targeted next research problem.

Methodology

Overall Framework

AREX uses a hierarchical bi‑level structure (Figure 2 in the paper). Given a query (x), it derives a research objective (q^{(1)}). The inner research loop executes actions, integrates evidence, and produces a provisional answer with supporting evidence and a confidence score. The outer self‑improvement loop evaluates this structured result: if confidence exceeds a threshold (\tau) the answer is accepted; otherwise it decides to refine (preserve useful findings and target unresolved constraints) or restart (discard the trajectory and begin anew from the original problem).

Inner Research Loop

At step (t) of recursive round (k), the interaction trajectory is

[ h^{(k)}_t = \left[ \left\langle m^{(k)}_i, a^{(k)}_i, o^{(k)}i \right\rangle \right]{i=1}^{t}, \tag{1} ]

where (m) is the model’s analysis, (a) a research action, and (o) the observation. The research policy generates the next analysis and action:

[ \left\langle m^{(k)}{t+1}, a^{(k)}{t+1} \right\rangle = \pi_\theta!\left(x, q^{(k)}, h^{(k)}t\right), \quad o^{(k)}{t+1} = \mathcal{T}!\left(a^{(k)}_{t+1}\right). \tag{2} ]

The trajectory is updated by appending the new interaction: (h^{(k)}{t+1} = h^{(k)}t \oplus \langle m^{(k)}{t+1}, a^{(k)}{t+1}, o^{(k)}_{t+1} \rangle).

Autonomous Context Updating (ACU)

To prevent the growing history from exceeding the context window and to focus on decision‑relevant information, AREX learns a dedicated update_context tool. Given the trajectory (h^{(k)}_t), it constructs a refreshed improvement state:

[ z^{(k)}t = f\theta!\left(h^{(k)}_t\right). \tag{4} ]

The state preserves verified findings, current candidates, unresolved constraints, validity concerns, rejected candidates, and the next‑step plan. After an update at step (\tau), the effective context becomes

[ \bar{h}^{(k)}t = z^{(k)}\tau \oplus \left[ \langle m^{(k)}_i, a^{(k)}_i, o^{(k)}i \rangle \right]{i=\tau+1}^{t}. \tag{5} ]

Subsequent actions are conditioned on (\bar{h}^{(k)}_t).

Structured Answer Externalization

When the inner loop finishes, it outputs a structured result:

[ r^{(k)} = F_\theta!\left( \bar{h}^{(k)}_{T_k} \right) = \left\langle y^{(k)}, \mathcal{E}^{(k)}, s^{(k)} \right\rangle, \tag{8} ]

where (y^{(k)}) is the provisional answer, (\mathcal{E}^{(k)}) contains supporting evidence, and (s^{(k)} \in [0,100]) is an answer‑level confidence score.

Outer Self‑Improvement Loop

The outer loop receives (r^{(k)}) and the effective context. The decision rule is:

[ d^{(k)} = \begin{cases} \text{Accept}, & s^{(k)} \ge \tau, \ \text{Refine}, & s^{(k)} < \tau \wedge v^{(k)} = 1, \ \text{Restart}, & s^{(k)} < \tau \wedge v^{(k)} = 0, \end{cases} \tag{10} ]

where (v^{(k)}) indicates whether the trajectory is recoverable. A new research objective (q^{(k+1)}) is generated for the next round.

Training Data Construction

1. Recursive Research Task Synthesis: Generates verifiable problems with multiple constraints (\mathcal{C}(y) = {c_1,\dots,c_n}) and transforms them into indirect, multi‑hop queries. Each task ((x, y)) is verified for correctness, uniqueness, and evidence availability.
2. Teacher Trajectory Collection: Strong teacher models sample trajectories (\tau_i \sim \pi_{\text{teacher}}(\tau|x)) in the same research environment. Quality control retains only trajectories that exhibit iterative investigation, valid tool use, evidence‑grounded answers, and confidence above a threshold.

Training Pipeline

Multi‑Stage Agentic Mid‑Training

• Progressive capability acquisition: First train on browse‑intensive multi‑round trajectories (tool use, evidence acquisition); then on reasoning‑intensive trajectories (long‑form thinking, hypothesis verification). This ordering reduces interference.
• Key‑step focused consolidation: Identify decisive steps (first evidence discovery, path rejection/redirection, key context‑updates) using rule‑based detectors. Apply supervision only to these steps while preserving the full preceding context. The step‑level loss is:

[ \mathcal{L}{\text{key}} = -\mathbb{E}{s_j \sim \mathcal{K}}!\left[ \frac{1}{|s_j|} \sum_{k=1}^{|s_j|} \log \pi_\theta(a_{j,k} | c_{j,k}) \right]. ]

Step‑level analysis (Figure 4) shows that key steps have substantially higher loss than ordinary steps after full‑trajectory training, confirming they are underlearned.

Step‑Aware Reinforcement Learning

Adopts turn‑level policy optimization with hierarchical step‑balanced normalization and key‑step shaping. For trajectory (i) with steps (j) and tokens (k), the token‑level probability ratio is:

[ r_{i,j,k}(\theta) = \frac{\pi_\theta(a_{i,j,k} | c_{i,j,k})}{\pi_{\theta_{\text{old}}}(a_{i,j,k} | c_{i,j,k})}. ]

The step‑level policy ratio uses the geometric mean for length normalization:

[ \rho_{i,j}(\theta) = \exp!\left( \frac{1}{L_{i,j}} \sum_{k=1}^{L_{i,j}} \log r_{i,j,k}(\theta) \right). ]

The objective averages over steps within each trajectory and then over trajectories:

[ \mathcal{L}{\text{step}} = -\mathbb{E}{x \sim \mathcal{D}}!\left[ \frac{1}{G} \sum_{i=1}^{G} \frac{1}{M_i} \sum_{j=1}^{M_i} \min!\left( \rho_{i,j}(\theta) A_{i,j},; \text{clip}(\rho_{i,j}(\theta), 1-\epsilon, 1+\epsilon) A_{i,j} \right) \right]. ]

Step rewards are shaped with an auxiliary key‑step bonus:

[ A_{i,j} = A^{\text{out}}i + \lambda{\text{key}} \tilde{B}{i,j}, \quad \tilde{B}{i,j} = \mathbb{I}[R_i > 0] \cdot B_{i,j}. ]

Empirical Validation / Results

Overall Performance

Table 1 shows results across six benchmarks.

Table 1: Comparison results on deep search and agentic reasoning tasks. AREX-Base (10B active) strongly outperforms Qwen3.5-397B and is competitive with larger frontier models.

Inference Framework Analysis

Context‑Update Behavior (Table 2):

• ACU is invoked in 80.3% of BrowseComp cases.
• Mean context at update is 25,721 tokens (well below the 128K limit).
• Most common trigger: revising search strategy (66.9%).
• Update content most frequently preserves unresolved constraints (95.5%) and next‑step plans (96.4%).

Table 2: Context‑update behavior on BrowseComp.

Recursive Self‑Improvement (Table 3):

• ACU alone improves accuracy from 59.6 to 71.4 (+11.8).
• Outer self‑improvement loop alone improves from 59.6 to 69.8 (+10.2).
• Full system (both) achieves 82.5 (+22.9 over baseline).

Table 3: Effect of ACU and outer loop on BrowseComp accuracy.

Confidence Score (Figure 3): Correct outputs concentrate in the 90–100 bin (89.3% without ACU, 95.9% with ACU). Incorrect outputs have a large low‑confidence mass (61.0% below 60 without ACU, 55.2% with ACU). This supports the confidence‑based decision procedure.

Ablation Studies (Table 4)

Table 4: Ablation of the AREX training recipe on BrowseComp.
All variants use the same initialization; replacing each component degrades accuracy. The largest drop (−8.4) occurs when key‑step supervision is replaced by random‑step replay.

Step‑Level Loss Analysis (Figure 4): After full‑trajectory mid‑training, ordinary steps have average loss 0.232, while evidence discovery (0.277, +19%), path rejection/redirection (0.298, +28%), and key context‑update (0.300, +29%) all have significantly higher loss, confirming these decisive steps remain underlearned.