Summary of "A New Role for Relevance: Guiding Corpus Interaction in Agentic Search"

Summary (Overview)

• New role for relevance: The paper argues that relevance should directly guide corpus interaction (traversal order and match visibility) rather than merely selecting top-k content or constructing a workspace.
• RARG agent: The Relevance-Aware RipGrep Search Agent injects document-level relevance into grep traversal order, entry-point initialization, and match-level reranking to accelerate search convergence.
• State-of-the-art results: On BrowseComp-Plus (100K docs), RARG++ achieves 84% accuracy with GPT-5.4-mini (vs. 78% for DCI and RISE) while using only 23.9 tool calls (vs. 99.1 for DCI). On GPT-5.4, RARG++ reaches 91% accuracy, exceeding RISE by 9 points.
• Robust to scaling: When expanding the corpus to 1M documents, RARG++ retains 79% accuracy (vs. 69% for RISE-BM25) with minimal cost increase.
• Reasoning retrieval: On BRIGHT, RARG+ achieves 53.36 average nDCG@10, outperforming the retrieval-specialized NeMo agent (52.89).

Introduction and Theoretical Foundation

The paper re-examines the role of relevance in agentic search. Relevance is defined as a query-dependent estimate of how likely a document, passage, or matched excerpt is to contain useful evidence. Traditional retrieval-augmented generation (RAG) and retrieval agents use relevance to rank documents and expose a top-k set, but this conflates document relevance with evidence utility:

• A relevant document may hide the decisive fact in a small span omitted by truncation.
• In multi-hop search, a document's usefulness may emerge only after an intermediate entity is discovered.
• Independent query-document ranking cannot itself localize, connect, compare, or verify evidence.

Direct Corpus Interaction (DCI) addresses this by allowing agents to search raw documents with general-purpose tools like grep, but DCI is relevance-agnostic – it scans all locations equally, causing useful clues to appear late or be lost under output truncation. This leads to poor search convergence (how quickly iterative interactions narrow an open-ended search into verified evidence).

Recent advances (Pi-Serini, RISE, DR-DCI) use retrieval to construct a bounded candidate space, but relevance remains a mechanism for selecting the space rather than a fine-grained execution signal for interactions within it.

The paper argues that relevance should guide corpus interaction at two complementary resolutions:

• Global relevance: determines where a pattern-matching operation searches first (traversal order).
• Local relevance: determines which matched excerpts become visible under the agent's observation budget.

This preserves DCI's high-resolution interface while using retrieval scores as an execution prior.

Methodology

3.1 DCI-based Agent Structure

RARG is built on DCI-Agent-Lite, exposing two tools:

• Bash(command): issues grep commands to find local matches across the corpus.
• Read(doc_path, offset, limit): returns lines of a document when a match calls for more context.

Context management uses Level-3 compaction: truncate each tool result to C characters, and once the context length exceeds a threshold, retain only the most recent T turns.

3.2 Document-Level Relevance Insertion (RARG)

The key innovation is a new tool embed_recall(scope_query) that takes an LLM-generated query, uses an embedding retriever to rank documents, and writes the top-10,000 document paths to a temporary file /tmp/scope_{N}.txt. The LLM then uses:

cat /tmp/scope_{N}.txt | xargs -d '\n' rg [OPTIONS] "PATTERN"

A rule-based pass injects -j1 to force sequential, single-threaded scanning, preserving the ranked order so that matches from more relevant documents are exposed earlier. This turns document-level relevance into search order rather than top-k content selection.

3.3 Entry Point Initialization (RARG+)

To avoid the agent wasting steps before finding a good entry point, the top-X documents of a scope are split into paragraphs (400–1000 characters), encoded with the embedding model, scored against the scope query, and the top-10 paragraphs are appended to the embed_recall output wrapped in <qr_paragraph></qr_paragraph>. This serves as a starting point without functioning as the answer itself.

3.4 Match-Level Relevance Insertion (RARG++)

To further speed convergence, the raw grep output is truncated, and informative excerpts from lower-ranked documents may be lost. RARG++ reranks up to M matches using the embedding model and presents only the top m to the LLM. The reranking query is constructed by concatenating the scope query (global goal) with the grep keywords (local intent):

Instruct: ...
Query: [scope query]
RG focus: [keyword1] [keyword2] ...

This gives more candidate clues a chance to reach the LLM under the observation budget. A generative variant (where the LLM supplies rerank_query) is also explored but degrades performance.

Empirical Validation / Results

4.2 Results on Challenging QA (BrowseComp-Plus)

Table 1: Main results on 100-query BrowseComp-Plus (100K doc corpus)

Key findings:

• RARG++ achieves the highest accuracy on every backbone while sharply reducing interaction cost.
• Relevance acts as an execution prior over rg traversal, not as the evidence channel itself (low Search counts).
• The generative variant of RARG++ converges fastest (17.8 tools) but drops to 75% accuracy, indicating a train–evaluation gap.

4.3 Scaling the Corpus (100K → 1M documents)

Table 2: Results after scaling to 1M documents (GPT-5.4-mini)

RARG++ maintains a 10-point margin over RISE-BM25. The cost increase is minimal, but performance degrades due to incidental lexical matches from long noisy documents.

4.4 Results on Reason-Intensive Retrieval (BRIGHT)

Table 3: Retrieval effectiveness on BRIGHT (NDCG@10)

RARG+ achieves the best average nDCG@10, outperforming the retrieval-specialized NeMo agent. The ordering flips relative to QA: RARG+ > RARG > RARG++, indicating that match-level reranking concentrates the observation budget (good for depth-first QA) but hurts when the task rewards wide recall.

4.5 Analysis of RARG Behavior

Figure 3: Heatmap of relevant-document hits over scope ranks. RARG stays front-loaded (relevant documents scanned first), while RARG++ recovers hits from lower-ranked documents via match-level reranking, confirming the two-level design.

Figure 4: Scope recall and RG coverage. Mini achieves 95–97% scope recall; nano is lower due to weaker instruction following. The -j1 sequential scan dominates Bash usage, confirming the relevance-aware traversal.

Theoretical and Practical Implications

Theoretical implications:

• Relevance should be viewed as an execution signal for corpus interaction, not a fixed content bottleneck. This shifts the paradigm from "retrieve then reason" to "retrieve to guide reasoning."
• The coarse-to-fine design (document-level order → entry-point initialization → match-level visibility) provides a principled framework for integrating retrieval with fine-grained search.
• The paper identifies a missing role of relevance in agentic search, filling the gap between corpus-level prioritization and local evidence exploitation.

Practical implications:

• RARG substantially improves the accuracy–efficiency trade-off, making expensive deep reasoning more affordable (e.g., 9-point accuracy gain with 46% cost reduction on GPT-5.4).
• The method is robust to corpus scaling, retaining effectiveness even when the corpus grows tenfold.
• For reasoning-intensive retrieval, relevance-aware interaction (RARG+) outperforms both pure retrieval agents and relevance-agnostic DCI.
• The approach is practical: it can be built on top of existing DCI harnesses and embedding models.

Conclusion

The paper revisits the role of relevance in agentic search, arguing that it should guide corpus interaction rather than merely select inputs. The proposed RARG agent turns retrieval scores into an execution prior for grep exploration at two resolutions:

• Document-level relevance orders rg traversal so promising documents are scanned first.
• Match-level relevance controls which local excerpts remain visible under a limited observation budget.
• Entry-point initialization gives the agent a precise place to begin.

Across challenging QA, corpus scaling, and reasoning-intensive retrieval, RARG advances the accuracy–efficiency frontier over retrieval-only agents, unrestricted DCI, and retrieval-constructed interaction spaces. The analysis confirms that RARG keeps hits front-loaded toward relevant documents while recovering useful evidence from lower-ranked ones.

Limitations:

• Depends on embedding model quality; different granularities (document vs. match) may not be served equally well by the same model.
• Sequential -j1 scanning and reranking incur extra latency.
• Sensitive to backbone instruction-following ability (GPT-5.4-nano is less reliable).
• Not immune to corpus-induced interference from long noisy documents.
• Generative reranking variant shows a train–evaluation gap that needs bridging.

Future work includes closing the generative reranking gap, broader validation across backbones and domains, and reducing the weakness of local matching under noisy distractors.