Summary (Overview)

• Conducts a controlled scaling study of four RAG paradigms (lexical, dense, graph-based, agentic) on a 28-tier corpus ladder spanning 1,144 to 511,959 documents (≈450-fold scale), with questions, gold evidence, and adversarial distractors held fixed.
• Reveals a scale-dependent crossover: raw-file agentic search (File-System Agent) leads at small corpus sizes, but BM25 overtakes around 10M corpus tokens and leads at every larger shared tier, with a margin approaching 20 points at full scale (601M tokens).
• BM25 anchors the low-cost end of the Pareto frontier, requiring no LLM-based construction, while graph-based RAG methods (HippoRAG 2, MS-GraphRAG, LightRAG, LinearRAG) encounter construction walls before reaching full scale.
• Agent+BM25 (replacing raw-file search with BM25 within the same agentic harness) achieves 69.4 official combined score at full scale, versus 36.9 for the raw-file agent, at one ninth of the query tokens, demonstrating that agency is most effective after global candidate ranking.
• Unified cost metering (build tokens, query tokens, latency) and dual-judge robustness checks confirm that the crossover is a genuine scaling effect, not an artifact of a single corpus size or evaluation protocol.

Introduction and Theoretical Foundation

Retrieval-Augmented Generation (RAG) improves LLM factual grounding by retrieving external evidence (Lewis et al., 2021). Methods have diverged into several paradigms:

• Lexical retrieval (BM25; Robertson & Zaragoza, 2009) builds an inverted index without LLM involvement.
• Dense retrieval (Karpukhin et al., 2020) uses embeddings and nearest-neighbor search.
• Graph-based RAG (e.g., MS-GraphRAG, LightRAG, HippoRAG 2) invests heavily in indexing time, running LLMs over every chunk to extract entities and relations, then exploiting the structure at query time.
• Agentic retrieval (e.g., File-System Agent) spends cost at query time, using an LLM to iteratively explore the corpus through tool calls (Yao et al., 2023).

Existing evaluations typically compare these paradigms on different benchmarks at a single corpus size, leaving the question of how they scale with corpus growth unanswered. The paper introduces a controlled scaling study that fixes the workload (questions, gold evidence, adversarial distractors) while varying the corpus size, to understand accuracy–cost scaling.

Methodology

• Corpus and Questions: EnterpriseRAG-Bench (Sun et al., 2026) – a fictional company corpus of 511,959 documents (600.8M tokens) from 9 sources, with 500 questions of 10 types, each annotated with gold documents, gold answer, and atomic answer facts. The corpus includes 7.7% realistic noise (wrong source, outdated facts).
• Bedrock and Nested Tiers: The smallest tier (bedrock) consists of the union of 722 gold documents, 326 mined traps, 99 lures, and 2 organizational pages (1,144 documents total). Traps are semantically similar but factually wrong distractors; lures are plausible but unanswerable for info-not-found questions. All adversarial documents reside in the bedrock, keeping the adversarial set fixed as the corpus grows.
• Scaling Ladder: Non-bedrock documents are added in a fixed, source-and-noise-stratified order. Tier sizes follow $n_{t+1} \approx 1.25\, n_t$, yielding 28 tiers. Scale is reported in corpus tokens.
• Paradigms Evaluated: Seven native pipelines: BM25, DenseRAG (chunk embeddings), HippoRAG 2, MS-GraphRAG (local search), LightRAG (hybrid mode), LinearRAG (lightweight NER+embeddings, no generative build), and File-System Agent (iterative tool calls over raw file tree, budget of 80 LLM calls per question).
• Reader and Metering: All paradigms use Qwen3.6-27B (temperature 0) as reader and policy model. Embeddings via Qwen3-Embedding-0.6B. A shared metering layer attributes prompt/completion tokens to build or query phase per paradigm per question. Latency measured end-to-end under single-stream conditions.
• Judging: Official protocol: holistic alignment, atomic fact entailment fraction, combined score (completeness gated by correctness), and document-level recall. All predictions scored in one session with one judge model; robustness checked with an independent judge and a binary protocol.

Empirical Validation / Results

Main Results: Scale-Dependent Crossover

Table 1 shows the official combined score (%) at selected tiers. At bedrock (N=1,144), File-System Agent (77.4) and BM25 (74.7) overlap with 95% CIs. By ~10M tokens, BM25 overtakes; at full scale (511,959 documents), BM25 leads with 50.5 versus 30.7 (File-System Agent) and 29.9 (DenseRAG). Graph-based methods stop at much smaller tiers due to construction walls.

Table 1: Official main results. Build/query tokens, completeness, and document recall are bedrock values. Dashes denote unbuilt tiers. ‡ = non-generative construction. § = LightRAG runs did not log retrieved-evidence identifiers.

Build Cost and Scaling Walls

Table 2 fits power-law build cost $C(x) = a x^b$ against corpus tokens $x$. HippoRAG 2 is near-linear ($b=1.01$) but extrapolates to 2.9B generative tokens at full scale (~3 days). MS-GraphRAG ($b=0.92$) extrapolates to 7.9B tokens (~50 days). LightRAG is super-linear ($b=1.36$) and does not complete beyond 2,254 documents; extrapolation yields ~102B tokens (~4 years). Embedding-only methods (DenseRAG, LinearRAG) are far cheaper.

Table 2: Build-cost fits. “Measured to” reports build tokens at the largest completed corpus; tok/s is measured single-instance throughput.

Query Cost and Latency

BM25, DenseRAG, and HippoRAG 2 have nearly scale-invariant query costs (5.8K–6.5K tokens/question). File-System Agent grows from 226K to 343K tokens/question as corpus grows, with median LLM calls rising from 5 to 8; budget exhaustion reaches 31% at full scale. Accuracy falls even among within-budget questions, indicating that truncation is not the sole cause.

Retrieval-Primitive Control (Agent+BM25)

Table 4 shows that replacing raw-file search with BM25 inside the same agentic harness at full scale raises the combined score from 36.9 to 69.4, while reducing calls from 36.12 to 5.79 and tokens from 895K to 101K per question. Document recall jumps from 36.8% to 72.4%. This confirms that the File-System Agent’s collapse is a candidate-discovery failure, not weak evidence synthesis.

Table 4: Retrieval-primitive control on same 150 questions at each scale. Combined score and full-scale document recall (DocR) from one shared rejudge.

Agency over Graph Substrates

Table 3 (not reproduced here fully) shows that the same agentic harness over graph substrates (HippoRAG 2, LightRAG, MS-GraphRAG) changes native scores by −6.7 to +29.9 points, depending on the substrate. Thus the same index can support different outcomes under an agent versus its native one-shot ranker.

Cross-Protocol Robustness

Binary re-scoring preserves family rankings across all nine shared scales. Independent judge agreement on pooled alignment verdicts is 96.2% (94.4–98.0% per cell). Close pairs can vary, but broad family separation is robust.

Theoretical and Practical Implications

• Lexical advantage: Enterprise questions contain precise lexical anchors; traps are semantically similar but factually wrong, so exact matching (BM25) is favored. Paraphrasing and depth controls (Table 6) confirm BM25 remains above dense and graph retrieval under comparable depth.
• Graph failure modes: Construction adds extraction noise (e.g., malformed entities), prohibitive scale cost, and graph retrieval often favors semantically similar but factually wrong neighborhoods (which traps punish). LinearRAG (no generative build) finishes within 1.8 points of MS-GraphRAG and LightRAG at bedrock, suggesting LLM extraction adds cost faster than signal here.
• Scaling mechanism: The decisive factor is corpus-wide candidate discovery. BM25 and DenseRAG amortize global ranking in an index; File-System Agent explores a local tree sequentially, making relevant branches harder to reach as corpus grows. Agent+BM25 isolates this: after global ranking, iteration helps most.
• Evaluation implications: Single-size evaluations can conceal crossovers. Cross-paradigm comparisons should report nested-scale accuracy, construction/query cost, and index coverage. Coverage-adjusted summaries (assigning zero to unbuilt tiers) describe deployability, not just accuracy.
• Practical guidance: For enterprise-style corpora, BM25 is the appropriate default. Aggregation-heavy questions favor Agent+BM25, which uses lexical ranking for discovery and agentic calls on narrowed candidates. LLM-built graph indexes are difficult to justify at 10^5–10^6 documents unless construction is near-linear and relational questions dominate.

Conclusion

The paper presents a controlled scaling study of four RAG paradigms on a 28-tier enterprise corpus ladder, holding tasks fixed while varying corpus size 450-fold. The central finding is a scale-dependent crossover: raw-file agency leads at small scales, BM25 overtakes around 10M corpus tokens, and its lead widens to nearly 20 points at 601M tokens while remaining Pareto-optimal without LLM-based construction. Replacing raw search with BM25 within the same agentic harness raises full-scale accuracy from 36.9 to 69.4 at one ninth of the query tokens. Graph-based RAG encounters construction walls before reaching full scale. The study implies that global lexical ranking becomes the stronger default as enterprise corpora scale, with agentic reasoning best applied after candidate ranking rather than in place of it. Future work should explore other corpus types (e.g., long-tail, high-recall), multi-hop reasoning, and dynamic budgets.