Summary (Overview)

• Proposes SetwiseEvalKit, the first multi-dimensional evaluation benchmark for document set quality in the LLM era, covering three levels (doc, set, global) and nine dimensions (Relevance, Authenticity, Quality, Complementarity, Redundancy, Conflict, Completeness, Density, Reachability) with ~28K query-specific rubrics.
• Systematic evaluation of 12 rerankers reveals that even the best method achieves ≤45% overall rubric coverage; cross-document coordination dimensions (Complementarity, Completeness) are universally weak, and no single method maintains top performance across both short-form and long-form scenarios.
• Introduces Rubric4Setwise, a training-free method that converts rubric-based evaluation criteria into document set selection signals, achieving state-of-the-art downstream generation with fewer documents and search rounds.
• Demonstrates strong correlation (Pearson r=0.92) between rubric coverage scores and downstream generation quality, validating the benchmark’s credibility.
• Provides public project page, GitHub repository, and HuggingFace dataset for reproducibility.

Introduction and Theoretical Foundation

Background and Motivation

The role of information retrieval is shifting from human browsing to serving as input for LLMs in RAG systems. In traditional retrieval, humans can filter redundancy, resolve conflicts, and identify gaps. In RAG, however, retrieved document sets are directly injected into LLM context windows, so document set quality directly determines the upper bound of generation quality. The evaluation standard must evolve from “useful to humans” to “effective for LLM generation.”

Structural Deficiencies of Traditional Metrics

Under limited context windows, relevance-only document selection exhibits critical defects:

• Redundancy: Highly overlapping documents waste context capacity.
• Conflict: Factual contradictions mislead generation.
• Missing key evidence: Forces additional search rounds.

Even with nDCG@5=100% (all documents relevant), a set may still suffer from these defects (Figure 1). Traditional evaluation (TREC-DL, BEIR, MTEB, RAGAS, ARES, BRIGHT, RA-nWG) has two structural blind spots:

1. Singular evaluation granularity: Scores documents independently and aggregates via nDCG, ignoring inter-document interactions.
2. Singular evaluation dimension: Only covers relevance, unable to diagnose why one set is better (e.g., complementarity, redundancy, reasoning chain coverage).

Theoretical Basis: Rubric-Oriented Evaluation

The paper transfers rubric-based evaluation from educational assessment and LLM-as-Judge (MT-Bench, RubricEval, ResearchQA) to retrieval set quality. Rubrics provide transparent, reproducible, multi-dimensional quality judgment. The key insight is that if rubrics can quantify set quality, they can also serve as selection signals, closing the loop from evaluation to optimization.

Methodology

Task Formulation

Given a query $q$ and reference answer $a$, a set of query-specific evaluation rubrics $R = \{r_1, r_2, \dots, r_K\}$ is constructed, covering nine dimensions. For each reranker, a document subset $S_m \subseteq C$ (candidate pool) is selected with $|S_m| \le k$. Instead of traditional independent scoring, the set is evaluated as a whole:

$$\text{Score}(S_m) = E(q, S_m, R) \in \mathbb{R}^9$$

where $E$ is an LLM-based evaluator scoring each dimension.

SetwiseEvalKit: Multi-Dimensional Rubric Benchmark

Data Collection:

• Short-form scenario: Multi-hop QA from HotpotQA, 2WikiMultihopQA, MuSiQue, Bamboogle. BM25 retrieves top-20 documents; after filtering, 2,061 high-quality samples.
• Long-form scenario: 200 instances from ResearchQA with multi-turn search trajectories using DR.Tulu-8B agent. Rerankers are embedded via MCP protocol after each retrieval turn.

Multi-dimensional Rubric Design (three levels, nine dimensions):

Hybrid Rubrics Generation: Two frontier LLMs (GPT 5.1, Gemini 3.1-Pro-Preview) independently generate candidate rubrics, aggregated by DeepSeek-V4 Pro. Rubrics explicitly reference entities, facts, or numerical values. Yields ~24K rubrics for short-form, ~4K for long-form. Human evaluation confirms ~70% are highly/critically discriminative, with ~77% causing moderate/significant ranking changes when removed.

Evaluation Protocol: Rubrics scored on 0–4 scale per dimension. Coverage percentage per dimension:

$$\text{Cov}_d = \frac{1}{|R_d|} \sum_{r \in R_d} \frac{\text{score}(r)}{4} \times 100\%$$

Relevance gating: if all documents score 0 on relevance, other dimensions are skipped.

Rubric4Setwise: Rubric-Oriented Set Selection

Training-free method that converts evaluation criteria into selection guidance. Given query $q$, rubrics $R$, and candidate pool $C$, selects optimal subset:

$$S^* = \arg\max_{S \subseteq C} f(S; q, R)$$

where $f(\cdot)$ is a rubric-based set utility function, implicitly modeled via chain-of-thought prompting using Qwen3-8B. The subset size $|S^*|$ is adaptively determined by rubric satisfaction, not fixed.

Empirical Validation / Results

Benchmark Credibility

Figure 4 shows strong positive correlation between overall rubric coverage scores and downstream generation performance (Pearson r = 0.92, p = 0.0013), confirming that higher evaluation scores translate to better generated answers.

Results on SetwiseEvalKit

Table 2: Short-form Scenario (top-5 documents for conventional methods, adaptive for setwise)

• Setwise reranking methods (SetR, Rank4Gen) dominate, especially at Doc-Level and Set-Level.
• Reasoning-enhanced methods narrow the gap at Global-Level.
• Conflict scores are uniformly high across methods (not a primary bottleneck in short-form).

Table 3: Long-form Scenario (union scoring after cross-turn deduplication)

• Reasoning-enhanced methods (ReasonRank, Rearank) become leading paradigm.
• Performance gaps shrink (~3 percentage points vs. ~9 in short-form), suggesting isolation between ranker and agent.
• Scores degrade across successive search turns (Figure 5).

Results on Rubric-Oriented Set Selection (Rubric4Setwise)

Table 4: Downstream Generation Performance

• In short-form: Rubric4Setwise achieves best EM (26.10) and F1 (29.32) using only 2.66 documents on average, outperforming SetR by 0.97 EM and 0.62 F1.
• In long-form: Rubric4Setwise achieves highest LLM-judge score (70.57) with fewer documents (20.52 vs. 29.23) and fewer search turns (4.52 vs. 4.73) than SetR.
• It is the only method that maintains state-of-the-art results across both scenarios.

Case Study (Figure 6)

A query about Miss USA crowning: only Doc [11] directly describes the event; others are retrieved via keyword overlap. Rubric scores: Redundancy (2/4), Conflict (2/4), Reachability (1/4) reveal deficiencies invisible to nDCG. This demonstrates how rubric design pinpoints weaknesses and guides optimization.

Theoretical and Practical Implications

Theoretical Implications

• New evaluation paradigm: Moves from document-level relevance to set-level quality, capturing inter-document interactions (redundancy, conflict, complementarity) and holistic utility (completeness, density, reachability). This aligns evaluation with the actual use case of RAG systems.
• Diagnostic power: Multi-dimensional rubrics enable precise diagnosis of reranker weaknesses (e.g., cross-document coordination is a universal bottleneck), guiding targeted improvements.
• Generalization gap: The reversal of optimal strategies between short-form (setwise methods) and long-form (reasoning-enhanced methods) highlights the need for adaptive, versatile rerankers.

Practical Implications

• Rubric4Setwise as training-free optimization: Demonstrates that evaluation signals can directly serve as selection signals, achieving better generation with fewer resources (documents, search rounds). This is immediately applicable in production RAG systems.
• Efficiency: Using ~2.66 documents in short-form vs. fixed top-5 saves context window capacity and reduces latency, while maintaining or improving quality.
• Multi-turn agent scenarios: Rubric guidance enables more efficient multi-turn information acquisition, reducing both document count and search rounds while improving answer quality.
• Upper bound establishment: The oracle setting (rubrics generated with reference answers) provides an empirical upper bound, motivating future work on distilling rubric preferences into trainable rewards.

Conclusion

The paper presents a complete evaluate-diagnose-optimize framework for document set quality in the LLM era:

• Evaluate: SetwiseEvalKit – first set-level evaluation benchmark with three levels, nine dimensions, and ~28K query-specific rubrics. Strong correlation (r=0.92) with downstream generation confirms its credibility.
• Diagnose: Systematic evaluation of 12 rerankers reveals universal weakness on cross-document coordination dimensions, and no single method maintains top performance across short-form and long-form scenarios.
• Optimize: Rubric4Setwise – training-free method that converts rubric signals into selection guidance, achieving state-of-the-art downstream generation with fewer documents and search rounds.

Limitations and Future Work

• Rubric4Setwise operates under an oracle setting (rubrics generated with reference answers), establishing an empirical upper bound.
• A natural next step is distilling rubric preferences into trainable rewards, enabling rerankers to internalize multi-dimensional quality awareness without reference answers.
• Future work should also explore co-optimizing ranker and search agent in multi-turn scenarios to overcome the current isolation bottleneck.

The work provides an evaluation foundation and optimization pathway for advancing retrieval systems from retrieving relevant documents to assembling optimal document sets.