RaguTeam at SemEval-2026 Task 8: Meno and Friends in a Judge-Orchestrated LLM Ensemble for Faithful Multi-Turn Response Generation

Summary (Overview)

• Winning Judge-Orchestrated Ensemble: The authors' system achieved 1st place in SemEval-2026 Task 8 (MTRAGEval) by using a heterogeneous ensemble of seven LLMs with two prompting variants. A GPT-4o-mini judge selects the best candidate per instance based on faithfulness, achieving a conditioned harmonic mean HM$_3$ = 0.7827, outperforming the strongest baseline (0.6390).
• Diversity Over Scale: The ensemble's success is attributed to complementary failure modes across diverse model families, scales, and prompting strategies. The ensemble consistently outperformed any single model, including the 357B parameter GLM-4.6 (HM$_3$ = 0.748).
• Domain-Adapted Lightweight Model: The authors introduce Meno-Lite-0.1, a cost-effective 7B model adapted for RAG tasks. While its direct contribution to the ensemble was minor, it demonstrated a strong cost-performance trade-off, matching 70B models on Russian benchmarks and achieving HM$_3$ = 0.681 on answerable instances.
• Benchmark Analysis: The paper identifies critical limitations in the MTRAGEval benchmark, most notably a target leakage shortcut where all unanswerable questions have empty reference passages, allowing trivial abstention. It recommends adding distractor passages for future iterations.
• Prompting Strategy: Category-aware few-shot prompting consistently outperformed an iteratively refined system prompt, especially for handling challenging underspecified questions, demonstrating the value of concrete behavioral examples over abstract instructions.

Introduction and Theoretical Foundation

The paper addresses the challenge of faithful multi-turn Retrieval-Augmented Generation (RAG), where a system must answer a user's query by grounding its response strictly in provided reference passages, while also incorporating the context of a multi-turn dialogue history. This requires handling coreference resolution, intent tracking, and abstention for unanswerable or underspecified queries.

SemEval-2026 Task 8 (MTRAGEval) formalizes this challenge. In Task B (generation with reference passages), the system receives a dialogue history and reference passages and must generate a final-turn response. The authors' core hypothesis is that a heterogeneous ensemble of LLMs, combined with an LLM-based judge for per-instance selection, can exploit complementary strengths to outperform any single model.

The theoretical foundation combines two lines of work:

1. LLM Ensembles: Exploiting complementary strengths across different model families, scales, and prompting strategies (Jiang et al., 2023; Wang et al., 2024).
2. LLM-as-a-Judge: Using LLMs for scalable, reference-free evaluation of qualities like faithfulness and appropriateness (Zheng et al., 2023; Liu et al., 2023).

Methodology

The system follows a three-stage pipeline:

1. Prompt Construction: Two distinct strategies are used.
   • Group 1 (System Prompt Only): Models use an iteratively refined system prompt ($P$) designed via a Gemini-based procedure (see Appendix B). The prompt emphasizes strict grounding, explicit abstention, concise formatting, and dialogue history use.
   • Group 2 (Few-Shot): Models use a simpler base prompt augmented with category-aware few-shot examples. Training instances are clustered into three categories (full context, empty context, empty history), and medoid exemplars are selected via embedding similarity (using gte-multilingual-base).
2. Candidate Generation: A heterogeneous ensemble of seven LLMs generates candidates in parallel. The ensemble is intentionally diverse in providers, training pipelines, and scales.

3. Judge-Based Selection: For each instance, GPT-4o-mini evaluates all seven candidate responses for faithfulness (whether all claims are supported by provided passages) and assigns a score from 0 to 1. The top-ranked candidate is selected. A deterministic post-processing step replaces any ensemble output with "I don't have an answer" for instances where the reference context is empty.

Meno-Lite-0.1 Details (Appendix A): This 7B model is derived from Qwen2.5-7B-Instruct via a two-stage pipeline:

1. Continued Pretraining (1.3B tokens): On a balanced bilingual mix of educational data (English FineWeb-Edu, Russian RuLM subset, Russian educational PDFs).
2. Supervised Fine-Tuning (50M tokens): On a bilingual instruction set focused on extraction, normalization, summarization, and multi-hop QA skills, including data from MTRAGEval itself. Its design philosophy prioritizes language skills (comprehension, extraction) over world knowledge (facts, dates), aiming for competitive performance on context-grounded tasks with minimal parameters.

Empirical Validation / Results

Dataset: MTRAGEval Task B evaluation set (507 instances across four domains: FiQA, IBMCloud, CLAPnq, Govt). Answerability distribution: 56.2% answerable, 19.1% unanswerable, 15.4% underspecified, 9.3% partially answerable. Metrics: Official score is the conditioned harmonic mean of three metrics: RB_alg (harmonic mean of BERTScore Recall, BERT-K-Precision, ROUGE-L), RB_llm (LLM-judge score), and RL_F (reference-less faithfulness). HM$_3$ denotes this final harmonic mean.

Main Results

The ensemble achieved 1st place out of 26 teams.

Individual Model and Ensemble Performance

The ensemble outperformed all individual models. The best single model was GLM-4.6.

Ablation Studies

Key findings from ablation experiments (unconditioned HM$_3$ scores):

• Diversity and Selection: The ensemble beats any single model overall. Judge selection provides a clear gain over random selection, especially on faithfulness (RL_F).
• Meno-Lite-0.1 Contribution: Its direct contribution to the ensemble was minimal (selected in only 2 of 424 instances), but its selected responses were high-quality (HM ≈ 0.707). Its value lies in cost-effective standalone performance.
• Few-Shot Superiority: Few-shot prompting consistently outperformed the refined system prompt, with substantial gains on underspecified questions.

Meno-Lite-0.1 Benchmark Performance

The domain-adapted 7B model showed strong performance on related benchmarks:

• MERA (Russian): Overall score 0.555, matching Llama-3.3-70B-Instruct (0.555) and surpassing base Qwen2.5-7B-Instruct (0.482).
• NEREL-bench (Knowledge Graph): Highest harmonic mean (0.468), outperforming Qwen2.5-32B-Instruct (0.416).
• LIBRA (Long-Context): Maintained near-perfect passkey retrieval and led 7B-class models on real-world multi-hop QA.

Theoretical and Practical Implications

Theoretical Implications:

1. Complementarity > Scale: For grounded generation tasks, diversity in model architectures and failure modes can outweigh raw parameter count. A carefully orchestrated ensemble of smaller models can surpass a single, much larger model.
2. Example-Based Learning: For complex, edge-case behaviors like handling underspecified queries, providing concrete few-shot examples is more effective than refining abstract, instructional system prompts.
3. Faithfulness vs. Appropriateness: The judge optimized for faithfulness, but this sometimes came at the cost of pragmatic appropriateness (e.g., failing to ask for clarification). This highlights the need for multi-objective selection criteria in real-world applications.

Practical Implications:

1. Cost-Effective RAG Models: The Meno-Lite-0.1 case study demonstrates that targeted domain adaptation and skill training can produce small models (7B) that are highly competent at context-grounded tasks, offering a viable path for resource:constrained deployments.
2. Ensemble Design: Practitioners building similar systems should prioritize architectural and behavioral diversity in their candidate pool. A simple, lightweight LLM judge can effectively harness this diversity for per-instance improvement.
3. Benchmark Design: The critical analysis reveals pitfalls in current RAG evaluation. Future benchmarks must avoid shortcuts (like empty contexts for unanswerable questions) and should align the guidelines used by both response generators and evaluators to ensure fair and meaningful assessment.

Conclusion

The paper presents a winning solution for multi-turn faithful RAG based on a judge-orchestrated, heterogeneous LLM ensemble. The core takeaways are:

• The ensemble's success stems from diversity in models and prompting, combined with informed per-instance selection.
• Meno-Lite-0.1 proves that small, domain-adapted models can achieve strong cost-performance trade-offs for RAG.
• Current evaluation benchmarks have critical limitations (e.g., the empty-context shortcut) that must be addressed to drive genuine progress.

Future work includes exploring adaptive routing to invoke specialized model subsets per instance, multi-objective judge selection, scaling domain adaptation, and improving calibration for underspecified queries. The authors also advocate for a fully open-weight pipeline to reduce dependency on proprietary APIs.