OCC-RAG: Optimal Cognitive Core for Faithful Question Answering

Summary (Overview)

• OCC-RAG is a family of small language models (0.6B and 1.7B parameters) specifically designed for faithful, context-grounded question answering.
• The models are mid-trained on a novel synthetic corpus of >3M examples that targets multi-hop reasoning, strict context faithfulness, and calibrated abstention.
• OCC-RAG produces structured reasoning traces with source citations (Query Analysis, Source Analysis, Reasoning, Status, Answer) anchored to literal quotes from the provided context.
• Despite being 2–6× smaller, OCC-RAG matches or exceeds general-purpose models up to 4B parameters on multi-hop reasoning (HotpotQA, MuSiQue, TAT-QA) and achieves the best faithfulness and refusal performance across all evaluated scales (ConFiQA, MuSiQue-Un).
• The work demonstrates that faithfulness does not require scale alone; it can be effectively learned through the right training curriculum and supervision format in compact, task-specialized architectures.

Introduction and Theoretical Foundation

Background

Frontier language models grow larger and absorb more world knowledge, but many practical applications benefit more from compact, task-specialized architectures (Small Language Models – SLMs). SLMs have shown competitive performance on commonsense reasoning, mathematical reasoning, tool calling, and retrieval-augmented generation.

Context QA and Faithfulness

The paper focuses on Context Question Answering (Context QA): models answer questions based exclusively on a provided context, generating responses strictly derived from that input. A central requirement is faithfulness:

• Outputs must be aligned with evidence from the context.
• The model must avoid hallucination and ignore parametric knowledge when it conflicts with the context.

Faithfulness thus measures both the alignment of the answer with the evidence and the absence of hallucinated content.

Three Core Capabilities for Context QA

OCC-RAG is built around three capabilities:

1. Multi-hop inference and commonsense reasoning – synthesizing information across disparate parts of the context and bridging logical gaps.
2. Avoidance of memorization – pretraining knowledge must not override or interfere with the provided context.
3. Safe abstention – declining to answer when the context is insufficient, ambiguous, or lacks the necessary information.

Why Mid-training?

Mid-training is a core stage that explicitly shapes the SLM’s reasoning architecture for Context QA. It enables:

• Strong multi-hop reasoning by training on reasoning-trace datasets that internalize the functional structure of multi-hop inference (subquestion decomposition, verification).
• Faithful, non-memorized QA by tying every reasoning step back to provided evidence.
• Calibrated abstention by including “context-insufficient” examples with explicit reasoning-trace patterns.

Methodology

Training Data Generation Pipeline

A synthetic corpus of ~3.25M QA pairs was created, targeting three properties: reasoning over context, strict faithfulness (answer recoverable from context alone), and a fraction of unanswerable examples.

1. Single-hop QA Generation (largest subset: 2.78M pairs)

• Ingest and chunk English Wikipedia XML – paragraphs become candidate chunks.
• QA Generation – for each gold paragraph, gpt-oss-120B generates ten short QA pairs (JSON array, answers must be extractive).
• Distractor mining – up to 1000 child pages from Wikipedia link graph, chunked and scored by TF-IDF cosine similarity; top 20 kept.
• Filtration – LLM-as-judge removes inaccurate or illogical QA pairs.

2. Multi-hop QA Generation (262k single-context, 165k multi-context pairs)

• Requires synthesizing multiple facts. Uses a Knowledge Graph (KG) extracted from context (Wikontic pipeline) to condition generation.
• Path sampling – adopts DRAGOn benchmark taxonomy: simple questions, two-hop families (set, multi-hop, condition), three-hop bamboo-style questions. Each type corresponds to a SPARQL template selecting a sub-graph.
• For each sampled path, gpt-oss-120B receives a type-specific prompt with gold path and supporting paragraphs, generating one QA pair per path.

3. Unanswerable Question Construction (43k abstain pairs)

• Uses a DeBERTa model fine-tuned on SQuAD to answer questions with reduced subsets of gold contexts. If the predicted answer does not match the original, the model should abstain (critical information missing).

4. Structured Reasoning Traces Every QA pair is enriched with an explicit reasoning trace (generated by Qwen3.5-27B in non-thinking mode):

• Query Analysis – what the question asks.
• Source Analysis – which sources are relevant.
• Reasoning – how facts combine.
• Status – explicit ANSWERABLE/UNANSWERABLE verdict.
• Answer – final answer string.

Filtering checks: format, answer match (exact match), LLM-as-judge (Qwen3-4B for borderline cases), and overthinking (traces > 1256 tokens or >10 thinking markers dropped).

Dataset Statistics

Distractor contexts consume the largest share of tokens (35%–75%), followed by gold contexts and reasoning chains.

Mid-training Procedure

• Base model: Qwen3-0.6B-Base and Qwen3-1.7B-Base (selected over Gemma3 and SmolLM3 based on held-out QA slice).
• Objective: Supervised fine-tuning, loss applied only to response tokens. Prompt includes question + context passages (random order, numeric source identifiers). Response is the structured reasoning trace.
• Special tokens delimit prompt elements and response sections; their embeddings are initialized from the mean of subword embeddings of natural-language names.
• Data mixing: Multi-hop examples are oversampled 3× per epoch; single-hop shown once. No curriculum schedule used.
• Training: ~9 × 10⁹ tokens, ~17 hours (0.6B) and ~28 hours (1.7B) on 8× NVIDIA H100 (80 GB).

Empirical Validation / Results

Benchmarks

ConFiQA subsets: QA (single counterfactual triple), MR (multi-hop chain, one counterfactual), MC (multi-hop chain, all counterfactual).
Memorization Ratio (MR): $M_R = \frac{P_o}{P_o + P_c}$ where (P_o) is rate of original (memorized) answer, (P_c) is rate of counterfactual (context-grounded) answer. Lower is better.

Main Results

• Best per column in bold, second-best underline.
• Parentheses for Qwen3/SmolLM3 indicate thinking mode results.
• OCC-RAG-0.6B (0.6B) exceeds Qwen3-1.7B (2.8× larger) by 9.5 points on ConFiQA and reduces MR from 8.2 to 5.2.
• OCC-RAG-1.7B achieves the highest ConFiQA accuracy (81.4) and lowest memorization ratio (5.0) across all models.
• On refusal (MuSiQue-Un), OCC-RAG-1.7B attains 87.2 R-Acc, on par with models 8B+.
• At 2–6× smaller size, OCC-RAG models are competitive with Qwen3 4B–14B on multi-hop reasoning while surpassing them on faithfulness and refusal.

Key Observations

• Faithfulness is not a function of scale alone: OCC-RAG’s training curriculum explicitly teaches context grounding, yielding MR values far below those of much larger models (e.g., Qwen3-32B MR=11.5).
• Structured reasoning traces (with explicit ANSWERABLE/UNANSWERABLE status) enable calibrated abstention behavior.
• Multi-hop reasoning benefits from the oversampling of multi-examples in training data.

Theoretical and Practical Implications

Theoretical Implications

• The results challenge the assumption that large parametric knowledge is necessary for robust QA. Faithfulness and reliable reasoning can be instilled in small models through meticulously designed training data and supervision formats.
• Mid-training on synthetic reasoning traces serves as an effective transfer mechanism for task-specific capabilities (multi-hop inference, evidence grounding, abstention) without requiring massive scale.
• The explicit ANSWERABLE/UNANSWERABLE status as a supervised target turns abstention into a structured behavior, not an ad-hoc heuristic.

Practical Implications

• Deployment efficiency: OCC-RAG’s compact size (0.6B–1.7B) makes it suitable for resource-constrained environments, while delivering performance competitive with or exceeding models 6× larger on key metrics.
• Trustworthiness: The model’s strict adherence to context, low memorization ratio, and learned abstention make it well-suited for high-stakes applications (e.g., legal, medical, financial QA) where hallucination is unacceptable.
• Transparency: Structured reasoning traces with source citations provide chain-of-thought-level interpretability at a fraction of the computational cost of full thinking-mode inference.
• Reusability: The synthetic data generation pipeline (single-hop, multi-hop, unanswerable) and the mid-training recipe provide a reusable blueprint for building compact, faithful QA systems.

Conclusion

OCC-RAG demonstrates that a compact, task-specialized small language model can match or exceed much larger general-purpose models on faithful context-grounded question answering. By combining:

• Large-scale synthetic mid-training covering multi-hop reasoning, context faithfulness, and calibrated abstention,
• Structured reasoning traces with explicit evidence citations,
• and