# Does Accuracy Equal Evidence? Reasoning Faithfulness under KV Cache Compression

> Accuracy alone masks severe reasoning degradation in KV-compressed large reasoning models, so evaluation must pair accuracy with faithfulness metrics like reasoning correctness and fidelity.

- **Source:** [arXiv](https://arxiv.org/abs/2608.01631)
- **Published:** 2026-08-18
- **Permalink:** https://picx.dev/p/tvYIqU
- **Whiteboard:** https://picx.dev/p/tvYIqU/image

## Summary

## Summary

### Research Question
This paper investigates whether **accuracy is a sufficient metric** for evaluating KV cache compression methods in large reasoning models (LRMs), questioning the assumption that final-answer accuracy alone validates compression quality.

### Key Findings

The authors identify an **"answer-evidence gap"**: compressed models can maintain high accuracy while losing the reasoning evidence that justifies their answers. The paper reveals that **accuracy is an asymmetric diagnostic**—while accuracy collapse is informative, preserved accuracy can conceal evidential degradation.

### Experimental Setup
- **Models**: Qwen3-8B (primary), plus DeepSeek-R1-Distill-Llama-8B
- **Tasks**: AIME26 (math reasoning), GPQA (question answering), MedCalc (medical calculations), RULER (retrieval)
- **Methods tested**: 10 KV compression methods including SnapKV, HeadKV, AdaKV, PyramidKV, StreamingLLM, and quantization baseline (KIVI-2bit)
- **Budget**: 256-token cache compression

### Key Metrics
1. **Accuracy (Acc.)**: Final answer correctness
2. **Reasoning Correctness (RWAC/crct.)**: Whether the reasoning chain supports the answer
3. **Fidelity (Fid.)**: Robustness to injected misleading evidence

### Key Findings

#### 1. **The Answer-Evidence Gap**
- Compression preserves answer accuracy but degrades reasoning quality and robustness
- **AIME baseline**: Full-KV achieves 56.7% accuracy with 11.8% RWAC; compressed methods (AdaKV, HeadKV, SnapKV) show 23-27% accuracy with dramatically reduced Fidelity (30-35% vs 91.1% full-KV)

#### 2. **Accuracy is Asymmetric** (Question 1)
- Accuracy collapse reveals compression damage, but preserved accuracy can hide invalid reasoning
- Failure modes are compression-method-specific:
  - **SnapKV**: mostly fails on chain inconsistencies
  - **StreamingLLM**: fails through low fidelity and poor accuracy simultaneously
  - **MedCalc**: formula-triggering info survives, but verification anchors don't

#### 3. **Rank Correlation Failure** (Question 2)
- Accuracy rankings correlate poorly with reasoning-quality rankings
- Example on GPQA: LagKV achieves even better accuracy than full-KV but performs catastrophically on reasoning consistency (RWAC/crct ≈ 55% vs 10. propagation)
- At 256-token budget, ranking by accuracy versus chain consistency reverses the top performers

#### 4. **Robustness to Perturbations**
- Fixed-trace perturbation testing shows compression affects resistance to misleading information
- Different methods fail at different perturbation positions:
  - StreamingLLM over-retains early tokens (high vulnerable to early intervention)
  - SnapKV ignores late content, making it vulnerable to end perturbations
- Attention-based methods bias toward conclusion-like patterns, sometimes hallucinating new "sinks" after perturbation

#### 5. **Task-Specific Divergence**
- Retrieval-heavy tasks (RULER): accuracy collapse is visible → accuracy is informative
- Reasoning problems: partial evidence suffices for correct answers but not for verification or robustness → accuracy hides the gap

#### 6. **Key Insight for Compression Methods**

Figure 3 shows that compression strategies differ dramatically in their retention patterns—neither recency-only (TOVA) nor attention-based (SnapKV) methods account for answer–evidence entanglement, leading to:
- Loss of proximal verification steps
- Distortion of reasoning chains
- Vulnerability to misleading perturbations

### Main Recommendations
1. **Evaluation**: Always pair accuracy with faithfulness metrics (RWAC/correct, fidelity)
2. **Design**: Future compressors must preserve dependency structure and provenance, not just salient tokens
3. **No method currently achieves** both high accuracy and high faithfulness simultaneously

### Significance
This work challenges the adequacy of accuracy-only evaluation in KV compression research, highlighting that for reasoning tasks, evidence preservation is as important as answer accuracy.

---

_Markdown view of https://picx.dev/p/tvYIqU, served by PicX — AI-generated visual whiteboard summaries of research papers._