# The Personalization Mirage: How LLMs Fabricate User Profiles, and Why Self-Monitoring Misleads

> MirageBench shows all 12 tested LLMs over-infer 35-49% of personalized claims, and models self-reporting least over-inference actually fabricate the most.

- **Source:** [arXiv](https://arxiv.org/abs/2608.04570)
- **Published:** 2026-08-07
- **Permalink:** https://picx.dev/p/W8U5Vs

## Summary

## Summary (Overview)

- **Introduction of MirageBench**: The first benchmark specifically designed to measure over-inference (OI) in personalized LLMs, comprising 150 personas, 6 personalization tasks, and a four-way faithfulness taxonomy (Grounded / Reasonable / Stereotype / Fabricated).
- **Pervasive over-inference**: All 12 tested models across 7 families over-infer 35%–49% of their personalized claims (cross-model mean 41.6%; claim-weighted 41.8%), with only 24%–31% of claims groundable in evidence.
- **Discovery of the Self-Monitoring Inversion**: At the model-selection level, models' self-assessed OI is negatively rank-correlated with judge-measured OI (ρ = −0.60, p = 0.044), meaning models that report the least over-inference actually fabricate the most.
- **Task-dependent and accumulative behavior**: OI rates range from 27% (gift recommendation) to 59% (apartment description), and in a multi-turn pilot, inferred attributes accumulate approximately linearly (R² > 0.90 for 9/12 models) with little revision.
- **Key contribution**: MirageBench positions external verification over model self-report as a more reliable foundation for trustworthy personalization.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Personalized LLMs with persistent memory have become mainstream. Systems like ChatGPT (OpenAI 2024b), Mem0 (Chhikara et al., 2025), and MemGPT (Packer et al., 2023) maintain evolving user profiles across sessions. These systems share a critical assumption: **that models can reliably determine what they know about users versus what they are guessing**. This paper demonstrates that this assumption is fundamentally violated in real-world interactions.

### Defining Over-Inference (OI)

Over-inference (OI) is defined as the phenomenon where LLMs generate personalized claims about users that go beyond what available evidence supports. It is distinct from:
- **Factual hallucination**: concerns world knowledge
- **Social bias**: concerns group-level stereotypes

OI sits in a dangerous middle ground: fabricating individual-level attributes that feel personalized but are not grounded in anything the user has actually shared.

### Problem Formulation

The user reveals a set of facts $E = \{ e _ { 1 } , \ldots , e _ { k } \}$ through interaction. When asked to perform a personalized task T, the system produces a response R decomposable into claims $C = \{ c _ { 1 } , \ldots , c _ { n } \}$ about the user. The quality of personalization is a property of the relationship between $C$ and $E$.

The four-way claim taxonomy:
- **Grounded**: restates what the user has said
- **Reasonable**: extends evidence by a single common-sense step
- **Stereotype**: substitutes demographic or occupational priors for individual evidence
- **Fabricated**: has no evidential basis at all

The over-inference rate of model M is defined as:

$$
\operatorname{OI} \text { Rate } (M) = \frac {\# \text { STEREOTYPE } + \# \text { FABRICATED }}{\# \text { TOTAL   CLAIMs }},\tag{1}
$$

The self-monitoring gap:

$$
\operatorname{Gap} (M) = \operatorname{OI} _ {\text { JUDGE }} (M) - \operatorname{OI} _ {\text { self }} (M),\tag{3}
$$

A positive gap indicates the model produces more over-inference than it admits under self-audit.

---

## Methodology

### Personas

Each persona is a pair $(P, E)$ where:
- $P$: a ground-truth profile of 15 attributes (occupation, hobbies, personality, living situation, dietary preferences, etc.)
- $E$: exactly three first-person facts the persona has "revealed" to the assistant

Fixing $|E| = 3$ is deliberate—revealed facts typically cover occupation and 1–2 hobbies, leaving 12 attributes unmentioned. The 150 personas are balanced across:
- **Stereotypical** (50): conform to common occupational/demographic associations (e.g., female nurse who enjoys yoga)
- **Counter-stereotypical** (50): defy expectations (e.g., male kindergarten teacher who competes in powerlifting)
- **Neutral** (50): no salient stereotype alignment

### Personalization Tasks

Six open-ended tasks spanning a "groundability gradient":

| Task | Description |
|------|-------------|
| Dating profile bio | Write a short dating profile |
| Weekend itinerary | Recommend itinerary in unfamiliar city |
| Letter of recommendation | Draft a convincing recommendation |
| Birthday gift | Choose a $100 gift |
| Apartment description | Describe the user's apartment |
| Stress source | Identify what stresses the user most |

### Evaluation Pipeline

The four-instrument pipeline:
1. **Probe**: elicits explicit inference—model asked "Given these three facts, list everything you can infer"
2. **Task**: elicits implicit inference—model performs six tasks, then self-audits its own claims
3. **Judge**: external adjudication by Claude-Opus-4-7 (not part of leaderboard), classifying every claim using the same taxonomy at temperature 0.0
4. **Accum**: longitudinal—8 rounds of continued interaction, tracking memory growth and revision

### Judge Validation

An independent annotator labeled a stratified sample of 400 claims (100 per predicted class), blind to judge's label:
- **Four-way agreement**: 89.8% (Cohen's κ = 0.863, macro-F1 = 0.896)
- **Binary agreement**: 95.0% (κ = 0.900)
- 71% of residual disagreements were between semantically adjacent classes

### Models Evaluated

12 models across 7 families: GPT-5.5, GPT-5.4-nano, GPT-4o-mini, Claude-Opus-4-6, Gemini-3.1-pro, Gemini-3-flash, DeepSeek-v4-pro, DeepSeek-v4-flash, Qwen3.6-plus, Qwen3-8B, GLM-5.1, Kimi-K2.5.

---

## Empirical Validation / Results

### 1. Over-Inference is Universal and Severe

**Table 1: MirageBench Leaderboard** (sorted by OI rate descending)

| Model | Claims | Grnd (%) | Stereo (%) | Fabric (%) | OI% |
|-------|--------|----------|------------|------------|-----|
| Qwen3-8B | 12,687 | 23.6 | 9.3 | 39.4 | 48.7 |
| DeepSeek-v4-pro | 13,170 | 24.4 | 10.7 | 34.7 | 45.4 |
| GPT-4o-mini | 10,408 | 26.6 | 7.1 | 38.0 | 45.1 |
| DeepSeek-v4-flash | 11,972 | 25.5 | 10.3 | 34.3 | 44.6 |
| Qwen3.6-plus | 13,146 | 23.7 | 11.8 | 32.7 | 44.5 |
| Kimi-K2.5 | 12,665 | 24.9 | 12.5 | 30.6 | 43.1 |
| Gemini-3-flash | 14,396 | 24.9 | 12.8 | 28.3 | 41.1 |
| GLM-5.1 | 12,869 | 26.0 | 12.2 | 28.0 | 40.2 |
| GPT-5.5 | 10,416 | 26.6 | 9.4 | 29.0 | 38.5 |
| GPT-5.4-nano | 7,790 | 30.9 | 8.6 | 29.0 | 37.6 |
| Claude-Opus-4-6 | 11,139 | 25.9 | 10.7 | 24.7 | 35.4 |
| Gemini-3.1-pro | 12,958 | 27.9 | 11.3 | 23.8 | 35.1 |
| **Mean** | | **25.9** | **10.5** | **31.1** | **41.6** |

Key patterns:
- Only 24%–31% of personalized content is grounded in evidence
- Fabrication (mean 31.1%) dominates over stereotyping (mean 10.5%)
- Stereotypical personas incur 44.8% OI vs. 37.0% for counter-stereotypical (7.8 pp gap present in all 12 models)

### 2. The Self-Monitoring Inversion

**Table 2: Self-Monitoring Inversion at the Model-Selection Level**

| Model | Self% | Judge% | Δ | Pattern |
|-------|-------|--------|---|---------|
| Qwen3-8B | 13.0 | 48.7 | +35.7 | Under |
| GPT-4o-mini | 20.1 | 45.1 | +25.0 | Under |
| DeepSeek-v4-flash | 33.0 | 44.6 | +11.6 | Under |
| DeepSeek-v4-pro | 40.8 | 45.4 | +4.6 | Calib. |
| Gemini-3-flash | 41.1 | 41.1 | 0.0 | Calib. |
| Qwen3.6-plus | 45.4 | 44.5 | -0.9 | Calib. |
| Claude-Opus-4-6 | 41.6 | 35.4 | -6.2 | Over |
| GPT-5.5 | 46.4 | 38.5 | -8.0 | Over |
| Gemini-3.1-pro | 43.2 | 35.1 | -8.1 | Over |
| GLM-5.1 | 49.6 | 40.2 | -9.3 | Over |
| GPT-5.4-nano | 49.0 | 37.6 | -11.4 | Over |
| Kimi-K2.5 | 58.2 | 43.1 | -15.1 | Over |

**Spearman ρ = −0.60 (p = 0.044 by permutation; 95% bootstrap CI [−0.90, +0.06], family-clustered [−0.87, +0.14]; n = 12)**

Qwen3-8B self-reports the lowest OI (13.0%) but receives the highest Judge OI (48.7%); Kimi-K2.5 self-reports the highest OI (58.2%) but sits mid-pack at 43.1%.

**Within-model signal**: Per-model AUROC of self-audit OI% for identifying above-median-Judge-OI records ranges from 0.58 (Qwen3-8B) to 0.83 (Qwen3.6-plus), with 9 of 12 models above 0.75.

**Under Probe**: incorrect rates of only 0.7%–4.6% (mean gap of 38.6 pp against Task-generation OI).

### 3. Task-Dependent Over-Inference

**Table 3: Claim Composition by Task (%)**

| Task | Grnd | Reas | Ster | Fab | OI% |
|------|------|------|------|-----|-----|
| Apartment/home | 15.0 | 27.2 | 19.8 | 38.0 | 57.8 |
| Rec. letter | 12.0 | 39.8 | 7.8 | 40.4 | 48.2 |
| Stress source | 22.0 | 38.2 | 9.1 | 30.7 | 39.8 |
| Weekend itinerary | 26.0 | 35.3 | 10.2 | 28.5 | 38.7 |
| Dating profile | 34.0 | 37.5 | 4.5 | 24.0 | 28.5 |
| Birthday gift | 40.0 | 32.9 | 8.8 | 18.3 | 27.0 |

The recommendation letter shows high fabrication (40.4%) from genre conventions rather than stereotypes. Gift recommendations succeed (27.0% OI) because they can be grounded in stated interests.

### 4. Inference Accumulates in Multi-Turn Pilot

**Table 4: Inference Accumulation over 8 Conversation Rounds**

| Model | R1 | R8 | Growth | /round |
|-------|----|----|--------|--------|
| GPT-5.5 | 18.5 | 125.0 | +106.5 | 15.2 |
| GLM-5.1 | 16.5 | 121.5 | +105.0 | 15.0 |
| Claude-Opus-4-6 | 15.5 | 104.5 | +89.0 | 12.7 |
| Qwen3.6-plus | 19.0 | 102.5 | +83.5 | 11.9 |
| DeepSeek-v4-flash | 13.0 | 82.0 | +69.0 | 9.9 |
| Kimi-K2.5 | 15.0 | 75.0 | +60.0 | 8.6 |
| Gemini-3-flash | 14.5 | 60.0 | +45.5 | 6.5 |
| DeepSeek-v4-pro | 8.0 | 53.5 | +45.5 | 6.5 |
| Gemini-3.1-pro | 14.0 | 50.5 | +36.5 | 5.2 |
| GPT-4o-mini | 8.5 | 27.0 | +18.5 | 2.6 |
| Qwen3-8B | 13.5 | 23.5 | +10.0 | 1.4 |
| GPT-5.4-nano | 14.5 | 15.0 | +0.5 | 0.1 |

For 9/12 models, growth is approximately linear (R² > 0.90) with slopes of 5–15 attributes/round. Removal rates: 0.4–5% for five fastest accumulators, 70–82% for Qwen3-8B and GPT-5.4-nano (memory replacement rather than accumulation).

---

## Theoretical and Practical Implications

### Why Models Over-Infer

1. **The verbosity trap**: Longer responses mechanically generate more claims from the same fixed 3-fact base (r = 0.59 correlation with OI rate). However, even the most concise model (GPT-4o-mini) reaches 45.1% OI.

2. **Pretraining priors as gap-fillers**: When evidence is sparse, models fill gaps with distributional knowledge (a software engineer "probably" has a minimalist apartment).

3. **Genre expectations demand fabrication**: Recommendation letters (40.4% fabrication vs. 7.8% stereotype) reflect RLHF-amplified compliance with genre conventions.

### The Self-Monitoring Inversion Mechanism

The authors propose a **metacognitive calibration asymmetry** across model families:
- **Strict self-auditors** (Claude, GLM, Kimi): more readily label own inferences as problematic → higher self-reported OI, but caution suppresses fabrication → lower judge-measured OI
- **Lenient self-auditors** (GPT-4o-mini, Qwen3-8B): classify larger fraction of own inferences as "reasonable" → low self-reported OI alongside higher judge-measured OI

### Design Implications

1. **Self-filtering is useful within a model, misleading across models**: Raw self-reported OI must not be used to choose which model is "safer."

2. **Provenance tracking is necessary**: Stored user information should be tagged with epistemic status (stated vs. inferred with evidence link vs. generated without evidence).

3. **The personalization-faithfulness tradeoff is fundamental**: With only 24%–31% of claims grounded, eliminating over-inference would mean eliminating personalization. The design challenge is managing uncertainty, not preventing inference.

### Connection to LLM Honesty

Three layers of behavior:
- **Probe** (what models claim to know): cautious, 1%–5% error
- **Accum** (what they store): accumulative, rarely revised
- **Task/Judge** (what they express): substantially ungrounded, 42% OI

This gap constitutes a form of **behavioral dishonesty**: models know (when asked) that certain inferences are unwarranted, yet freely produce them when not explicitly monitored—paralleling the explicit/implicit bias gap in LLM bias literature.

---

## Conclusion

This paper introduced **MirageBench**, the first benchmark for over-inference in personalized LLMs, evaluating 12 models across 7 families on 143,616 judged claims. Key findings:

1. **Over-inference is broadly shared** (35%–49%, mean 41.6%) and intensifies along a groundability gradient (27%–59%)
2. **Self-audit is inversely correlated with judge-measured OI across models** (ρ = −0.60) yet remains a useful within-model ranking signal (AUROC 0.58–0.83)
3. **In a multi-turn pilot**, inference accumulates for the most capable models with near-zero revision

The paper argues for **external verification and provenance tracking** rather than self-report as the basis for trustworthy personalization. The Self-Monitoring Inversion directly implies that memory systems picking models based on self-reported confidence are "picking in the wrong direction."

**Limitations acknowledged**: single judge (validated against one human annotator), evidence-only scoring, exploratory Self-Monitoring Inversion (n = 12, wide CI containing zero), Accum as a pilot (2 personas), and no evaluated mitigations.

**Future work**: cross-judge robustness checks, additional human annotators, larger longitudinal samples with prune-permitting prompts, and evaluation of specific mitigation strategies. MirageBench will be released in full to support subsequent research.

---

_Markdown view of https://picx.dev/p/W8U5Vs, served by PicX — AI-generated visual whiteboard summaries of research papers._