Summary (Overview)

• Problem: Existing role-playing agent (RPA) benchmarks use fixed dialogue histories and generic rubrics, which conflate the RPA’s own capability with the quality of the borrowed history and fail to capture individual user satisfaction.
• Key Contribution: PALATE (Person-Aligned LLM-Simulated-User Assessment with Tailored Evaluation) introduces free multi-turn evaluation where a per-user simulator co-constructs the trajectory with the candidate RPA, and a personalized rubric (induced from the same user’s annotated history) scores the interaction.
• Methodology: Five user simulators are trained via per-user LoRA fine-tuning on real conversational histories; a meta-prompt constructs a frozen personalized rubric from each user’s training sessions. Candidates are scored along three tracks: personalized experience, generic turn quality, and whole-session quality.
• Main Results: On 16 candidates, PALATE reveals user-dependent winners and cross-track capability mismatches. Personalized scoring achieves higher agreement with human judgments (macro 0.613) than generic scoring (0.507) or a MiniMax-aligned baseline (0.467).
• Impact: PALATE changes the evaluation unit from an isolated RPA to a specific user–RPA pair, producing interpretable performance profiles rather than a single ranking.

Introduction and Theoretical Foundation

Role-playing agents (RPAs) provide interactive storytelling, companionship, and emotionally engaging conversation. Their value unfolds over multiple turns, and reliable evaluation is essential for measuring capability, comparing systems, and guiding improvement. Unlike task assistants, RPA dialogues are jointly constructed by the agent and user, with no single correct answer. Existing benchmarks (e.g., Wang et al., 2024c; Tu et al., 2024; Zhou et al., 2025) adopt fixed-history evaluation: the RPA continues a pre-written dialogue history, and an LLM judge scores the continuation on criteria like persona consistency and narrative coherence.

The authors identify two fundamental limitations:

1. Borrowed history bias: The RPA’s output is conditioned on the external history. A controlled experiment shows that high-quality character-side histories inflate scores by ~0.21 (on a 5-point scale) while degraded histories lower them by ~0.13. Thus fixed-history measures $Q(c|H)$ (quality conditioned on history) rather than the RPA’s independent role-playing ability.
2. User-independent scoring: Quality is treated as independent of the user. A detailed, slow-burn relationship arc may satisfy one user but frustrate another seeking rapid conflict. A static rubric cannot capture individual differences in experience.

To address these, PALATE redesigns both the evaluation trajectory and the scoring perspective. It trains a dedicated user simulator from each person’s real dialogue history, lets that simulator interact freely with the candidate RPA from the character’s opening, and automatically induces a personalized experience rubric from the same user’s annotated history. The evaluation unit becomes a specific user–RPA pair.

Methodology

3.1 User-Simulator Training and Validation

Data collection: 300 parallel Chinese–English character cards are constructed. Five users provide 5,133 user turns with turn-level satisfaction labels (~1,000 turns per user). Ten character cards are pre-frozen as the main evaluation panel. Sessions are split by complete session into training and evaluation sets; evaluation set labels are never used for rubric construction or scoring.

Training: Each session is converted into a user-side next-action task with two target types: free-text continuation and exit via [QUIT]. Character messages become model input, human user messages become output, and a [QUIT] target is appended to naturally completed sessions. Per-user LoRA fine-tuning (Hu et al., 2022) is applied to Qwen3.5-35B-A3B Instruct.

Validation: Two metrics are used:

• 2AFC fooling rate: A judge chooses between the real human message and the simulator-generated message given the same context. Closer to 0.5 indicates better indistinguishability.
• Identity consistency: A judge assesses how well the simulator’s response matches the target user’s personal expression without seeing the human answer.

Table 1 shows that per-user LoRA achieves a macro fooling rate of 0.561 (vs. 0.002 for raw instruct, 0.180 for prompted LLM, 0.242 for LLM+profile) and identity consistency of 77.6 (vs. 15.0, 55.9, 61.7).

3.2 Free Multi-Turn Role-Playing

Each user simulator interacts freely with each candidate RPA on the ten pre-frozen character cards. Each cell is independently repeated twice, yielding $5 \times 10 \times C \times 2$ trajectories ($C=16$). The simulator may terminate via [QUIT]; otherwise, the session stops at a preset depth limit. No external orchestrator controls turn order or plot development—only alternating generations by the simulator and candidate.

3.3 Personalized Rubric Construction and Three-Track Evaluation

Personalized rubric: For each user, an automated meta-prompt receives that person’s training sessions and experience annotations. It contrasts high- and low-rated responses, identifies recurring personal rewards, penalties, and semantics of subsequent reactions, and compiles them into an interpretable rubric under a common schema. The rubric is frozen once constructed.

Three-track scoring (Figure 3):

• Personalized: The personalized rubric scores sampled RPA responses together with their next user reactions. Turn scores are aggregated into trajectory-level satisfaction $P_u(c)$.
• Generic: A shared set of 12 generic quality dimensions scores the same sampled RPA responses (e.g., fluency, coherence, persona consistency).
• Session: A whole-session rubric evaluates multi-turn properties like persona stability, relationship development, and long-horizon coherence across 11 dimensions.

Aggregation: Let $P_u(c)$ be the mean turn-level experience score of candidate $c$ for user $u$, and $G(c)$ and $S(c)$ the mean Generic and Session overall scores. For $K=5$ users and maximum score $s=5$:

$$\bar{P}(c) = \frac{1}{K} \sum_{u=1}^{K} P_u(c)$$ $$U\text{-Score}(c) = \frac{100}{s} \bar{P}(c)$$ $$G\text{-Score}(c) = \frac{100}{2s} \left( G(c) + S(c) \right)$$ $$\text{Overall}(c) = \frac{100}{3s} \left( \bar{P}(c) + G(c) + S(c) \right)$$

U-Score weights users equally, G-Score weights Generic and Session equally, and Overall is a display-only summary.

Empirical Validation / Results

4.2 Main Evaluation Results

Table 2 reports results for 16 candidates across five users (U1–U5), Generic turn quality, and Session quality. Key findings:

• No dominant candidate: GPT-5.4 leads in Generic (4.73) but not in Session (4.18, where Claude Sonnet 4.6 leads at 4.32). Claude Sonnet 4.6 leads Session but not Generic.
• User-dependent winners: U1 and U4 favor Qwen3-Max and Claude Sonnet 4.6 respectively; U2 prefers DeepSeek V4 Pro; U3 and U5 prefer Claude Opus 4.8.
• Open-weight models: GLM-5.1 reaches the leading tier by balancing all three tracks, while role-play-specialized MiniMax M2-her and CoSER-Llama-3.1-70B remain lower.
• Stability: All model-rank Spearman correlations across independent repetitions are at least 0.959.

4.3 Agreement with Human Experience

On held-out turns, within-session pairwise agreement (ties scored 0.5, chance = 0.500) is measured for different rubrics and reaction visibility. Table 3 shows that Personalized with reaction achieves the highest macro agreement (0.613), outperforming Personalized without reaction (0.551), Generic overall with reaction (0.507), Generic without reaction (0.480), and a MiniMax-aligned baseline (0.467). Gains are positive for every user.

Theoretical and Practical Implications

• Evaluation methodology: PALATE demonstrates that fixed-history evaluation introduces systematic conditional bias, and that user-independent scoring misses genuine individual differences. The framework shifts the evaluation unit from an isolated RPA to a user–RPA pair, providing a more ecologically valid assessment.
• Interpretability: By decomposing evaluation into three tracks (personalized, generic, session), PALATE produces interpretable performance profiles that reveal cross-track capability mismatches and per-user preferences, rather than compressing all information into a single ranking.
• Practical use: Developers can use PALATE to identify which user segments a given RPA serves well or poorly, and to diagnose specific weaknesses (e.g., good turn quality but poor session coherence). The personalized rubric construction is automated and reusable, enabling scalable user-centric evaluation.
• Generalizability: The framework is not limited to role-playing; the principle of learning both a behavior policy and an experience preference from the same person could extend to other interactive systems (e.g., tutoring, therapy, game NPCs).

Conclusion

PALATE addresses two fundamental problems in RPA evaluation: borrowed history bias and user-independent scoring. By training per-user simulators from real dialogue histories and constructing personalized rubrics from experience annotations, it enables free multi-turn evaluation where each candidate co-constructs its own trajectory and is scored from the perspective of the same user. Experiments across 16 candidates show that advantages on generic turn quality, session quality, and personalized experience do not coincide, and that users do not share a single best candidate. Personalized scoring achieves higher agreement with human judgments than generic or baseline rubrics.

The central output of PALATE is not a scalar leaderboard but an interactive evaluation profile that locates cross-track capability mismatches and per-user differences. Future work includes expanding the user panel beyond five users and calibrating cross-candidate rankings against human judgments. The released resources (300 character profiles, 5,133 annotated turns, user simulators, rubrics) enable further research on individual preferences, interaction dynamics, and evaluation methods.