Summary (Overview)

• Novel Benchmark: Introduces Taste-Bench, the first benchmark measuring "taste" in LLM agents—the ability to make good long-horizon decisions at decision forks where outcomes are not immediately visible.
• Automatic Mining: Constructs 502 taste questions automatically from existing agent trajectories (2,677 engineering rollouts on SWE-bench Pro and 1,132 research runs from RE-Bench/HCAST) without human annotation, using two complementary methods: parallel trajectories (divergent attempts at the same task) and detour trajectories (self-corrections within a single run).
• Key Findings: Frontier models struggle significantly—the best model (GPT-5.6 Sol) achieves only 59.7% accuracy (random guessing = 25% under dual-order protocol). Accuracy drops dramatically with time horizon (62.3% for in-prefix evidence to 21.0% for more-work evidence), and increased reasoning budget does not improve accuracy.
• Trainable Taste: Demonstrates that taste can be distilled—a student model trained via teacher distillation (with privileged outcome information) improves from 30.0% to 47.9% on held-out tasks and boosts end-to-end executor success on SWE-bench Pro from 14.6% to 33.7%.
• Benchmark Quality: Human review confirms 98.8% agreement with mined labels, and the benchmark is publicly released with code and dataset.

Introduction and Theoretical Foundation

The paper addresses a critical gap in LLM agent evaluation: while agents increasingly work on long-horizon tasks (ML research, software engineering, scaffold refinement), existing benchmarks only measure end-to-end success—whether the agent finishes the task—but provide no measure of the quality of decisions made along the way.

Definition: The authors formalize "taste" as the ability to make good long-horizon decisions—choosing the better direction at points where multiple paths are available and the consequences of the choice only appear much later in the trajectory.

Key Insight: The later part of a trajectory provides hindsight evidence for earlier decisions. Agent systems routinely make multiple attempts at the same task; when attempts diverge at a common point (a decision fork), the recorded outcomes identify which direction was better. This allows automatic labeling without expert annotation.

Formal Framework: Given attempts at task $q$ sharing prefix $h_t = (o_0, a_0, \ldots, o_t)$ that diverge into candidates $c_1$ and $c_2$ at fork time $t$, each branch $i$ produces evidence $E_i$ mapped to a scalar outcome $U$. The label is:

$$y = \arg \max_{i \in \{1, 2\}} U(E_i) \tag{1}$$

The question is $x = (q, h_t, c_1, c_2)$ where everything after the fork is hidden. A model's taste is estimated as:

$$\widehat{T}(\pi) = \text{fraction of questions where } \pi(x) = y$$

Methodology

Fork Mining — Two Complementary Constructions

1. Parallel Trajectories: Aligns pairs of attempts at the same task that diverge at a fork. The shared prefix becomes the question context, and the two directions become candidates. The recorded outcomes (test passes, research scores) determine the label. Captures wrong judgments the agent never notices.

2. Detour Trajectories: A generator model reads a single trajectory with its outcome and locates three events: (a) the step where the agent takes a direction, (b) the observed failure ending that direction, (c) the step where the agent takes a different direction that completes the task. The fork is placed before the abandoned direction. Captures wrong judgments the agent corrects later.

Filtering Pipeline

• Trivial questions removed: Questions answerable from candidate wording alone (judged by models seeing only candidates, no trajectory)
• Undecidable questions removed: Questions where the label is not clearly consistent with the full record (judged by models seeing the complete task record)
• Human review: 100 sampled questions reviewed by two independent reviewers; 98.8% agreement with mined labels (Cohen's κ = 0.973)

Evaluation Protocol

• Each question evaluated twice (seeded order and exact reverse) to address position bias
• A question counts as correct only when both orders are answered correctly
• Random guessing accuracy = 25% (product of ½ per order); position-biased models score 0%

Distillation Recipe

• Base model: Qwen3.6-27B with LoRA adapters
• Task-disjoint folds: 390 engineering questions split into two folds; each student trains on one and evaluates on the other (no task seen in both)
• Teacher-student setup: Privileged teacher (same frozen base model) sees the question plus a demonstration of the supported candidate; student sees only the benchmark-visible context
• Loss: Token-level forward KL from student to teacher, computed on teacher-sampled continuations (following SDPO framework)
• End-to-end evaluation: Student's judgment at each fork is written as advice into the task context; a fixed executor agent completes the task independently

Empirical Validation / Results

Model Performance on Taste-Bench

Finding 1: Current frontier models show limited taste—even the strongest cannot reliably identify the better direction on binary choices.

Effect of Time Horizon

Forks are annotated with four ordinal time-horizon levels (in-prefix, inferable, next-step, more-work):

Finding 2: Model errors concentrate on forks with long time horizons—accuracy falls as horizon increases, confirming that answering requires predicting later work.

Effect of Reasoning Budget

• Increasing reasoning effort (3 settings) changed GPT-5.6 Sol by −0.2 points and GPT-5.6 Luna by +2.2 points (statistically insignificant)
• Models produce most reasoning tokens at the more-work level—the level with lowest accuracy

Finding 3: Larger reasoning budget does not improve taste; models recognize hard forks but cannot resolve them.

Comparison with End-to-End Benchmarks

• Pearson correlation with SWE-bench Verified: $r = +0.63$ ($R^2 = 0.39$)—only partial correlation
• Engineering subset correlation: $r = +0.37$ (despite being mined from SWE-bench Pro tasks)
• Top 4 SWE-bench Verified models are within 4.0 points there but 10.7 points apart on Taste-Bench

Distillation Results

End-to-end SWE-bench Pro success (41 held-out tasks):

Finding 4: Taste is distillable—the student gains 17.9 points on unseen tasks and achieves most of the upper bound on end-to-end success (+19.1 points over baseline).

Theoretical and Practical Implications

• Measurement without annotation: The paper demonstrates that decision quality can be measured automatically from hindsight over existing trajectories, eliminating the need for expensive expert annotation and enabling scalable evaluation across new domains.

• Complementary benchmark dimension: Taste-Bench measures process quality rather than just outcome success, providing a new axis for model evaluation that is only partially correlated with end-to-end benchmarks ($r = 0.63$). This suggests that end-to-end success does not fully capture decision-making ability.

• Time-horizon insight: The finding that accuracy degrades with time horizon (from 62.3% to 21.0%) reveals a fundamental limitation in current models' ability to reason about long-term consequences—a critical capability for autonomous agents.

• Trainability of judgment: The successful distillation of taste (requiring only a privileged teacher with outcome knowledge) suggests that judgment can be improved without additional task-specific data, and that improved judgment directly translates to better task outcomes.

• Practical framework: The advice-injection paradigm (student judges forks, executor follows advice) provides a practical architecture for improving agent performance on complex tasks.

Conclusion

The paper makes three primary contributions:

1. Formalization: Defines "taste" as the ability to choose the better direction at decision forks and shows it can be measured from hindsight over existing trajectories without human annotation.

2. Benchmark: Releases Taste-Bench (502 questions across software engineering and ML research), demonstrating that current frontier models achieve only ~60% accuracy, with errors concentrating on long-horizon forks where increased reasoning effort does not help.

3. Training method: Shows that taste is trainable via teacher distillation, producing students that generalize to unseen tasks and improve end-to-end task success.

Future directions include: extending the benchmark to more domains, exploring whether taste can be further improved through other training paradigms, and investigating whether the advice-injection framework can be integrated into production agent systems. The authors hope Taste-Bench provides "a practical basis for measuring and training the judgment of long-horizon agents."