The Third Variable in Attribution: Serving Stack and the Cost Reality of Memory/Compression
Highlights of This Issue
The strongest signal this issue is the expansion of "attribution" from the two-dimensional framework of model vs. harness to a third variable, while the two sub-directions of memory and compression face a reality check on costs and benefits.
Measuring the Serving Stack Instead of the Model is the first to treat serving stack configurations (tokenizer batching, context truncation, inference scheduling) as controlled independent variables, demonstrating that the same model+harness pair can differ by 18pp Pass@1 solely due to stack-level parameters—this is the third dimension extending the attribution framework from "model vs. harness" to "model vs. harness vs. serving stack," following the Scaffold Effect in Issue 2 and Don't Blame the LLM in Issue 1. What Does Multi-Harness RL Learn? provides corroborating evidence on the RL training side: the evaluation harness is the dominant variable (2.14%→9.27%, 4.3×), while the Within/Cross GRPO grouping rules have almost no impact; cross-harness credit only yields configuration adaptation rather than transferable capability—adding hard evidence on the training side that "harness update ≠ harness benefit."
This issue simultaneously presents "benefit reality" and "cost reality" for the memory and compression sub-directions. VibeMemBench uses a memory-on/off paired protocol to isolate memory interventions, finding that existing memory systems (Mem0/SimpleMem/MemoryOS/A-MEM) do not exceed the memory-off baseline in 11/12 pairings, with failures attributed to form degradation (69.3%) rather than semantics; An Empirical Cost Attribution of Context-Compression Gateways decomposes the token bill of compression gateways into three levers, proving that compression ratio is not end-to-end savings, and introduces the "recall trap"—precise recovery in non-destructive gateways consumes one compression segment per recall; CliffCompaction demonstrates with a rule-based "truncate-only, no rewriting, discard old compressions" design that training-free methods can match or exceed learned compression on long-horizon tasks. Together, these three works place the interaction costs from Issue 5's Context Compression Cost and the trainable memory from Issue 6's SWE-MeM under stricter cost-benefit scrutiny.
On the benchmark validity side, SWE-Serve is the first to treat "production inference service correctness" as a first-class evaluation target, using model-serving E2E tests to separate "locally correct" from "production correct" and quantify a 23.4pp gap, with matched test-removal controls proving the unique contribution of E2E tests; RRSI systematically introduces regularization ideas into harness self-evolution, validating OOD generalization on five held-out benchmarks (up to +4.7pp); The Tasteful Agent is the first to make "taste at decision forks" a measurable target, using automatically mined decision forks to assess intermediate decision quality rather than end-to-end success. Community-side independent audits continue to provide a skeptical lens on benchmark validity: the git show leak audit quantifies the performance drop on SWE-Bench Pro after removing gold patches and pinpoints specific leak channels, while the DeepSWE defect audit and DeepSWE speculative reward hacking question the pass-rate interpretation from the angles of task defects and "imagined hidden scorers," respectively.
Community and Dynamics
The State Of AI Harness Engineering 2026 performs a meta-analysis of 246 repositories and 57 papers, quantifying the 68%→88% variance of the same model across 8 harnesses, providing a baseline anchor for harness attribution; HarnessTax quantifies on 21 model-harness pairs that harness choice changes success rates by only ±2-5 points but changes costs by up to 5×, mutually reinforcing the cost-accounting discipline from Issue 6. The Clean Coding Index rescore frontier benchmarks with continuously updated trajectory-level contamination filtering and per-model hacking-attempt metrics, serving as a dynamic increment to static hackability audits. On the product side, the Luna regression report for Codex CLI provides community evidence that a harness-only update caused a sharp performance drop at high effort; It Was the Harness, Not the Model claims that approximately 90% of coding-agent failures can be attributed to harness components (stopping, done-gates, self-grading), but lacks full text and code, so it should be read as a directional observation.
Open Questions
- Measuring the Serving Stack Instead of the Model demonstrates that serving stack parameters can cause an 18pp difference. When "model vs. harness" attribution has expanded to the third variable of the serving stack, should evaluation reports include tokenizer batching, context truncation, and inference scheduling parameters in a mandatory reporting list to make harness comparisons reproducible under stack-agnostic conditions?
- VibeMemBench proves that existing memory systems do not exceed the memory-off baseline in 11/12 pairings, with the primary failure being form degradation rather than semantics. Does this imply that the bottleneck for memory systems lies in the "recording format" rather than the "retrieval algorithm," and that trainable memory systems like SWE-MeM from Issue 6 need to first solve the prerequisite question of "what form of experience is actually usable by the solver"?
- What Does Multi-Harness RL Learn? proves that cross-harness credit does not yield transferable capability. When the gains from multi-harness RL mainly come from the evaluation harness rather than grouping rules, does this mean that the benefits of "training with more harnesses" are overestimated, and that harness-RL reports must declare grouping boundaries and test on unseen harnesses?
- The recall trap in An Empirical Cost Attribution of Context-Compression Gateways combined with the re-acquisition costs from Issue 5's Context Compression Cost suggests that compression evaluation must simultaneously report the three elements of "compression ratio, recall frequency, and session length," and that single-turn compression benchmarks (e.g., 86.5% quality retention) can never serve as evidence for multi-turn cost arguments?
Papers in this issue
RRSI regularizes both proposal and selection in recursive harness evolution, achieving the only out-of-distribution gains that clear the base harness while using 37% fewer tokens than unregularized search.
Editor's noteRRSI systematically introduces regularization ideas into the harness self-evolution loop: on the proposal side, it uses time-annealed edit budgets and evidence-aware credit assignment to constrain candidates; on the selection side, it uses leak filtering, noise-adjusted acceptance, and cost-aware pruning. It validates OOD generalization on eight benchmarks spanning coding/agentic workspace/engineering design (up to +4.7pp on five held-out benchmarks, with ~30% reduction in policy tokens). Compared to HarnessCompass's generalization gate and active feedback in Issue 4, it implements "overfitting prevention" as joint regularization on both proposal and selection sides, with cross-model robustness validation (Claude Opus 4.8 and Gemini 3.5 Flash) and public code trajectories. For any researcher building or auditing self-evolving harnesses, its leak filtering and cost-aware acceptance are directly usable engineering constraints.
Existing memory systems fail to improve coding agent performance because raw transcript volume, not instruction semantics, degrades retrieved fix patterns in 69.3% of failures.
Editor's noteThe first benchmark to make "executable downstream benefits of memory systems for repository-level coding tasks" a controlled paired evaluation: using SIEVE construction to ensure each target has execution-verified useful experience, then isolating memory interventions with a memory-on/off paired protocol. The core finding is that existing memory systems (Mem0/SimpleMem/MemoryOS/A-MEM) do not exceed the memory-off baseline in 11/12 pairings, with failures attributed to form degradation (69.3%) rather than semantics. Compared to SWE-MeM in Issue 6 (which treats memory management as a trainable tool), it shifts the evaluation target to "memory systems as the component under test," using executable tests rather than recall scoring, serving as a template for memory system evaluation protocols. Limitations include a small scale of 111 targets and bootstrap intervals crossing zero, so it should be read as directional evidence.
SWE-Serve reveals a production correctness gap where removing end-to-end serving tests boosts agent benchmark pass rates by 23.4 points, exposing that local success often fails in real inference systems.
Editor's noteThe first repository-level benchmark to treat "production inference service correctness" as a first-class evaluation target: 53 SGLang production tasks, six inference engineering families, using model-serving E2E tests to separate "locally correct" from "production correct," quantifying a 23.4pp gap, with matched test-removal controls proving the unique contribution of E2E tests (2.0× fail-to-pass conversion). Compared to DeepSWE's original long-horizon tasks in Issue 4, it is the first to bring full-stack coordination of inference engineering into executable evaluation, with contamination disclosure (closed-book + canary + trajectory audit) and hackability audits (no-op/oracle controls + adversarial probing) meeting hard standards. Limitations include a single repository (SGLang) and single GPU, but the gap quantification and validation protocol are worth direct reuse.
Cross-harness credit assignment adds no detectable portability over within-harness grouping when exposure is held fixed, differing by only 0.25 pp on a held-out harness.
Editor's noteThe first systematic study of credit assignment in multi-harness RL: fixing the Qwen3-8B starting point, replaying the same frozen records across four harnesses (Aider/OpenHands/Qwen Code/SWE-agent), comparing Within (one group per task-harness) and Cross (pooled within task) GRPO grouping rules. The key finding is that the evaluation harness is the dominant variable (2.14%→9.27%, 4.3×), while grouping rules have almost no impact (Cross−Within only +0.25pp on held-out harnesses with confidence intervals crossing zero)—cross-harness credit only yields configuration adaptation rather than transferable capability. Compared to LEGO-RL and HarnessOpt-Bench in Issue 2, it turns the harness from a fixed scaffold into a training variable and provides the discipline that "multi-harness RL reports should declare grouping boundaries and test on unseen harnesses." This is another hard piece of evidence for "harness update ≠ harness benefit" on the RL training side.
Tool-schema filtering, not content compression, dominates coding-agent token savings, though content compression's quadratic accumulation overtakes it after turn six.
Editor's noteThe first work to perform lever-level cost attribution for compression gateways in multi-turn coding agents: decomposing the token bill into three levers—tool schema filtering, content compression, and history summarization—proving that compression ratio is not end-to-end savings. The core finding is that content compression savings grow quadratically with turns (≈3,350N²) while tool filtering is linear, and it introduces the "recall trap"—precise recovery in non-destructive gateways re-sends just-compressed bytes, consuming one compression segment per recall. Compared to Issue 5's What Does Context Compression Cost an Agent? which isolates interaction costs, it adds lever-level attribution and a cost model with caching pricing, warning that single-turn compression benchmarks (86.5% quality retention) are orthogonal to multi-turn agent costs. For compression gateway designers, tool schema filtering is the only unambiguously positive lever.
SWE-Review closes the loop on AI-generated pull requests with agentic code review, boosting resolve rates from 27.5% to 56.9% and enabling efficient test-time scaling.
Editor's noteThe first work to integrate agentic code review end-to-end into a generate-review-revise loop, accompanied by a 1,384 PR benchmark and 8,914 trajectory dataset, using DA/RRR as two operationalized metrics to measure review decision correctness and downstream revision utility, rather than merely comparing review text. Compared to CodeRabbit's wild feedback mining in Issue 5, it is the first to treat "whether the review improves the final patch" as a first-class metric and validate it with an executable environment, with review trajectories used for SFT distillation of the reviewer, mixed training with issue-resolution data, and test-time scaling selectors. Note that the main benchmark is the saturated SWE-bench Verified and the 500 issues are exactly the verified set itself (no held-out control), so DA/RRR absolute values should be read with benchmark validity skepticism.
Local serving stacks silently confound tool-use benchmarks: Ollama rejects some models' tool requests before inference, making capable models score 0% without ever running.
Editor's noteIntroduces serving stack configurations (tokenizer batching, context truncation, inference scheduling) as controlled independent variables in agent evaluation, demonstrating that the same model+harness pair can differ by 18pp Pass@1 solely due to stack-level parameters. Compared to the Scaffold Effect and Harness-Bench in Issue 2 which only vary harness logic, it adds the previously ignored third variable—serving infrastructure—providing a stack-agnostic reporting protocol and public trajectories. Limitations include a focus on local tool-use rather than repository-level tasks, but as the third dimension of evaluation attribution, it is a methodological reference for anyone designing or auditing coding-agent evaluations.
Taste-Bench reveals frontier LLM agents achieve only ~60% accuracy on long-horizon decision quality, but this taste is distillable, boosting end-to-end task success from 14.6% to 33.7%.
Editor's noteThe first benchmark to make "taste at decision forks" a measurable target: Taste-Bench automatically mines labeled decision forks from existing agent trajectories (no human annotation), measuring the model's ability to choose the better direction without seeing fork consequences. The strongest model only answers 59.7% correctly, with later fork evidence making it harder and larger inference budgets not improving accuracy. Compared to existing benchmarks that only measure end-to-end success, it advances evaluation granularity to intermediate decision quality and proves this judgment can be distilled into student models, improving held-out SWE-bench Pro end-to-end success. For long-horizon software evolution and harness engineering, it provides the previously missing evaluation dimension of "intermediate decision quality."
Approval laundering occurs when durable approval records omit effects from transitive workflows, and no record-only policy can guarantee correct decisions when identical visible fields require different effect-specific actions.
Editor's noteThe first work to formalize the coverage relationship of "approval records-execution closure" as closure-bound approval: proving that when policy-visible fields are identical but required effect decisions differ, any deterministic or randomized record-only policy cannot guarantee both simultaneously, and providing a six-category operation effect vocabulary, a 111-pair fixed approval object/trajectory gradient of 40→17→13, and a forward-looking effect-bound record fix (residual reduced from 10 to 3 on holdout). Compared to BenchShield's reward completeness formalization in Issue 6 and When Context Gets Root's privilege escalation in Issue 3, it moves the audit target to "coverage of transitive effects by approval records." For researchers building or auditing approval-gated harnesses, its effect vocabulary and evaluation protocol are directly usable safety constraints; limitations include paper-only with no public artifact.
CLIFFCOMPACTION, a rule-based autocompaction method that discards stale context verbatim, cuts inference costs by up to 50% while improving coding agent performance and enabling state-of-the-art continual learning.
Editor's noteA rule-based autocompaction method that deliberately sacrifices compression recall for precision: it only truncates/discards original text, never rewrites or summarizes, and each compression discards the previous compression history, only compressing the active session since the last event—this "cliff" design avoids summary-of-summary drift and maintains KV-cache effectiveness, achieving up to 50% cost savings with maintained or improved performance, and supporting over a million tokens of continual learning on KernelBench (3.58× speedup at 400 steps). Compared to learned/attention-guided compression like CompactionRL in Issue 3 and AttnCompress in Issue 5, it provides a training-free, model-agnostic, cache-friendly strong baseline, directly challenging the assumption that "compression requires learning." Note that the main benchmark is saturated with no held-out control, so gains should be read with benchmark validity skepticism.
