# AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses

> Strong-to-weak scaffolding, where a strong builder creates inference-time harnesses for a weaker model, nearly doubles Theory-of-Mind accuracy without any parameter updates.

- **Source:** [arXiv](https://arxiv.org/abs/2608.12307)
- **Published:** 2026-08-14
- **Permalink:** https://picx.dev/p/Rruz4J
- **Whiteboard:** https://picx.dev/p/Rruz4J/image

## Summary

# AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses

## Summary (Overview)

- **Core contribution**: This paper introduces and systematically studies "strong-to-weak scaffolding"—a test-time capability transfer paradigm where a strong "builder" model constructs inference-time harnesses (scaffolds) that improve a weaker "target" model's performance **without any parameter updates** to the target.
- **Key finding**: Scaffolding nearly doubles the average target model performance from 0.49 to 0.91 macro-average accuracy on Theory-of-Mind benchmarks, with 100% of runs exceeding the baseline and all 11 builder configurations producing positive uplift.
- **Mechanism**: Gains come primarily from offloading unstable model reasoning into deterministic code, benchmark-specific routing, and strict answer-format enforcement—not from encouraging longer reasoning or broader sampling.
- **Key principles discovered**: Builder capability and reasoning effort dominate platform choice; target model benefits scale with its "headroom" (available correctable errors); validation efficiency is high (median 5 evaluations, minimal overfitting).
- **Implications**: This reframes capability transfer as a complement to training-time distillation, suggesting models can transfer cognitive structure through external harnesses rather than weight updates.

## Introduction and Theoretical Foundation

The paper addresses a fundamental question in model deployment: **When a small model fails on a task, is the failure due to insufficient internal capability or excessive cognitive load imposed by task presentation?**

Drawing on cognitive load theory (Sweller, 1988), the authors identify two complementary routes to improving performance:

1. **Make the model more capable** — the dominant approach in distillation research (Hinton et al., 2015; Hsieh et al., 2023; Agarwal et al., 2024)
2. **Make the task easier to solve** — realized through inference-time harnesses: external structures with routing logic, prompt templates, verification checks, memory, and tool use

The paper formalizes **strong-to-weak scaffolding** as a distinct setting where a strong builder model $M_{build}$ constructs a scaffold for a fixed weaker target model $M_{tar}$, with the builder never seeing the hidden test set. This isolates a practical form of transfer where capability flows through the harness rather than model weights.

The Theory-of-Mind (ToM) domain is chosen as a stress test because it requires tracking nested beliefs, perspective shifts, hidden information, and Bayesian goal inference—demanding tasks for smaller models that nonetheless contain exploitable structure.

## Methodology

### Formal Setting

The builder searches over a space of possible scaffolds $\mathcal{S}$:

$$S^{\star} = \arg \max_{S \in \mathcal{S}} \operatorname{Acc}\bigl(S, M_{\mathrm{tar}}; \mathcal{T}\bigr)$$

Since the test set $\mathcal{T}$ is hidden, the builder uses validation performance as a proxy:

$$\hat{S} = \arg \max_{S \in \mathcal{S}_{\text{build}}} \operatorname{Acc}(S, M_{\mathrm{tar}}; \mathcal{V})$$

### Experimental Setup

- **Benchmarks**: Four ToM datasets aggregated into a 3,900-item hidden test set:
  - **BigToM** (1,200 items): Binary belief/goal/action questions on observation status
  - **Hi-ToM** (1,200 items): Nested belief questions with recursion order 0–4, deception, multi-room tracking
  - **MMToM-QA** (600 items): Binary Bayesian goal/belief inference from action traces
  - **MuMA-ToM** (900 items): 3-choice multi-agent belief/social-goal/belief-of-goal questions

- **Validation**: Each builder receives a 195-item (5%) validation sample; test set is held out
- **Models**: 11 builder configurations (Opus-4.7 with 4 reasoning effort tiers, Sonnet-4.6, GPT-5.5, GPT-5.4-mini, Codex-5.3, Gemini-3.1-Pro, Gemini-3.5-flash, Grok-0.1); 2 target models (GPT-5.4-mini, Gemini-3.5-flash)
- **Platforms**: 3 agentic coding harnesses (Cursor, Claude Code, GPT Codex)
- **Total**: 72 experiment runs, each repeated 3 times for stability analysis

### Baselines

- **Vanilla**: Direct model call without scaffolding (GPT-5.4-mini: 0.488; Gemini-3.5-flash: 0.761)
- **Human-Inspired Harness**: UserHarness framework (GPT-5.4-mini: 0.939; Gemini-3.5-flash: 0.941)

## Empirical Validation / Results

### Main Results (Aspect 0)

| Metric | Value |
|---|---|
| GPT-5.4-mini vanilla baseline | 0.488 |
| GPT-5.4 vanilla baseline | 0.619 |
| GPT-5.4-mini human-inspired | 0.939 |
| Mean over all scaffolded runs | 0.763 (+0.275) |
| Best run (GPT-5.5, GPT Codex) | 0.912 |
| – uplift over baseline | +0.423 (86.7%) |
| Builder beating vanilla baseline | 100% |

The best scaffold outperforms both vanilla baselines on all four benchmarks and matches UserHarness on BigToM (1.00 vs. 0.95), with remaining gaps on Hi-ToM (0.80 vs. 0.87), MMToM-QA (0.84 vs. 0.98), and MuMA-ToM (0.88 vs. 0.96).

### Key Findings by Aspect

| Aspect | Key Finding |
|---|---|
| **Stability** | Mean within-cell std = 0.036; variance concentrated in deterministic-solver strategies |
| **Validation Efficiency** | Median 5 validation evaluations; best validation score tracks full-set accuracy (Pearson r = 0.96); refinement count uncorrelated with final performance (r = 0.17) |
| **Techniques** | Format enforcement (100%), greedy decoding (98%), benchmark routing (95%), forced CoT (79%), polarity logic (79%) are most prevalent |
| **Platform Effects** | Native-platform advantage averages only +0.013 (p = 0.484); platform×effort interaction exists—native harness helps only with high reasoning budget |
| **Target Dependence** | Uplift follows a **headroom law**: realized uplift strongly predicted by target's headroom (1 − baseline) on that benchmark (Pearson r = 0.75) |
| **Builder Effort** | Monotonic improvement: 0.711 → 0.793 → 0.807 → 0.856 from low to extra-high effort (Spearman ρ = 0.77) |

### Cognitive-Load Reduction

Determinism fraction (share of items answered entirely by code/rules) strongly predicts accuracy:

$$r = 0.72 \text{ (Pearson correlation between determinism fraction and final accuracy)}$$

Benchmark ofloadability varies sharply: BigToM ≈ 0.94, Hi-ToM ≈ 0.51, MMToM-QA ≈ 0.44, MuMA-ToM ≈ 0.36.

### Residual Errors

Top scaffolds fix 83% of baseline errors while breaking only 7% of correct items. Residual errors concentrate in:
- **Hi-ToM**: Accuracy declines from 0.999 (order 0) to 0.700 (order 4); deception reduces performance
- **MMToM-QA**: Bayesian goal-inference subtypes (qtype 2.1: 0.680)
- **MuMA-ToM**: Social-goal (0.872) and belief-of-goal (0.880) labels

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Formalizes strong-to-weak scaffolding** as a distinct inference-time capability transfer setting, complementing training-time distillation
2. **Identifies the "headroom law"**: Scaffolding acts as competence recovery—its payoff is governed by the target's available correctable errors, not model identity
3. **Establishes cognitive-load reduction as the mechanistic core**: Builders act as "compilers of task competence," spending one-time reasoning to encode task structure into executable form

### Practical Design Principles

- Use the **strongest available builder** with **high reasoning effort**
- Spend only a **modest number of validation evaluations** (median 5 suffices)
- Prioritize **cognitive offloading** for provable sub-tasks
- Build **several independent scaffolds** and select/ensemble for complementary repairs
- Apply scaffolding **selectively at the sub-task level** for already-strong targets, gated by measured headroom

### Evaluation Reframing

The setting reframes evaluation from "How well can a model solve a task?" to "How well can a stronger model construct the conditions under which a weaker model can solve it?"—creating a potential benchmark for builder-model capability that measures externalization of reasoning into procedures, tools, and constraints.

## Conclusion

Strong-to-weak scaffolding works because a capable builder can act as a **compiler of task competence**: it spends a one-time reasoning budget to identify and encode task structure into an inference-time scaffold, after which a weaker and cheaper target model can execute the task at a level approaching much stronger models.

Key takeaways:
1. **Scaffolding does not replace raw reasoning capability—it reallocates it**: the builder performs structural reasoning once; the target handles residual model-dependent cases
2. **The approach is strongest for structured sub-problems** with explicit decision procedures, but weaker for genuinely hard reasoning cores (nested belief tracking, deception, counterfactual goal inference)
3. **Two complementary routes to stronger systems exist**: improving internal capability (training) and making tasks easier to execute (harness design)—future progress depends on both, and on the co-evolution of models and inference environments

### Future Directions

- Testing on broader benchmark families with different mixtures of symbolic structure, ambiguity, and open-ended reasoning
- Developing strong-to-weak scaffolding as a **standard benchmark for builder models** with secondary metrics (validation usage, inference cost, scaffold complexity, robustness)
- Studying **harness self-evolution**: agents automatically improving the environment that structures their own reasoning
- Exploring the interaction between training-time and test-time capability transfer

---

_Markdown view of https://picx.dev/p/Rruz4J, served by PicX — AI-generated visual whiteboard summaries of research papers._