Weekly Gallery
Latest edition
Oct 31 – Nov 7, 2026
Directions: 9 · Updated this issue: 8 · Picks: 57
Pretraining Data · Issue 13
The benefit boundary of synthetic data begins to be priced by compute
The clearest signal this issue is that the benefit boundary of synthetic data has shifted from a 'fixed ratio' to 'pricing based on compute and data constraints…
Rewriting Scaling Laws for Synthetic Data with Compute and Data ConstraintsFor the first time, explicitly incorporates the generation FLOPs of synthetic rewriting into the scaling law, connecting model size, training tokens, rewriting ratio R, and natural data repetition penalty, and proving that the optimal rewriting ratio is extrapolable: under compute-limited settings, it rises from <10% to 32% (3×10^18→10^22 FLOPs); extrapolating to 33× FLOPs to train a 22B model yields an error of only 1.1%, and this model surpasses its rewriter on 10 of 16 Paloma sources—the rewriter's capability is not an upper bound on gains. Compared to Scaling Repeated-Data (issue 10) and Scaling Laws for Mixture Pretraining (issue 3), which treat repetition/mixing as scaling variables, the increment is bringing 'rewriting generation cost' into the shared budget and turning the synthetic ratio into a decision variable predictable by compute. The criterion remains base loss (Paloma BPB), but scale, extrapolation validation, and the data-limited same-size rewriter comparison suffice to change ratio decisions.- Rethinking Pre-pretrainingFor the first time, systematically examines synthetic pre-pretraining (PPT) under compute-matched settings: across three training regimes (data-limited, unlimited, over-Chinchilla, up to 881M) and four curricula, allocating compute to PPT never outperforms equal-compute pure natural language training; previously reported NCA gains are highly sensitive to weight initialization (the original scheme reproduces a 5.4% gain, but after changing initialization, the baseline surpasses it by 9.0%), and what is learned during PPT is forgotten when transitioning to natural language; interleaved mixing only prevents forgetting but does not improve natural language performance. Relative to studies like Priming Base Models (issue 11) that treat mid-training as a ratio lever, the increment is treating the 'synthetic pre-pretraining stage' as an independent object of study and first making initialization a key ablation axis, directly refuting ratio hypotheses that rely on PPT warm-up. Limitations are the maximum 881M not reaching the ≥1B threshold and the criterion being validation BPB and downstream tasks, but it is strong negative evidence for the synthetic data benefit boundary.
- Data Schedules and Learning Rate Schedules Should Be Designed TogetherFor the first time, systematically proves that the optimality of data schedules (temporal allocation of PT/MT mixing ratios) in mid-training and learning rate schedules are interdependent: the same data schedule shows reversed final MT loss rankings under full LR decay (FDR) versus constant LR; the optimal joint design is 'LR decay starts after MT data enters', with the optimal relative LR decay duration stably in 10–40%; TS and ZL data schedules prefer different decay durations. Relative to Priming Base Models (issue 11), which treats mid-training ratios as a corpus design variable, the increment is explicitly making LR schedule a design axis coupled with data schedule and providing a bias-variance theoretical explanation. With 630M/2.99B, 250+ controlled runs across dense/MoE and multiple budgets, it imposes a strong design constraint on mid-training ratio evaluation: 'data schedules cannot be evaluated independently of LR schedules'.
MoE & Sparse Experts · Issue 13
Symmetric Recovery: A New Principle of Routing Consistency for FP4 MoE RL
The main theme of this issue is new evidence for "training-inference consistency" at both the quantization and execution levels. On the FP4 RL side, Aligning Tr…
Aligning Training and Rollout Policies for End-to-End FP4 RL on Mixture-of-Experts LLMsMust-read. The first to systematically prove that the FP4 RL recovery recipe effective on dense models (RolloutResQ) is not only ineffective but harmful on MoE, and attributes the root cause to the asymmetry of quantization recovery—only when residual correction is applied simultaneously on both the training and inference sides (symmetric recovery) can routing consistency be restored, reducing the accuracy gap from 20+ points to within 1 point on Qwen3-30B-A3B. Compared to TRACE's rollout-guided quantization-aware training in Issue 11, this paper explicitly takes symmetry as the principle and reports complete routing alignment metrics with dense baselines via PCC and top-k matching rate.- Matching the Training-Time Weight Rule in Inference-Time Expert DeactivationMust-read. The first to conduct a controlled comparison of four inference-time expert deactivation implementations on the same set of disabled experts, directly linking 'high-loss implementations' to the training-time weight rule (original probabilities vs. normalized weights) via a weight-sum model, providing quantitative evidence of perplexity gaps ranging from 14 to 28,740 times across six checkpoints of OLMoE/DeepSeek-V2-Lite/Granite/Qwen3. Compared to the controlled comparison in Issue 11's Don't Skip the Baseline and the normalization decoupling in Issue 5's Training-Free Halving, this paper establishes 'whether the training-time weight rule is matched' as the first-order factor explaining implementation differences, and extends to predictions of width adjustments with varying numbers of active experts.
- FastAFD: When Does Attention-FFN Disaggregation Pay on Rack-Scale NVLink?Must-read. The first to conduct an end-to-end empirical comparison of AFD with workload-tuned wide EP on rack-level NVLink in GB200 NVL72, providing a three-factor decomposition (batch gain × overlap gain × disaggregation cost) and a zero-refit analytical model with a prediction error of 2.9%, and the first to measure the impact of speculative decoding on the optimal A:F ratio and benefits of AFD. Compared to Afore's micro-batch-level demand prefetching in Issue 10, this paper answers with empirical measurements rather than analytical models whether AFD is worthwhile on rack-level NVLink, yielding a conditional conclusion of 0.99–1.17×.
Efficient Sequence Modeling · Issue 13
Hierarchical Sparse Attention Reduces Selection Cost to Logarithmic
The most prominent signal this issue is that hierarchical sparse attention reduces the selection cost from O(L) to O(log L), and for the first time achieves mil…
Recursive Sparse Attention: Million-Token Length Extrapolation at Logarithmic CostFirst to reduce the selection cost of block sparse attention to O(log L) and achieve 2M extrapolation under a fixed 32K training length: candidate-local RoPE keeps relative distances at each level within the training range, fundamentally eliminating the need for extrapolation; summaries of unselected candidates share a softmax with exact tokens to maintain normalization consistency, and a shared pooler is used for both selection and summary keys. Relative to HiLS's level-wise scoring and PISA's discarding of unselected blocks from Issue 5, the increment is a complete solution with logarithmic selection cost plus extrapolation-free, RULER 1M 60.5 vs DSA+DCA 36.8, with a full 32K→2M degradation curve and compute-matched comparisons. Anyone working on trainable block-sparse indexers or long-context extrapolation should read.- Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter ReadoutFirst to systematically prove the sensitivity asymmetry between erase and write gates in the gated delta rule for state updates, advancing gate rank allocation from heuristics to sensitivity-analysis-guided: erase gates can be safely low-rank, write gates need higher rank; also derives a closed-form scaling law for state row energy, designing matched-filter readout gains to dynamically balance state rows with different decay rates and extend the memory horizon. Relative to the fixed gate designs of Multi-State Linear Attention and GDN-2 from Issue 12, the increment is establishing "gate rank allocation" and "dynamic readout gain" as two new design axes, with 4K training extrapolating to 10M tokens without perplexity drift and near-100% passkey retrieval. Anyone working on mixed linear attention architectures should read.
- Shallow Encoding Emits Deep MemoriesFirst to systematically reveal two structural properties of recurrent states in hybrid models: the "state sink" (a rank-one term along the per-head mean direction accounts for most energy) and "writing is easier than reading," proposing shallow prefill plus storing decoding states as a sink plus low-rank factors, and proving that GDN/KDA/Mamba-2 updates are exactly closed under this form, with periodic compression error not accumulating with compression count. Relative to ReplaySSM's "caching inputs rather than states" from Issue 12 and sparse anchor replay from Issue 8's Just Let Linear States Forget, the increment is treating the low-rank structure of the recurrent state itself as a provably exact storage form, validated across five production model classes. Limitation: long-context degradation curves only cover 16K decoding drift.
Scaling Laws & Training Methods · Issue 13
Muon's win/loss begins to flip with data scale and tuning protocol
The most prominent clue this issue is that Muon's advantage over AdamW is beginning to be proven 'conditional', and the conditions themselves are shifting with …
Multi-Token Prediction Done Right (Soft-MTP)For the first time precisely decomposes the standard MTP cross-entropy gradient into a KL self-distillation term and a residual term, proving that the residual term has low SNR and worsens with scale, and is anti-aligned with the NTP learning direction; accordingly proposes the KL gradient–recovery trick, using straight-through gradient routing to recover the KL gradient exactly from existing NTP/MTP cross-entropy gradients at zero cost, implementing Soft-MTP with one line of code. Compared to Issue 11's ECHO, which reframed MTP as a causal autoregressive main objective, this paper is the first to reveal the hidden self-distillation signal in standard MTP at the gradient decomposition level and eliminate noise residuals, and demonstrates efficiency and effectiveness advantages over explicit KL distillation. Validated at 167M–1.2B multiple scales, with public code and combination with the Muon optimizer. Limitation: scale stops at 1.2B; extrapolation to larger scales requires self-assessment.- A Scaling Theory of Multi-Epoch Pretraining: Late-Time Overfitting and EnsemblingFor the first time incorporates multi-epoch training and ensembling jointly into scaling law theory: deriving the DMFT for multi-epoch SGD under fixed data repetition, giving optimal stopping time and compute allocation laws (ensemble size grows at least as fast as training time with compute), and validating on FineWeb that ensembling can delay optimal stopping and outperform wider/deeper single models under matched parameter-token budgets. Compared to Issue 12's Improving Data-Constrained Scaling, which treats ensemble size as an independent axis in empirical scaling laws, this paper provides theoretical mechanisms (variance reduction with E^-1, cross-correlation saturation) and stopping/allocation laws, and clearly distinguishes compute-matched from data-matched. Limitation: theory is based on random feature models; quantitative transfer to real Transformers requires self-assessment.
- Pretraining Data Scale Reverses the Muon-AdamW Fine-Tuning GapFor the first time systematically reveals that pretraining data scale can reverse the fine-tuning performance gap between Muon and AdamW: under small data budgets Muon dominates, under large data budgets AdamW overtakes, and the crossover point advances as model size increases. Proposes margin-to-spread ratio (MSR) as a new metric for predicting fine-tuning robustness, and proves in a linear associative memory model that GD's MSR grows with data budget and surpasses Muon's. Compared to Issue 11's When Does Muon Actually Win paired benchmark, this paper is the first to extend optimizer comparison from pretraining loss to the learning-forgetting trade-off after fine-tuning, and provides a theoretical explanation. Validated at 30M–600M multiple scales and multiple data budgets. Scale stops at 600M, fine-tuning covers only four datasets; extrapolation to larger scales and broader tasks requires caution.
Coding Agents · Issue 13
Verification Gap Outweighs Code Quality: Reliability Bottlenecks in Long-Horizon Chained Builds
The strongest signal this issue: when evaluation shifts from "single request" to "agent chaining builds on its own code," bottlenecks are collectively attribute…
Built on Broken Assumptions (BOBA): Why Coding Agents Cannot Be Left AloneUses 280 chains of 20 requests, 5,600 requests in the largest-scale paired experiment to separate "who wrote the code" from "whether it was verified": verified agent code builds almost as reliably as human code (gap ≤4pp), while unverified status doubles failure rates and each unverified mistake doubles the odds of failure again. Compared to last issue's SWE-Shift matched continuation, it is the first to treat "verification status" as a first-class control variable and provides intervention quantification showing a single external 'verification failed' message raises per-model completion rates by 8–16 points. 280 chains, 10 models, fully public trajectories (sealed images and grader), and contamination disclosure meet hard standards, making it the strongest positive attribution evidence for "harness updates ≠ harness gains."- ViBench 2.0: A Benchmark for Long-Horizon Software BuildingFirst to make "greenfield multi-turn product building" an end-to-end benchmark: agents build complete full-stack products from empty projects following 9–18 feature requests in a single session, with final all-or-nothing scoring on 544 test plans from a user perspective, plus held-out 9 products, independent external team reproduction, and wrong-fail/wrong-pass dual error validation. The core is the "reliability gap": plan pass rate 94–98% but pass@1 only 49–65%, open-weight zero passes. Compared to last issue's SWE-Shift matched continuation and Issue 11's LoLBench proposal inputs, it makes "from-scratch building + multi-turn self-evolution + final whole-product acceptance" a first-class evaluation object.
- Hard Issue, Weak Model, or Bad Run? Rethinking Failure Prediction for Coding AgentsAdvances last issue's Continue or Restart finding that "within-task attempts have no discriminative power" to "trajectory predictors have statistically zero gain after controlling for issue difficulty and model capability," using a non-specialized reference distribution to test that oracle gap and upset rates do not exceed random variation, and model-switching gains come almost entirely from extra attempts rather than choosing the right model (specialization explains only 4.6% of variance, run-to-run accounts for 42%). Compared to last issue, it adds model capability as an independent confounder, equivalence tests, and repeated-run pilots; its decomposition protocol (within-issue AUROC, non-specialized reference) is a directly reusable audit template for any researcher designing stop/switch strategies.
Recursive Self-Improvement · Issue 13
Self-Improvement Measurement Enters Statistical and Theoretical Foundations: Noise Bands, Adaptive Testing, and Computability Boundaries
The main thread this issue is that self-improvement measurement is beginning to develop 'statistical and theoretical foundations': claimed gains are re-examined…
Trace Evolution Bench: Measuring Whether Agents Improve from Their Own Environment TracesShould read. It explicitly makes 'frozen rescoring variance' a control baseline for self-improvement measurement: using four controls—locked held-out ids, validation-gated acceptance/rollback, four frozen no_evolve rescans, and per-seed paired excess—it separates the portion of endpoint Gain attributable to evaluation noise. Compared to Issue 9's Winner's Curse analytical modeling of selection noise, the increment is a reproducible protocol that specifically reports the magnitude of 'scores moving while versions are unchanged'—DeepSeek moves 7.5–12.5 percentage points across seeds with unchanged versions, seven units exceed their own noise band, but GLM/MiniMax do not reproduce, and no method exceeds its own band across all seeds. It is a methodological warning for any closed-loop evaluation relying on endpoint Gain.- Which Self-Improvements Should We Trust? Reliable Self-Improvement When Agents Reuse Their BenchmarksShould read. It is the first to systematically introduce the statistical theory of adaptive data analysis and multiple testing into the evaluation layer of self-improvement loops: the Reuse framework returns only promotion decisions, explicitly allocates error budgets across all possible decision histories, controls three types of false promotion sources (within-round selection, repeated testing, and adaptive dependence), and provides theoretical guarantees for simultaneous validity and cumulative improvement lower bounds. Compared to Issue 9's Winner's Curse analytical modeling of shared selection error, the increment is incorporating 'adaptive dependence' itself as a provable error source into a unified framework, with an operational algorithm and a report of false promotion counts over 30 seeds. It has direct governance implications for any self-improvement system relying on fixed benchmarks.
- The Self-Improvement ProblemShould read. It elevates the empirical assertion that 'evaluator hacking is inevitable' to a verifiable formal boundary: any fully computable gate (certifier or searcher) cannot simultaneously satisfy reliability and non-conservatism under unbounded semantic behavior and self-referential candidates, and randomization, hiding, and stateful gates cannot circumvent this; it also proves recovery guarantees under bounded evaluation and finite proposal budgets, but agent drift can still cause true scores to decline round by round. Compared to the empirical approaches of Issue 9's CATCH and Issue 7's Reward Hacking Challenges, the increment is making 'gate-proposal self-reference' a formal theorem and providing drift constructions. Purely theoretical with no runtime data, but as a theoretical anchor for closed-loop constraints, it has direct value for designing governance layers.
Looped Transformers & Latent Reasoning · Issue 5
Looping Is Not Free: Scaling Boundaries of Post-Training Loops and Positive RLVR Results
The most prominent signal this issue is the debate over "whether looping is worth it" being brought to the table—two papers provide evidence from opposite ends.…
Dory1.0-4B: A Looped Normalized-GPTFirst large-scale fully post-trained looped language model (4B, 6T tokens), combining RLVR with a difficulty-aware looping curriculum—routing harder prompts to deeper loops based on prompt pass rates, yielding the largest gains on agentic benchmarks (Terminal-Bench v2.1 from 14.0% to 21.4%). Also provides last-loop KV caching (decoupling KV memory from loop count), self-speculative decoding with different loop counts from the same checkpoint, and a token efficiency analysis showing "deeper loops generate fewer output tokens." Relative to the recipe improvements in Issue 4's Closing the Loop, the increment is treating looping as a routable compute axis in RLVR with a complete production pipeline.- There Is No Free Loop: When Does Looping a Trained Model Pay for Its Inference Compute?A direct cold shower on "post-training-added looping." Proposes a dual-test protocol (isoparametric twins + equal-FLOP best non-looped usage), systematically proving on 0.8B–9B that only 0.8B looping passes both tests, with 4B/9B showing no gains even ignoring compute, and using the benchmark gap between 4B and 9B to rule out ceiling effects. Relative to the positive recipes in Issue 4's Closing the Loop and How to Loop, this is the first scale-dependent negative result with a rigorous control protocol, noting that part of the 0.8B gain comes from the decoding artifact of "completing the answer," requiring budget-forced matching.
- Extending Reasoning in Typed Decision Models via Adaptive Looping (RELOAD)First to prove in typed decision models that damped looping of middle blocks in pretrained models with joint training significantly improves long-chain reasoning (up to +23 points on encoder tasks beyond training chain lengths), and this gain cannot be replaced by matched training compute, more data, or non-looped depth scaling with nearly double the parameters. Relative to Issue 1's SanSi (looped decision models but relying on looped pretraining backbones) and Issue 4's Accordion (fixed-prefix elastic depth), the increment is a retrofit looping recipe for non-looped pretrained models (damped updates + full-budget supervision + request-level controller), proving that the update rule (damping vs. gating) is key to retrofit success.
Code Agent Trajectory Synthesis · Issue 4
Data Scaling Becomes an Allocation Problem: Environment Diversity Outweighs Task Count
The strongest signal this issue: data scaling is shifting from "how much to add" to "where to add." How Data Scales in Agentic Reinforcement Learning is the fir…
SWE-Trail: History-Guided Trajectory Collection for Coding Agent TrainingAdvances trajectory collection from 'post-hoc filtering of independent samples' to 'using history to guide during collection': compiling prior attempts into detectors, deciding which states to invest extra collection in, and executing candidate actions in parallel at the same checkpoint, selecting routes based on repository evidence, but history never enters the agent's context, avoiding train-test mismatch. Compared to TQS and SWE-Prime's post-hoc trajectory filtering in Issue 1, the increment lies in the collection process itself—ablations show that Top-5-of-10 selection alone cannot catch up to H5's gains, and random detection/retention both significantly degrade. On SWE-bench Multilingual and Pro, controlled SFT with multiple student models and fixed data scale attributes gains to trajectories rather than base models.- How Data Scales in Agentic Reinforcement Learning: Laws and Synthesis StrategiesFirst to decompose agentic RL training data into three cardinal axes—task, environment, trajectory—plus reward density control, fitting scaling exponents per axis and giving a Chinchilla-style optimal allocation recipe. Core finding: exponent ranking environment >> task >> trajectory, with composite collapse ratios differing by an order of magnitude (r_E*≈9.4 >> r_T*≈3.4 >> r_τ*≈1.4), mechanistically attributed to groundedness. This provides a quantitative framework for 'where to spend the budget,' directly pricing methods like CodeMidas and SWE-at-Scale; limitation: exponents fitted on math+code, and under SWE coefficients environment and task exponents converge, requiring re-derivation with own costs.
- Scaling Laws for Tool-Use Agents via Controllable Environment SynthesisFirst to provide explicit controllability over three dimensions—tools, tasks, database states—in synthetic environment generation for tool-use agents, and accordingly fit scaling laws related to data composition. Finds that tool diversity and database diversity are more effective than increasing task count when the budget grows, and the law's extrapolation error at 16x budget is far lower than scaling models based solely on task count. Though focused on tool use rather than repo repair, the methodology of 'controlling diversity, fitting scaling, held-out validation' is highly transferable.