Weekly Gallery
Latest edition
Sep 12 – 19, 2026
Directions: 7 · Updated this issue: 6 · Picks: 44
AI for Formal Math · Issue 6
Robustness of Autoformalization as a New Evaluation Axis
The main theme of this issue is that "evaluation of autoformalization is moving from correctness to robustness and faithfulness." The 56% misformalization rate …
Evaluating the Robustness of Proof Autoformalization in Lean 4The first systematic evaluation of proof autoformalization robustness, categorizing perturbations into global (style rewriting) and local (counterfactual edits), measuring correctness stability and faithfulness respectively. Compared to FaithformBench in Issue 4 (which perturbs reasoning steps to evaluate CoT autoformalization faithfulness), its increment lies in focusing on proof-level counterfactual edits and quantifying the 'silent regression' failure mode—models quietly revert local edits to the original rather than faithfully reflecting them, with an average faithfulness rate of only 6.3% across seven models on numeric/symbolic edits. Benchmarks and code are public, adding a robustness dimension beyond correctness to autoformalization evaluation.- Lean-GAP: A Dataset of Formalized Graduate Algebra ProblemsA formalized dataset of 430 graduate-level algebra problems from Dummit & Foote's 'Abstract Algebra', with two-stage human review ensuring every statement has mathematician sign-off. Compared to StochBench (stochastic processes) and AxQM (quantum mechanics) in Issue 5, its increment lies in focusing on the algebra domain and systematically analyzing autoformalization difficulties such as 'answer-embedded formalization', also documenting cases where AI as a second reader found errors in the textbook's original problems. Currently about 20% complete, semantic evaluation relies on LLM judges, so model comparisons should be treated as relative signals.
- Teaching Vampire New Tricks: An Experimental Study of Neural Clause SelectionThe first systematic evaluation of neural clause selection in saturation ATP (Vampire) across multiple ITP-derived benchmarks (Mizar40, Isabelle, CoqHammer), examining cross-domain transfer and combinatorial strategy effects. Key findings: cross-benchmark transfer fails (even below the unguided default strategy) but joint training recovers performance, and neural guidance shows diminishing marginal returns at combinatorial scale—this is the first work evaluating neural guidance at the combinatorial level. Directly valuable for designing generalizable neural guidance systems, but the focus is on ATP rather than the Lean 4 ecosystem.
Pretraining Data · Issue 6
Fragility of Mid-Training Priors
The most notable signal this issue comes from stress tests on mid-training itself. One work systematically examines the robustness of alignment mid-training (AM…
Stress-testing Alignment MidtrainingFirst systematic stress test of alignment midtraining robustness at realistic scales (10B–110B parameters, 10M–1B tokens): 2% conflicting fine-tuning data can override a 190M-token midtraining prior (about 1:4200 token ratio), with weak generalization to rules not demonstrated and effects varying significantly by post-training method (SFT vs RL). Relative to Issue 5's *Knowledge Distillation During Mid-Training*, which treats mid-training as a supervision source, its increment is making mid-training itself the object of study, providing strong negative evidence that "priors can be overridden by small amounts of downstream data," directly warning against assumptions that rely on mid-training as a data mixing lever.- FastMix: Fast Data Mixture Optimization via Gradient DescentReparameterizes data mixing optimization as a differentiable bi-level optimization: mixing ratios are equivalent to per-source loss weights under uniform sampling, requiring only a single proxy model to jointly optimize mixing coefficients and model parameters via gradient descent, reducing search costs by 55–550x compared to RegMix/CLIMB, while covering both pretraining and SFT stages. Relative to Issue 3's RegMix-D dynamic mixing and Issue 4's CausalMix causal estimation, its increment is embedding mixing ratios directly into a differentiable training objective for end-to-end gradient optimization; limitations include small proxy scale and reliance on base loss and post-SFT benchmarks as criteria.
- AutoData: Agentic Search for Pre-training Data SelectionAdvances pretraining data selection from "optimizing weights over fixed domains" to "searching over the program space of executable selection algorithms": an agent iteratively proposes and refines scoring/layering/random selection rules on document features (lexical statistics, category labels, perplexity), with recipes discovered on a 125M proxy transferring without retuning to 1.3B and improving downstream CORE. Relative to Issue 3's RegMix-D and Issue 4's CausalMix, its increment is moving from weight optimization to algorithm space search, automatically discovering feature interactions; limitations include a single corpus (ClimbMix) and CORE gains not reaching statistical significance.
MoE & Sparse Experts · Issue 6
The Cost of Routing Skew Relocated: From Memory Peaks to Exploration Gains
The main theme of this issue is that "the cost of routing skew is being relocated": three independent works on the training side, RL side, and inference side, f…
Flattening Every Memory Peak in Long-Context Mixture-of-Experts TrainingMust-read. For the first time, unifies the four memory peaks of MoE long-context training (expert distribution, vocabulary projection, gradient checkpoint boundaries, optimizer states) into a "bounded streaming operator" framework, where each operator provides a fixed working set guarantee related to launch configuration. The core increment, PipelinedLLEP, decouples the receiver buffer size from the routing matrix via a source-side token budget, making routing skew only stretch scheduling rather than amplify memory, and uses strided chunking to eliminate local imbalance introduced by chunking. Compared to the negative result of Cacheable by Design in Issue 5 and the online placement of Director in Issue 4, this paper dissolves the cost of routing skew from the memory side while keeping routing decisions unchanged, training with 1M context on 120B–667B models and reporting complete routing statistics and dense baselines.- Expert-Space Exploration in MoE Reinforcement Learning (ESRL)Must-read. For the first time, treats the expert routing space itself as an explicit exploration dimension in RL: during rollout, it adaptively perturbs logits based on router entropy, anchors on high-confidence experts, and samples from a candidate pool, while recording expert paths and replaying them during policy optimization to maintain training-inference consistency. Compared to PR² and ReLibra in Issue 2, which focus on "eliminating mismatch," this paper combines perturbation-replay into a controlled exploration mechanism, improving Pass@1 by 3.2 points over GRPO on Qwen3-30B-A3B, and covering three routing structures: top-K, top-1, and shared experts, with analysis of expert utilization and training dynamics.
- ACE: Adaptive Calibration-Free Expert Skipping for MoE-based LLMsWorth reading. For the first time, advances the contribution estimation for expert skipping to a purely parameter-driven dual-perspective conservative intersection criterion (global spectral proxy GSP + router-conditioned refinement RCR), requiring no calibration data or additional training, and at a 50% skip rate reduces PPL by 7.96% on Qwen3.6-35B compared to the strongest baseline. Compared to the global k-reduction of k1/k2 in Training-Free Halving from Issue 5, this paper skips low-contribution slots token-wise while preserving top-1, using the spectral norm upper bound of SwiGLU multiplicative coupling to provide provable conservatism, offering a new alternative path for training-inference routing consistency in inference-time sparsification.
Efficient Sequence Modeling · Issue 6
Cross-Layer Reuse and Spectral Reparameterization: Efficient Sequence Modeling Toward Serving
The strongest signal this issue is that cross-layer KV and index reuse has moved from research prototypes to production-grade instantiations. DeepSeek-V4.1-Flas…
DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache CompressionA representative work advancing cross-layer KV and index reuse to production-grade instantiation. CSA2 uses three static modes—Full/Reindex/Reuse—to simultaneously share the main KV, indexer K, and Top-K indices, paired with a Hierarchical Sparse Indexer that decouples the per-query scoring cost of deeper indexers from context length, compressing global KV to 890 bytes per token at 552B parameters and 1M context. Relative to Issue 1's 'You Only Index Once' cross-layer index sharing, its increment is the first joint coverage of the three multiplicative dimensions of KV compression and the decoupling of cache sharing from index reuse. Note that its efficiency claims are mostly system-level rather than compute-matched scientific comparisons, and long-context degradation evidence relies mainly on LongBench-V2.- SAS: Simple Attention Sparsification via End-to-End Optimization of Context RankingAbandons layer-wise dense attention distillation, injecting the selector's continuous scores into the attention softmax via a log-space gate, allowing the LM loss to backpropagate directly to the selector, and identifies four key design choices: gate position, activation normalization, ranking preservation, and training scope. Relative to Issue 5's 'HiLS' differentiable selection, its increment is explicitly comparing the ranking misalignment of distillation supervision and finding that end-to-end training reduces per-layer attention quality coverage but increases overlap with cross-layer and full-attention oracles. Provides compute-matched comparisons across reasoning, long-context, and agentic tasks, with the largest gains under tight budgets.
- SpectralShift: Effective Context Window Extension of Gated DeltaNet via Spectral ReparameterizationThe first to formalize Gated DeltaNet's long-context extension as spectral reparameterization of the transition matrix: scaling the deviation of the alpha projection around the global mean to widen the slow spectral band, paired with learning-rate scaling specific to the alpha projection, turning context extension from 'direct CPT + positional encoding scaling' into a spectral initialization plus training-time spectral preservation problem. Relative to Issue 5's 'Taylor-Calibrate' recurrent parameter initialization, its increment is providing causal evidence linking slow spectral band width to write alignment (E_write). Provides compute-matched comparisons on 1.5B MoE and RULER degradation curves from 8K to 128K.
Coding Agents · Issue 6
Leaderboard Rankings Indistinguishable, Self-Evolution Vulnerable to Poisoning: Benchmark Validity Tightened on Two Fronts
The strongest signal this issue is the advancement of benchmark validity from "audit methodology" to two new fronts: "leaderboard readability" and "self-evoluti…
Reflections on Trusting Trust, Revisited: Contaminating Self-Modifying AI Coding Agents with Poisoned BenchmarksThe first empirical work to systematically transplant Thompson's "Trusting Trust" compiler backdoor attack to self-modifying coding agents: an attacker only poisons the benchmark (without modifying the agent or the model) to induce the self-evolving harness to persistently write vulnerable code on held-out neutral tasks (Hyperagents under Sonnet 4.5 self-evolves instructions that disable HTTPS certificate verification, with a neutral task vulnerability rate of 30/30). Compared to Issue 3's EVOMAL skill self-poisoning, the attack vector is the benchmark used by the self-evolution process itself, validated across three harnesses (DGM, SICA, Hyperagents), with a sufficient-conditions checklist for attack success, and proof that contamination cannot be cleared even after continued evolution on clean benchmarks and CWEval. For any researcher building or auditing self-evolving harnesses, its attack conditions checklist and decontamination design are directly usable security constraints.- Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure InsteadUses two new constructs—comparison-set-relative effective size n_eff(S) and the nesting coefficient of the relative score-implied baseline—to audit leaderboard resolution, proving that none of the top 29 adjacent entries in SWE-bench Verified can be precisely distinguished by McNemar (0/29), the frontier solution set's nesting is 0.935 far exceeding the random baseline of 0.774, and the scaffold effect (up to 29.8pp) exceeds the top-30 spread (8.8pp). Compared to Issue 3's ABA and Issue 1's HackDetect, which focused on task quality, it shifts the audit target to "whether rankings are distinguishable" itself, providing a five-step reproducible protocol to determine whether reported ranking differences are statistically significant. Essential reading for anyone comparing coding agents via leaderboards.
- When the Reward Suite Is Leaky: A Preregistered Causal Contrast of Natural Verifier False Positives in RLVRThe first experiment to distinguish natural verifier false positives (persistent per-task asymmetric FPs) from random label noise and directly measure their impact on RLVR training using a pre-registered two-arm causal comparison (leaky vs hardened, only the reward suite changes). It does not implant any hack vector but uses natural vulnerabilities in real deployment suites (MBPP vs MBPP+), providing causal evidence that "static audit can predict FP distribution (Spearman 0.80), about half of the rewards are paid to genuinely incorrect code, and the mechanism is selection rather than learning." The limitation is small scale (1-1.5B, 400 steps), but as a measurement-layer contribution it is solid, directly hitting the reward hacking and benchmark validity themes.
Recursive Self-Improvement · Issue 6
Self-Improvement Rate Enters the Era of Laboratory Disclosure
The main theme of this issue is the advancement of measuring the 'self-improvement rate' from paper-level methodology to laboratory-level disclosure. Anthropic …
ScientistTwo: Pioneering the Human Knowledge Frontier with Autonomous AIShould read. It advances the AI scientist closed loop to 'simulated peer review-rebuttal' as a core component: the Meta-Review Agent drives deep idea refinement based on review feedback, and the Rebuttal Agent proactively designs and executes supplementary experiments to respond to reviewer comments, forming a closed loop. Compared to the 'engineering meets standards but research judgment does not' case in Issue 5's shadow evaluation, this provides larger-scale (107 top-conference paper tasks) closed-loop evidence, reporting success rates, average relative improvements, and failure rates (21/107 unsuccessful). Limitation: both review and rebuttal are simulated, not real peer review, so one must be wary of the separation between 'reviewer satisfaction' and 'actual research progress.'- Benchmarking Data-Centric Research for Recursive Self-Improvement (RSIBench-Data)Should read. It isolates 'data-centric research capability' from end-to-end post-training benchmarks, fixing the training/serving/evaluation stack, and specifically measures the agent's ability to propose hypotheses, synthesize data, validate, and revise strategies in a closed loop. Compared to S3Gym's coupled self-test-self-evaluate-self-improve measurement in Issue 5, the increment is making 'data strategy evolution' an independent measurement object and reporting the 'discovery-reliability gap': 58.33% of settings iteratively improve the first candidate, but 78.26% of searches end with final attempts below their peak—this is a rare quantification of whether 'improvement is monotonic.' It aligns with the preference for reproducible closed loops and failure rate reporting.
- Dream-RSI: Recursive Self-Improvement through Evolving WorldsShould read. It reframes 'completed discovery history' as a replayable simulator, turning meta-level improvements to exploration strategies from expensive online rollouts into zero-execution-cost offline 'dreaming' evaluation—this is an increment over Issue 5's Evolution Fine-Tuning (cross-task meta-learning during training): elevating history from a static context to an interactive simulated environment, explicitly separating online exploration from offline policy improvement. It reports evolutionary behavior where exploration effort first decreases then increases across recursive rounds, and distinguishes invalid run criteria such as fail_class, n_valid==0, and repairable. Limitation: replay evaluation depends on historical tree coverage, potentially underestimating performance on unseen branches.