Competitive Learning: Division of Labor in Hybrid Architectures Emerges During Training
Highlights
The biggest signal this issue pushes a long-standing question in this direction—how efficient attention as an optimization prior affects the formation of retrieval heads—for the first time into the training dynamics itself. Competing Learning in Hybrid Architectures is the first to systematically reveal the 'competitive learning' phenomenon during pretraining of hybrid architectures: due to short-range inductive bias, the recurrent layer (KDA) learns retrieval first, thereby suppressing the soft attention layer from learning this function, resulting in suboptimal division of labor; it accordingly proposes the BLD training procedure, which freezes the recurrent layer early to cede the first-mover advantage to the attention layer, providing compute-matched comparisons and RULER/LongPPL degradation curves at 180M–1B scale. This corroborates the 'lazy large window' from Issue 1 and 'Sliding-window beats linear attention' from Issue 4: efficient attention changes 'who learns what first' as an optimization prior, not merely a capacity trade-off. Concurrently, What Attention Recalls and Recurrence Controls uses cache-level interventions (split-prefill, state-swap) to prove that precise retrieval survives only through the attention channel, while language and personality survive only through the recurrent state, providing a causal foundation for the role division in 'competitive learning.' Worth contrasting is How much full attention does a hybrid need?, which reframes the layer-wise ratio phenomenon as 'capacity competition between cheap and expensive paths in hybrid-cost architectures'—this aligns with the mechanism description of competitive learning, but it is only an annotative analysis lacking compute-matched comparisons, so it should be used as an interpretive lens rather than evidence.
On the KV compression side, the substantive advance is that compression itself becomes an end-to-end trainable object for the first time. DCKV turns the prompt KV cache into a differentiable codec: the transform, quantizer, and entropy model are jointly rate-distortion optimized against the frozen language model's output, with a conditional entropy model exploiting prompt tokens and decoded neighbors (checkerboard 2D order for two-pass parallel decoding), achieving at most 1.8–3.6 points loss on Qwen3.5-2B at about 55–100× byte compression, and using 72% fewer bytes than KVTC at the same score (RULER 4K). Notably, its transform has converged to 'a single joint basis + nonlinear residual correction,' without dimensional cliffs. Complementing this is the negative diagnosis from BellowsKV: it is the first to point out that the failure of cross-layer KV sharing lies in the shared object—adjacent layers' key/value caches are nearly orthogonal while hidden states are highly similar, so one should share latent representations derived from hidden states, and under the same budget, dimensional compression is a performance cliff while quantization/eviction can be further compressed. Reading these two together converges the questions of 'what to compress, how to compress, and along which dimension' to a trainable/diagnosable level. An introductory analysis provides spectral measurements of GDN's carried state: truncating to top-48 has almost no effect on loss but degrades some long-context tasks; this subtlety of 'loss-insensitive but long-range-task-sensitive' is exactly the tension to weigh when designing state size.
Hybrid linear attention architectures and the inference side each have worthwhile advances. ParaTTT is the first to advance the parallelization of TTT-like nonlinear memory from block approximation to iterative refinement fixed-point solving of the inner-loop gradient trajectory: each refinement consists of three chunkwise-parallel scans, the exact online recurrence is the fixed point of the iteration, a two-layer variant provides a geometric convergence proof, and decoding requires no iteration—at matched error, it is 32× faster than TTT-MLP, 4.9× faster training for a 340M model, and superior in recall to GDN/Mamba-2. ReplaySSM changes the state representation for SSM decoding from 'writing back the full state each step' to 'checkpoint + recent input buffer,' making speculative decoding rollback a pointer move and verification only compute outputs, and uses double-precision checkpoints to break the accumulation of quantization error in low-precision states—this generalizes the suffix replay idea from Issue 8's 'Just Let Linear States Forget the Distant Past' to three new scenarios: decoding phase, speculative decoding, and low precision. On the inference-side quantization front, there is also new first-principles understanding: LinQuant is the first to systematically characterize the function invariance space of linear recurrences under low-precision prefill, unifying seven transformations (rotation, diagonal rescaling, decay and write factor absorption) into one framework, providing an applicability matrix across six recurrence classes, and proving that the standard floating-point placement (WY form) is not optimal after quantization—this complements the algebraic degrees-of-freedom side of Issue 4's 'Why Gated DeltaNet Survives 4-Bit.'
Community and Dynamics
- FG2-GDN introduces two fine-grained delta-rule learning rates for Gated DeltaNet at token and channel levels, with channel-level gating within a 3:1 linear/MLA hybrid, directly landing on the design axis of hybrid linear attention.
- HALO/HypeNet uses a 2.3B-token distillation pipeline (with new positional encoding HyPE) to convert a pretrained Transformer into a hybrid linear attention model, providing a realistic path for those without compute for from-scratch pretraining to obtain compute-matched hybrid baselines.
- On the trainable block-sparse indexer side, kreasof-ai/foveal-sparse-indexer combines top-p attention quality routing (rather than fixed k) with dual-gradient training (LM output stream + block-level KL to dense attention quality) into a multi-resolution page indexer, a direct increment over fixed-k indexers.
- On the engineering side, prime-rl's DSA modification PR reproduces the standard recipe for two-stage indexer modification (differentiable top-k indexer + KL to dense attention) with training-time freeze stripping, serving as an implementation reference for dense-to-sparse conversion, but without evaluation results, it cannot serve as 'lossless' evidence.
- For newcomers, How Do You Evaluate Long-Context LLMs Beyond Needle-in-a-Haystack? summarizes RULER/NoLiMa/NoCha/HELMET and reiterates that degradation curves are better than saturated synthetic retrieval—the conclusion is community consensus with no new methods, useful as a quick index rather than evidence.
Open Questions
- Competing Learning's validation stops at 1B parameters—does the first-mover advantage of freezing the recurrent layer remain a fix for suboptimal division of labor at larger scales and longer-context CPT, or will it be naturally resolved by competition after sufficient training? How does it jointly determine the freezing timing of BLD with What Attention Recalls's 'precise retrieval survives only on the attention KV side'?
- DCKV's rate-distortion objective targets the next-token distribution of a frozen language model—when the compression object shifts from static prompts to model-generated long CoT (e.g., AbridgeKV's scenario in Issue 10), can the entropy model and checkerboard context still hold? Can its per-layer distortion from a single joint basis be stacked with per-layer quantization without breaking entropy coding gains?
- BellowsKV's dimensional cliff and LinQuant's algebraic invariance loosening both suggest that 'the redundancy in hidden capacity lies not in dimensions but in transformation degrees of freedom'—so how far can the recurrent and attention sides each compress dimensions, and is there a shared low-dimensional representation determined by optimization priors rather than architectural claims?
- ParaTTT's fixed-point iteration converges at 4K context, but decoding requires no iteration—does refinement noise accumulate over time like low-precision quantization after long prefill and expose itself in longer-context recall (it currently tests only up to 32K NIAH)? Does its fixed-point view still hold for nonlinear memory in light of Issue 8's conclusion that 'runtime memory is almost entirely transferred to full-attention KV states'?
Papers in this issue
Competing Learning in hybrid LLMs lets recurrent layers suppress attention's retrieval learning; freezing recurrent layers early gives attention a first-mover advantage, improving long-context and recall performance.
Editor's noteFirst to study the 'division of labor' formation for retrieval in hybrid architectures as training dynamics itself: due to short-range inductive bias, the recurrent layer (KDA) learns retrieval first, suppressing the attention layer from developing this ability, resulting in suboptimal division of labor; accordingly proposes the BLD training procedure, freezing the recurrent layer early to cede the first-mover advantage to the attention layer, validated across KDA/GDN/GLA with compute-matched comparisons and degradation curves. Relative to Issue 5's 'What Attention Recalls' channel-level causal separation, its increment is providing gradient evidence for the competitive learning mechanism and an actionable training intervention—this is a direct negative result of efficient attention as an optimization prior affecting retrieval heads, and anyone doing layer-wise ratios in hybrid architectures should read it.
DCKV is the first end-to-end trainable KV-cache codec for hybrid language models, achieving 55-60x compression with minimal quality loss via joint transform, quantizer, and entropy model training.
Editor's noteFirst to make the prompt KV cache an end-to-end trainable codec: the transform, quantizer, and entropy model are jointly rate-distortion optimized against the frozen language model's output, with a conditional entropy model exploiting prompt tokens and decoded neighbors (checkerboard two-pass), skipping prior heuristic/fixed probability models. On Qwen3.5-2B, it loses AT at about 55–100× byte compression, using 72% fewer bytes than KVTC at the same score; relative to Issue 10's AbridgeKV for model-generated content compression, its increment is providing a trainable rate-distortion trade-off framework aligned with the output distribution, and anyone doing KV compression in hybrid architectures should read it. Limitation: validated only on the Qwen3.5 family.
ParaTTT parallelizes test-time training layers with deep memories by iterative refinement, achieving 32× speedup over TTT-MLP and competitive language modeling without sacrificing accuracy.
Editor's noteFirst to advance the parallelization of TTT-like nonlinear memory from block approximation to iterative refinement fixed-point solving of the inner-loop gradient trajectory: the exact online recurrence is the fixed point of the iteration, a two-layer variant provides a geometric convergence proof, and decoding requires no iteration. At matched error, it is 32× faster than TTT-MLP, 4.9× faster training for a 340M model, and superior in recall to GDN/Mamba-2; relative to Issue 10's Latent Compaction with Test-Time Training's per-document gradient descent, its increment is making the evaluation of nonlinear memory itself a provably convergent parallel bad path, and anyone doing hybrid linear attention or TTT layers should read it.
ReplaySSM represents SSM states as a checkpoint plus recent-input buffer, enabling 1.2-1.47x faster decoding, 1.8x speculative throughput, and 36x lower KL divergence in low-precision serving.
Editor's noteChanges the SSM decoding state from 'writing back the full state each step' to 'checkpoint + recent input buffer,' separating output computation from checkpoint updates: speculative decoding rollback becomes a pointer move, verification only computes outputs, and double-precision checkpoints break the accumulation of quantization error in low-precision states. Provides a vLLM implementation and degradation evidence for 128K long generation; relative to Issue 8's 'Just Let Linear States Forget the Distant Past' suffix replay, its increment is generalizing 'cache inputs rather than states' to the decoding-phase state representation itself, systematically addressing two previously unhandled issues: speculative rollback and low-precision error propagation.
LinQuant's seven functional invariances—basis changes and decay/write absorptions—enable low-precision linear recurrent inference, improving perplexity over SSDi8 on Mamba-2 and Nemotron-H without added latency.
Editor's noteFirst to systematically characterize the function invariance space of linear recurrences under low-precision prefill: seven transformations (rotation, diagonal rescaling, decay and write factor absorption) unified into one framework with an applicability matrix across six classes (LA/Mamba-2/GLA/DN/GDN/KDA). Relative to Issue 4's 'Why Gated DeltaNet Survives 4-Bit Quantization' mechanism decomposition, its increment is making the 'algebraic degrees of freedom of the recurrence operator itself' an explicit design axis for quantization robustness, and proving that the standard floating-point placement (WY form) is not optimal after quantization—a must-read for those doing inference-side quantization in hybrid architectures. Limitation: evaluation is mainly perplexity-based, with production validation only covering Mamba-2 geometry.
KV-SURGEON couples cache-entry deletion with closed-form compensation for surviving entries, improving LLM long-context accuracy by 3.0 points over prior state-of-the-art at 50x compaction.
Editor's noteAdvances offline KV compression's 'selection' and 'compensation' from select-then-compensate to joint optimization: using a softmax-normalized second-order Taylor quadratic proxy to allow closed-form compensation of surviving entries after each deletion, with ablations proving compensation itself is a source of gain (dropping it reduces from 66.9 to 63.9). Provides compute-matched cost-accuracy frontiers and real degradation curves under 4B–30B, 5×–100× compression; relative to Issue 4's Still amortized synthesis, its increment is providing a closed-form OBS-style importance form for compensation-aware selection, a methodologically rigorous increment in the selection-reconstruction synergy direction.
Sparse attention should optimize matrix-product approximation error, not entrywise magnitude, and MASA's plug-in scoring correction improves all eight matched benchmarks across three sparse attention frameworks.
Editor's noteReframes block-sparse attention's selection criterion from 'picking the largest from a bag of numbers' to a closed-form score for matrix product approximation error (MASA), with a proof of equivalence to attention quality ranking in the identity Gram special case, arguing that non-identity VVᵀ under long context makes quality ranking suboptimal. Relative to Issue 11's Certified Sparse Attention's provable error and Issue 4's Faster Than Flash's top-δ, its increment is a plug-and-play closed-form score that can directly replace the ranking rule in sparse attention frameworks; its theory has potential explanatory power for retrieval head formation, and it provides degradation curves on RULER/InfiniteBench/LongBench in matched settings.
BellowsKV compresses KV caches 13-20x with near-dense accuracy by sharing a full-rank latent across layers and buying compression from bits rather than rank reduction.
Editor's noteFirst to systematically point out that the failure of cross-layer KV sharing lies in the choice of shared object: adjacent layers' key/value caches are nearly orthogonal while hidden states are highly similar, so one should share latent representations derived from hidden states; accordingly provides a training-free closed-form optimal projection construction, and proves that under the same budget, dimensional compression is a performance cliff while quantization/eviction can be further compressed, with 13–14× compression maintaining 97–98% of full-cache accuracy. Relative to Issue 9's VFold symmetry folding and Issue 6's DeepSeek-V4.1-Flash pattern sharing, its increment is a negative result diagnosing 'what to share' as the primary failure cause. Limitation: evaluation is mainly LongBench/agentic, without long-context retrieval degradation curves.
MERLIN improves multi-state linear attention by using both mean and variance of segment keys for aggregation, yielding consistent gains of up to 13.5% on long-context benchmarks with 4x less memory.
Editor's noteAdvances segment-level aggregation in multi-state linear attention from first-order means to second-order moments (mean + variance) routing: storing first and second moments of each segment's keys, queries recover aligned mean and variance, with learnable gating mixing as aggregation weights, and provides a theoretical connection to softmax per-segment grouped cumulant expansion. Relative to Issue 11's HLA query-dependent blockwise dynamic mixing, its increment is explicitly using aligned variance to distinguish sharp intra-segment alignment from uniform weak alignment, with compute-matched comparisons and position-controlled retrieval analysis for GDN/Mamba-2; limitation: evaluation only up to 16K, without compute-matched comparison to full attention.
HyScale accelerates long-context sparse-attention training 2x by treating selection as discrete indices, enabling on-the-fly top-k and replay without recomputing scores.
Editor's noteFirst to treat sparse attention's selected outputs as discrete indices rather than continuous activations, proposing HyTopK (optimistic threshold sampling + host-free recovery + deterministic selection) and selection replay (re-scoring only the selected k entries during recomputation), with decoupled CP communication and locality-aware kernels, reporting end-to-end speedups at 1M context and generalization across DSA/CSA/QSA. Relative to Issue 6's DeepSeek-V4.1-Flash CSA2 and Issue 9's SPIN indexer acceleration, its increment is making the selection bottleneck on the pretraining side (materializing score tables and repeated selection) a system optimization target; worth reading for those doing trainable sparse attention training, but without long-context degradation curves.
