# xHC: Expanded Hyper-Connections > xHC scales residual streams to N=16 with 1.19x compute efficiency, gaining +4.0 average points on 18B MoE models. - **Source:** [arXiv](https://arxiv.org/abs/2607.14530) - **Published:** 2026-07-21 - **Permalink:** https://picx.dev/p/37GlhA - **Whiteboard:** https://picx.dev/p/37GlhA/image ## Summary Here is a comprehensive summary of the academic paper "xHC: Expanded Hyper-Connections". ## Summary (Overview) - **Problem**: Existing Hyper-Connection (HC) methods, which expand the residual stream into \(N\) parallel streams, saturate beyond \(N=4\). Scaling to larger \(N\) yields diminishing returns while rapidly increasing FLOPs and memory traffic. - **Bottlenecks**: The authors diagnose two root causes: (1) an **information bottleneck** – each layer injects only a single write-back signal, limiting the diversity of stream histories; (2) a **cost bottleneck** – generating the dense residual mixing matrix scales as \(O(N^3 C)\), making large \(N\) expensive. - **Proposed Solution**: xHC (**E**xpanded **H**yper-**C**onnections) overcomes both bottlenecks with two coordinated designs: **temporal feature augmentation** (enriching the write-back signal with multi-scale causal convolutions) and a **sparse residual-stream architecture** (updating only \(k=4\) of \(N=16\) streams while keeping dense read access to all streams). - **Key Results**: On 18B and 28B MoE models, xHC significantly outperforms mHC and vanilla baselines in downstream tasks (e.g., +4.0 average points over mHC at 18B). Scaling-law experiments show that to reach the same loss, vanilla requires \(1.50\times\) and mHC requires \(1.19\times\) the compute of xHC. - **Practical Efficiency**: xHC-Flash reduces per-sublayer memory traffic from \(73.5C\) to \(40C\) (comparable to mHC at \(N=4\)) while retaining most performance gains. ## Introduction and Theoretical Foundation **Background**: Residual connections in Transformers typically maintain a single identity stream across layers, offering no learnable control over cross-layer information flow. Hyper-Connections (HC) [55] replace this with \(N\) parallel residual streams governed by learnable mixing matrices. Manifold-Constrained HC (mHC) [50] stabilizes this formulation at scale via Sinkhorn normalization, achieving strong results at \(N=4\). However, scaling beyond \(N=4\) in mHC leads to diminishing returns: on a 2.5B MoE model, increasing \(N\) from 4 to 16 reduces loss by only 0.006 while increasing training FLOPs by 32%. **Theoretical Basis**: The authors identify two bottlenecks that prevent larger \(N\) from translating into useful capacity: 1. **Information Supply Bottleneck**: Each stream updates via \(\Delta x_{l,i} = h_{l,i}^{\text{post}} \cdot \text{out}\), where the newly injected information is spanned by only one vector (the layer output). As \(N\) grows, additional streams cannot form non-redundant histories because they draw from the same single write-back component. 2. **Cost Bottleneck**: Generating the residual mixing matrix \(H_l^{\text{res}} \in \mathbb{R}^{N \times N}\) from an \(NC\)-dimensional state requires predicting \(N^2\) coefficients, leading to an \(O(N^3 C)\) projection cost. This makes large-\(N\) expansion increasingly expensive even when performance gains are marginal. **Goal**: To make the expansion rate \(N\) a practical scaling axis orthogonal to width and depth by addressing both bottlenecks. ## Methodology The proposed xHC modifies the HC framework (Eq. 1) with two key innovations. ### 1. Temporal Feature Augmentation (Enriching the Write-Back Basis) To address the information bottleneck, xHC enriches the single layer output with local contextual features from neighboring tokens using lightweight causal depthwise 1D convolutions. After the MLP sublayer, the layer output \(out \in \mathbb{R}^{S \times C}\) is augmented with \(r\) convolution branches: \[ out_{\text{aug}} = [out; \text{DWConv}_{\kappa_1}(out); \dots; \text{DWConv}_{\kappa_r}(out)] \in \mathbb{R}^{S \times K_r \times C},\quad K_r = r+1. \] To prevent correlated components from amplifying the write-back signal uncontrollably, modified Gram–Schmidt orthogonalization is applied over the \(K_r\) components: \[ v_{j+1} = g_j - \sum_{i=1}^{j} \frac{\langle g_j, v_i \rangle}{\langle v_i, v_i \rangle} v_i, \] where \(g_j = \text{DWConv}_{\kappa_j}(out)\) and \(v_1 = out\). The orthogonalized components \(v_1, \dots, v_{K_r}\) are then used for write-back. Only MLP sublayers use augmentation; attention sublayers do not. ### 2. Sparse Residual-Stream Architecture (Reducing Cost) To address the cost bottleneck, xHC activates only \(k\) out of \(N\) streams for residual mixing and write-back, while keeping the read path dense. - **Stream Routing**: A router computes per-stream importance scores \(s = \sigma(\tilde{x}_l W_r) \in \mathbb{R}^N\), where \(W_r \in \mathbb{R}^{NC \times N}\). A fixed-plus-routed scheme selects \(k\) active streams: \(m\) fixed streams are always active, and the remaining \(k-m\) are selected by TopK routing over non-fixed streams. Routing weights \(p_j\) are 1 for fixed streams and the sigmoid score for routed streams. - **Dense Read**: The pre-mapping \(H_l^{\text{pre}} \in \mathbb{R}^{1 \times N}\) is generated from the full \(N\)-stream state, and the sublayer input is computed as \(input_l = \sum_{i=1}^N h_{l,i}^{\text{pre}} \cdot x_{l,i}\). This preserves cross-layer information flow even though only \(k\) streams are updated. - **Sparse Residual Update**: Only the active state \(X_{\text{active}} \in \mathbb{R}^{k \times C}\) is used to generate the residual and post-mappings: \[ H_l^{\text{res}} = \text{SK}(f_{\text{res}}(X_{\text{active}})) \in \mathbb{R}^{k \times k},\quad H_l^{\text{post}} = f_{\text{post}}(X_{\text{active}}) \in \mathbb{R}^{k \times K_r}. \] The dominant mapping generation cost is reduced from \(O(N^3 C)\) to \(O(k^3 C)\). The Sinkhorn constraint is applied only on the active stream subset. The write-back uses the augmented components: \[ \Delta X_{\text{active},j} = p_j \sum_{r=1}^{K_r} H_{l,j,r}^{\text{post}} \cdot out_{\text{aug},r},\quad j=1,\dots,k. \] Updated streams are scattered back; inactive streams carry forward unchanged. The full forward pass is summarized in Algorithm 1 of the paper. ## Empirical Validation / Results ### 1. Main Results (18B and 28B MoE) Table 1: Downstream evaluation on 18B and 28B MoE models. All numbers are scores (%; higher is better). mHC uses \(N=4\), xHC uses \(N=16, k=4\); comparable training FLOPs. | Benchmark | 18B Vanilla | 18B mHC | 18B xHC | 28B Vanilla | 28B mHC | 28B xHC | |--------------------------|-------------|---------|---------|-------------|---------|---------| | Language Understanding & Knowledge | | | | | | | | MMLU | 48.9 | 54.7 | 57.2 | 54.6 | 56.8 | 60.5 | | MMLU-Pro | 21.1 | 27.4 | 29.7 | 30.1 | 34.9 | 36.0 | | MMLU-Redux | 46.4 | 49.9 | 52.8 | 50.6 | 53.9 | 56.4 | | Reasoning & Mathematics | | | | | | | | BBH | 32.4 | 33.7 | 39.5 | 41.7 | 43.6 | 43.4 | | CommonsenseQA | 54.6 | 56.6 | 60.9 | 60.5 | 63.9 | 69.6 | | ARC-Challenge | 55.7 | 66.3 | 72.2 | 70.8 | 74.9 | 77.7 | | GSM8K | 37.7 | 44.5 | 48.4 | 50.3 | 56.3 | 59.2 | | Code | | | | | | | | HumanEval | 25.6 | 23.2 | 29.3 | 27.4 | 26.8 | 31.1 | | LCBench | 9.9 | 12.2 | 14.6 | 15.1 | 14.8 | 17.9 | | Chinese | | | | | | | | CMMLU | 42.7 | 47.6 | 50.4 | 47.6 | 50.1 | 53.4 | | CEval | 44.5 | 48.8 | 52.4 | 50.2 | 51.2 | 54.9 | | C3 | 67.1 | 72.7 | 78.3 | 75.2 | 78.7 | 82.5 | | **Average** | **40.6** | **44.8**| **48.8**| **47.8** | **50.5**| **53.6**| xHC achieves substantial gains at both scales, with the largest improvements in reasoning, code, and Chinese benchmarks. ### 2. Scaling Laws The loss as a function of training compute \(C\) is fitted with a shifted power law: \[ L(C) = A C^{-\alpha} + E, \quad E = 0.72. \] xHC traces a consistently lower loss curve. To reach the same loss, vanilla requires \(1.50\times\) and mHC requires \(1.19\times\) the compute of xHC. ### 3. Ablation Study (10B MoE, Validation Loss on Pile) Table 2: Ablation study. Parentheses show extra training FLOPs over vanilla baseline. | Method | N | Temp Aug. | Sparse | D. Read | k | Fixed | Router | Val. Loss ↓ | |------------------------------------------|-----|-----------|--------|---------|-----|-------|----------|-------------| | (1) Vanilla baseline | - | - | - | - | - | - | - | 2.029 | | (2) mHC (+0.6%) | 4 | - | - | - | - | - | - | 2.004 | | (3) mHC (+18.8%) | 16 | - | - | - | - | - | - | 1.998 | | (4) mHC w/ Temp Aug (+20.1%) | 16 | ✓ | - | - | - | - | - | 1.984 | | (5) xHC (+3.3%) | 16 | ✓ | ✓ | ✓ | 4 | 2 | Sigmoid | 1.983 | | (6) w/o D.Read & Fixed Streams | 16 | ✓ | ✓ | ✗ | 4 | 0 | Sigmoid | 1.997 | | (7) w/o Dense Read | 16 | ✓ | ✓ | ✗ | 4 | 2 | Sigmoid | 1.985 | | (8) w/o Fixed Streams | 16 | ✓ | ✓ | ✓ | 4 | 0 | Sigmoid | 1.986 | | (9) \(k=2\) | 16 | ✓ | ✓ | ✓ | 2 | 1 | Sigmoid | 1.991 | | (10) \(k=8\) | 16 | ✓ | ✓ | ✓ | 8 | 2 | Sigmoid | 1.982 | | (11) Softmax Router | 16 | ✓ | ✓ | ✓ | 4 | 2 | Softmax | 1.988 | Key findings: (4) vs (3) shows temporal augmentation improves loss at large \(N\). (5) preserves that gain while reducing FLOPs overhead by sparse architecture. Removing dense read or fixed streams degrades performance, confirming their importance. Sigmoid routing outperforms softmax. ### 4. Expansion Rate Scaling (2.5B MoE, Figure 1) xHC shows consistent loss improvement from \(N=2\) to \(N=16\) with modest FLOPs increase. Over \(N=4\) to \(N=16\), xHC reduces loss by 0.012 with only 4% extra FLOPs, while mHC reduces loss by only 0.006 with 32% extra FLOPs. ### 5. Compatibility with Muon Optimizer (Table 3) xHC also provides substantial gains over the Muon baseline (average 49.9 vs. 43.1), demonstrating effectiveness beyond AdamW. Gram–Schmidt orthogonalization is removed for Muon; Muon is applied only to backbone parameters. ## Theoretical and Practical Implications - **Theoretical**: The paper provides a clear diagnosis of why scaling residual-stream expansion saturates (information and cost bottlenecks). The proposed solutions are principled: temporal feature augmentation directly addresses the information bottleneck by providing a richer write-back basis, while sparse residual updates address the cost bottleneck by reducing the dominant \(O(N^3 C)\) generator cost to \(O(k^3 C)\). The two designs are synergistic: larger \(N\) makes augmentation more useful, and sparsity makes larger \(N\) affordable. - **Practical**: xHC makes residual-stream expansion a practical scaling axis for LLM pre-training. The gains are consistent across model scales (18B, 28B), compute budgets (scaling laws), and optimizers (AdamW, Muon). The memory traffic analysis and xHC-Flash variant (Table 4) show that the method can be deployed with memory traffic comparable to mHC at \(N=4\) while retaining most gains. The fused kernel implementation further reduces runtime overhead. Table 4: Per-token memory access of residual-stream operations (sublayer function I/O excluded). Totals averaged per sublayer. | Operation | mHC (N=16) | xHC (N=16, k=4) | xHC-Flash-4sub | |---------------------------------|------------|------------------|----------------| | Total I/O (per sublayer avg.) | 130C | 73.5C | 40C | | Reference: mHC (N=4) | 34C | – | – | ## Conclusion The paper introduces xHC, the first Hyper-Connection method to achieve meaningful expansion beyond \(N=4\). By combining temporal feature augmentation and a sparse residual-stream architecture, xHC addresses the information and cost bottlenecks of scaling. Extensive experiments on 18B and 28B MoE models, scaling laws, ablations, and Muon compatibility demonstrate that xHC provides consistent improvements in performance and compute efficiency. The introduction of xHC-Flash and fused kernels makes large- --- _Markdown view of https://picx.dev/p/37GlhA, served by PicX — AI-generated visual whiteboard summaries of research papers._