SparkDiffusion: Unified Sparse Attention, Distillation, FP8 Speeds Video Generation 265x
Researchers from Peking University, Tsinghua University, and Alibaba propose SparkDiffusion, a unified framework integrating sparse attention, few-step distillation, and FP8 quantization to accelerate video generation. It addresses the 'high-sparsity trap' where extreme sparsity (97%) degrades output quality despite falling training loss, using staged sparse warm-up and trajectory-mixed distillation. The system achieves 265x speedup, generating a 5-second 720P video in 18 seconds, with open weights and code.
Coverage timeline
机器之心机器之心
随着 Sora、Runway 等视频生成模型的快速迭代，AI 视频生成正在变得越来越普及。但生成速度慢、算力门槛高的问题，让大多数开发者和创作者难以将其落地。一段 5 秒 720P 视频，传统方法需要超过 1 小时才能完成扩散生成——这给实时创作、批量生成与消费级硬件部署带来了巨大挑战。 更令人困惑的是，当研究者试图通过极致稀疏化(97%稀疏率)来突破性能瓶颈时，却发现了一个 反常现象 ：训练损失持续下降，生成视频质量却急剧恶化——人物破碎、背景扭曲、时序混乱。 针对这一问题，北京大学、清华大学、阿里巴巴等机构的研究者提出了 SparkDiffusion：首个将稀疏注意力、少步蒸馏与低精度量化整合到统一框架(含开源权重及完整训练代码)的视频生成加速系统 ，推动 DiT 视频生成从「分散优化单个组件」转向「端到端系统化加速」的新范式。 论文标题: SparkDiffusion: Mitigating the High-Sparsity Trap via Staged Sparse Warm-up, Trajectory-Mixed Distillation, and FP8 Quantization 项目主页: https://sparkdiffusion.github.io/ 代码地址: https://github.com/AlibabaResearch/SparkDiffusion 研究机构: 北京大学、清华大学、阿里巴巴、电子科技大学、哈尔滨工业大学 核心发现 "高稀疏陷阱"——当训练越久,质量越差 极致稀疏的反常现象 研究团队发现，当稀疏率从 90%推到 97%时(计算量从 10%降至 3%),训练损失持续下降, 生成视频却开始破碎、扭曲、失去时序连贯性 。更诡异的是,延长训练、增大数据集、扩大补偿分支容量,都无法修复这个问题。 团队将这种现象命名为" 高稀疏陷阱 "(High-Sparsity Trap)： 在极高稀疏率下,逐步监督架构的稀疏视频 DiT 会陷入一种失效模式——尽管单步验证损失持续优化,但终端生成质量却停滞不前甚至退化,而且延长逐步训练也无法实质性恢复终端质量。 问题出在哪?误差累积在高噪声阶段 扩散模型沿着去噪轨迹逐步生成视频。传统的 Flow Matching 等逐步监督方法,只要求模型在某个噪声时刻预测正确的速度,却不直接约束"这一步
Related stories
Alibaba Qwen and OSU Propose TRACE to Speed Up FP4 RL Rollout by 5.4x
Alibaba Qwen and Ohio State University researchers proposed TRACE, a rollout-guided quantization-aware training method for FP4 reinforcement learning of mixture-of-experts (MoE) language models. By using the actual FP4 quantization results from the rollout side to guide rounding on the training side, TRACE reduces discrepancies between training and rollout paths, recovering performance close to BF16 while achieving up to 5.4x decoding throughput improvement in rollout with FP4 weights/activations and FP4 KV cache at 128K output length. The paper is available on arXiv and Hugging Face.
1 reports · 1 sources
Stanford PhD Alex Wang derives Transformer architecture from memory storage perspective
On October 15, 2026, Stanford CS PhD and former Cartesia research scientist Alex Wang published a blog post titled 'You could have invented transformers from memory', proposing that linear layers, MLPs, and self-attention can all be viewed as associative memory stores. The post derives Transformer components from memory capacity considerations, showing that softmax and the 1/d_k scaling factor emerge naturally from interference suppression. It reports experimental capacities: a simple linear memory stores about 64 items, a GLU-style MLP about 2,700, and self-attention with fixed scaling about 1.2 million, sparking discussion in the AI community.
Stanford, Together AI propose Next-Concept Modeling for parallel text generation, 10-31x speedup
Researchers from Stanford University and Together AI introduced Next-Concept Modeling (NCM), a coarse-to-fine parallel text generation paradigm adapted from next-scale prediction used in image generation. In experiments, NCM achieved 10-31x speedup over diffusion language models with comparable or better quality, offering an alternative to both autoregressive and diffusion approaches.
Tsinghua team's MGFlow and Elastic Forcing achieve SOTA one-step image generation and efficient 14B video model post-training
Researchers at Tsinghua University and collaborators introduced distributional training, a method that supervises generation by comparing the overall distribution of generated samples with reference samples rather than assigning unique answers per sample. Their MGFlow method achieved state-of-the-art one-step generation on ImageNet 256×256 (pMF-H, JiT-H: 1.45, 1.64 FID) and outperformed the original 4-step FLUX.2 [klein] 4B model in text-to-image (GenEval 0.900, PickScore 21.98). Elastic Forcing, the video counterpart, scored 84.64 on VBench, surpassing Self Forcing, and post-trained a 14B streaming video model on 8 H200 GPUs in 23.2 hours.
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe