Mutian He @mutiann_

@flyingpetal472 什么理论推导哇（好奇

@yzhang_cs @nathancgy4 @Xinyu2ML @Yulun_Du It is certainly THE Sputnik moment for a lot more than OSS!

hybrid linear attn is scalable!!

Kimi.ai @Kimi_Moonshot

2 months ago

Introducing Kimi K3: Open Frontier Intelligence 🔹 2.8 Trillion Parameters, 1 Million Context, Native Multimodal 🔹 Kimi Delta Attention enables up to 6.3x faster decoding in million-token contexts 🔹 Attention Residuals deliver ~25% higher training efficiency at <2% additional

keyboard_arrow_left Previous keyboard_arrow_right Next

2K 8K 57K 25.5M 23K

Today, we are introducing Inkling. Inkling reasons efficiently across text, image, and audio modalities. We are making the full weights available. thinkingmachines.ai/news/introduci… Available today for fine-tuning on Tinker. Play with it in the Inkling Playground. 🧵

To beast Oblivionis we need the many-headed beast...

Mutian He @mutiann_

5 months ago

Excited to share our work on hybrid linear-sparse attention at ICLR MemAgents (Rm 205, 2:50pm, 4.27)! Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Eviction arxiv.org/abs/2510.20787 Code: mutiann.github.io/papers/laLTE

6 0 4 141 1

TLDR: With a Learnable Token Eviction CNN to drop KVs from memory based on past&future context, a hybrid Gated DeltaNet+sparse attention is as fast as pure GDN and hybrid GDN+SWA, but forgets much less, and performs close to std transformer and GDN+NSA (which uses full KV cache)

Excited to share our work on hybrid linear-sparse attention at ICLR MemAgents (Rm 205, 2:50pm, 4.27)! Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Eviction arxiv.org/abs/2510.20787 Code: mutiann.github.io/papers/laLTE

TLDR: With a Learnable Token Eviction CNN to drop KVs from memory based on past&future context, a hybrid Gated DeltaNet+sparse attention is as fast as pure GDN and hybrid GDN+SWA, but forgets much less, and performs close to std transformer and GDN+NSA (which uses full KV cache)

@LLMenjoyer @kamesh_ai @leonlufkin Agree. Hybrid attn in a more ''integrated'' way so that the linear attn layers interact with/guide sparse attn should be a really promising direction! BTW we've also built a variant of the learned router: x.com/i/status/20223…

Mutian He @mutiann_

7 months ago

I always firmly believe that hybrid attn (w/ linear attn for locality & sparse attn for retrieval) IS the next gen LLM architecture. It's so exciting to see such a model at a seriously large scale! Also check our previous work on this topic! arxiv.org/abs/2510.20787

keyboard_arrow_left Previous keyboard_arrow_right Next

0 0 0 96 0

immensely proud to share 2 of my bros' work, Hybrid Associative Memories by @kamesh_ai and @leonlufkin It's basically SSM and Attention merged into one layer, where the attention is also sparse like DSA. Except you pretrain with the sparsity. It's lowkirk based (1/N) 👇

@OpenBMB I always firmly believe that hybrid attn (w/ linear attn for locality & sparse attn for retrieval) IS the next gen LLM architecture. It's so exciting to see such a model at a seriously large scale! Also check our previous work on this topic! arxiv.org/abs/2510.20787

keyboard_arrow_left Previous keyboard_arrow_right Next

I always firmly believe that hybrid attn (w/ linear attn for locality & sparse attn for retrieval) IS the next gen LLM architecture. It's so exciting to see such a model at a seriously large scale! Also check our previous work on this topic! arxiv.org/abs/2510.20787

keyboard_arrow_left Previous keyboard_arrow_right Next

OpenBMB @OpenBMB

7 months ago

🚀 We are excited to unveil #MiniCPM-SALA, the industry's first 9B model trained with Sparse-Linear Attention (SALA) hybrid architecture. At OpenBMB, we believe the next leap in #LLM scaling lies in Hybrid Architectures. MiniCPM-SALA, has validated this vision by achieving a

keyboard_arrow_left Previous keyboard_arrow_right Next

78 87 291 435K 249

@XingweiQu48400 @GeZhang86038849 Thanks for the reply! Actually my question is, when the KV-pair of a concept is replicated, say, for m times, will it receive m times pre-softmax attention weight for each query? Also I wonder what exactly is S in Sec 3.5.1? Thanks in advance!

@GeZhang86038849 Wonderful work! While I'm still a bit confused by the repeat_interleave approach: As KVs are replicated/upsampled, will it be no longer equivalent to the original cross-attn? Or there is some impl. tweak to ensure the equivalence?

Kimi Linear Tech Report is dropped! 🚀 huggingface.co/moonshotai/Kim… Kimi Linear: A novel architecture that outperforms full attention with faster speeds and better performance—ready to serve as a drop-in replacement for full attention, featuring our open-sourced KDA kernels! Kimi Linear offers up to a 75% reduction in KV cache usage and up to 6x decoding throughput at a 1M context length. Key highlights: 🔹 Kimi Delta Attention: A hardware-efficient linear attention mechanism that refines the gated delta rule. 🔹 Kimi Linear Architecture: The first hybrid linear architecture to surpass pure full attention quality across the board. 🔹 Empirical Validation: Scaled, fair comparisons + open-sourced KDA kernels, vLLM integration, and checkpoints. The future of agentic-oriented attention is here! 💡

When we put lots of text (eg a code repo) into LLM context, cost soars b/c of the KV cache’s size. What if we trained a smaller KV cache for our documents offline? Using a test-time training recipe we call self-study, we find that this can reduce cache memory on avg 39x (enabling 26x higher tok/s and lower TTFT) while maintaining quality. These smaller KV caches, which we call cartridges, can be trained once and reused for different user requests! Github: HazyResearch/cartridges

Today Thinking Machines Lab is launching our research blog, Connectionism. Our first blog post is “Defeating Nondeterminism in LLM Inference” We believe that science is better when shared. Connectionism will cover topics as varied as our research is: from kernel numerics to prompt engineering. Here we share what we are working on and connect with the research community frequently and openly. The name Connectionism is a throwback to an earlier era of AI; it was the name of the subfield in the 1980s that studied neural networks and their similarity to biological brains. thinkingmachines.ai/blog/defeating…

Today we're putting out an update to the JAX TPU book, this time on GPUs. How do GPUs work, especially compared to TPUs? How are they networked? And how does this affect LLM training? 1/n

A distributed conference....

NeurIPS Conference @NeurIPSConf

a year ago

NeurIPS is pleased to officially endorse EurIPS, an independently-organized meeting taking place in Copenhagen this year, which will offer researchers an opportunity to additionally present their accepted NeurIPS work in Europe, concurrently with NeurIPS. Read more in our blog

10 122 795 117K 95

Highly recommended! The historical development of attention mechanisms is tightly coupled with the evolution of mainstream model architectures in AGI. A lot insights and bitter lessons buried inside the blog!

jianlin.su @Jianlin_S

a year ago

The Linear Attention Odyssey: From Imitation to Innovation and Back kexue.fm/archives/11033

6 19 98 13K 58