Simon Mo @simon_mo_

@norapom23 I hold Waterloo grads very high regards 🫡 zero miss, all 10x engs

@googlecloud and Inferact are announcing today a partnership to make TPU a first-class citizen in @vllm_project. This partnership puts both teams on one engineering roadmap to bring TPU to the broader open model ecosystem, optimizing vLLM as the agentic production serving engine for TPU: • Production serving features and optimized kernels • A native PyTorch path via TorchTPU • Moving towards day-0 support for frontier model releases We're also launching a community program: shared TPU capacity for open-source contributors, plus dedicated review and design help from the core vLLM maintainers at Inferact. Everything this collaboration produces is open source. Read the full announcement: inferact.ai/news/google-tp…

The first YOCO (you only look once) model for agentic use with shared kv cache and efficiency. vLLM is the go to production engine for agentic workload!

vLLM @vllm_project

6 days ago

🐳 DeepSeek-V4.1-Flash is out, and vLLM serves it from day 0, verified on NVIDIA and AMD GPUs! 🎉 552B MoE backbone, native vision, 1M context. Built for agents: 8B active while it reads your prompt, 16B while it writes. If you already run DeepSeek-V4 on vLLM, most of this stack

25 54 478 67K 75

Sparse MLA only attends to the top-K tokens, so the rest of the KV need not live on the GPU. Hybrid HiSparse in vLLM builds on that, and a request keeps decoding after its KV stops fitting in HBM. It keeps KV on the GPU while there is room. Under pressure a request releases its coldest pages to host memory, keeps a small hot buffer of what the indexer asks for, and keeps decoding instead of being preempted. 📊 Demonstrated on GLM 5.3, one 8× H200 node, full 1M context. Same host memory, configured concurrency 32: KV offloading kept 5-6 requests running. Hybrid HiSparse kept 19-25. 🔹 Hot pages are ordinary KV blocks from the same pool (Hybrid Memory Allocator) 🔹 One fused kernel resolves resident, hot and missing rows, CUDA-graph capturable 🔹 Prefix caching, OffloadingConnector, P/D imports and MTP keep working Built by @RedHat_AI and @PrimeIntellect with the vLLM community. Planned for v0.30; pinned commit, flags and calculator are in the post👇 🔗 vllm.ai/blog/2026-09-0…

A stronger open AI ecosystem is built by communities 🌏 Great to see @PyTorch bringing the community together and the open AI ecosystem continuing to grow across China ❤️

PyTorch @PyTorch

a week ago

More than 250 organizations across China contribute to PyTorch Foundation projects, including @DeepSpeedAI, Helion, PyTorch, @raydistributed, Safetensors, and @vllm_project. “Open source has become the default way the world builds AI, and the PyTorch Foundation is a leading hub

keyboard_arrow_left Previous keyboard_arrow_right Next

5 6 40 18K 10

x.com/i/article/2097…

🫡 vLLM is your engine of choice for agentic workload. The pareto curve is easily understood; but very hard to optimize against. The @inferact team did amazing work here, please check it out (and join us!)

vLLM @vllm_project

a week ago

New blog is out: vLLM x AgentX: Optimizing for Real-World Agentic Serving. Agent traffic stresses every layer of the serving stack at once. This post walks the full-stack work for optimizing vLLM on Agentic workloads, including the architecture, framework, and runtime

9 38 190 27K 107

New blog is out: vLLM x AgentX: Optimizing for Real-World Agentic Serving. Agent traffic stresses every layer of the serving stack at once. This post walks the full-stack work for optimizing vLLM on Agentic workloads, including the architecture, framework, and runtime optimizations, measured on AgentX, @SemiAnalysis_'s public agentic benchmark. 🧵1/6

TPU is an interesting chip with enormous potential. Shoutout to @googlecloud for all great work and support!

Dylan Patel @dylan522p

a week ago

We are excited to bring the first open benchmarking of Google's TPUs to the world Running every day, on many models + scenarios $/token is better than B200 and B300 Huge shout-out to Google @inferact and the InferenceX team at SemiAnalysis to this effort that's taken many months

34 53 952 199K 230

I couldn't find a public performant MXFP8 GEMM on MI355X, so I just asked @KimiDevs K3 to write one, and it beats torch._scaled_mm by >2x times 🤯 This is an adaptation of the official Gluon MXFP4 example. FlyDSL adaptation is WIP. @AnushElangovan you guys should try K3 for kernel engineering if you haven't!

Congratulations to @HUMAIN and @MiniMax_AI on HUMAIN-M3, a frontier Arabic model now available on HUMAIN Node 🎉. @vllm_project is running the inference under the hood, and we're looking forward to more Arabic use cases with this state-of-the-art stack.

HUMAIN @HUMAIN

2 weeks ago

HUMAIN unveils HUMAIN-M3, a frontier Arabic language model, commissioned by HUMAIN and developed by @MiniMax_AI, now available in research preview on HUMAIN Node. Read more: humain.com/news/humain-un… #HUMAIN #LEAP26 #TheEndOfLimits

21 18 68 55K 25

Shoutout to the cracked team at @vllm_project that implemented recent agentic workload optimizations. (1/5)🧵

IYKYK

SemiAnalysis @SemiAnalysis_

2 weeks ago

Shoutout to the cracked team at @vllm_project that implemented recent agentic workload optimizations. (1/5)🧵

7 26 185 52K 110

🎬Video generation faster than playback! 🚀MiniMax H3 on vLLM-Omni + FastVideo's FastH3: a complete 10.1s MP4 - video AND synchronized audio - rendered in 8.7s!⚡️ Thanks to @MiniMax_AI for the great Minimax H3 release, the FastVideo team @haoailab for open-sourcing FastH3 and helping on the serving integration, and @NVIDIAAI for the continued sponsorship and joint optimization efforts!

Production quality is a bar, as a community, we should never compromise. Great collaboration between @vllm_project and @FireworksAI_HQ investigating this! 🫡

Lin Qiao @lqiao

2 weeks ago

GLM-5.3-Flash is live on Fireworks. Day 2. Why not Day 0? Because being first isn't the goal. Being correct is. We unplugged peculiar behaviors of over-thinking from the initial tests, and shared all fixes back to open source. Quality trumps hype. Let's hold a high quality

21 6 213 42K 28

Personally I believe open research that benefits the broader OSS community deserves encouragement, not dismissal. Academia → production is one of the hardest jumps in MLSys and the FastVideo team did it with limited compute. Efforts like this deserve support, not trash talk.

Hao Zhang @haozhangml

3 weeks ago

oh, thanks for evaluting this for us. We wish we had more compute to distill with more data (like you would have), but it is happy to see that, with our very limited dozens of GPUs, we are already generating good videos. Also, I hope you could be less mean to us next time, as I

22 27 404 140K 51

(1/6) Open Weight @MiniMax_AI FastH3 v1: Generate 15s 768p video in 13s 🚀 - FastVideo collab w/ @nuvalab + FastGen - Up to 14x speedup on @NVIDIAAI Blackwell GPU - Fully open so community can run and improve the acceleration recipe. The era of open weight video models just started, that calls for post training. With great community effort such as Minimax H3 Max based on @MiniMax_AI , we have more to share about how post training can improve speed, quality and most importantly, how it’s done, with open weights and recipes.

@thesephist 🐻

@yvbbrjdr Semianalysis didn’t have other token based metrics yet but the team is working on adding it! github.com/SemiAnalysisAI… More than happy to reference the new metrics once they’re online.

Right on time. While the first vLLM conference is being held in the U.S., vLLM Github star crossed 90k! Wow