-
Tweets119
-
Followers187
-
Following2K
-
Likes228
Apologies that I haven't written anything since joining Thinking Machines but I hope this blog post on a topic very near and dear to my heart (reproducible floating point numerics in LLM inference) will make up for it!
Apologies that I haven't written anything since joining Thinking Machines but I hope this blog post on a topic very near and dear to my heart (reproducible floating point numerics in LLM inference) will make up for it!
building pretraining infrastructure is an exercise in complexity management, abstraction design, operability/observability, and deep systems and ML understanding. reflects some of the trickiest and most rewarding problems in software engineering. which makes it really fun!
New in-depth blog post - "Inside vLLM: Anatomy of a High-Throughput LLM Inference System". Probably the most in depth explanation of how LLM inference engines and vLLM in particular work! Took me a while to get this level of understanding of the codebase and then to write up…
This is the best public resource on scaling hardware for AI, and its free. "How to Scale Your Model" is the bible from Google DeepMind that covers the math, systems and scaling laws for LLM training and inference workloads. Approachable yet thorough. Absolute Must-read.
节选： 1. 开源了就意味着第一方再也不能用各种 hack 的方式粉饰效果，必须拿出足够通用、任何第三方拿到同样的 weights 都要能很简单地复现出你的效果才行。 2. 绝大多数 Agent 产品，离了 Claude 以后，什么都不是。
节选： 1. 开源了就意味着第一方再也不能用各种 hack 的方式粉饰效果，必须拿出足够通用、任何第三方拿到同样的 weights 都要能很简单地复现出你的效果才行。 2. 绝大多数 Agent 产品，离了 Claude 以后，什么都不是。
Getting mem-bound kernels to speed-of-light isn't a dark art, it's just about getting the a couple of details right. We wrote a tutorial on how to do this, with code you can directly use. Thanks to the new CuTe-DSL, we can hit speed-of-light without a single line of CUDA C++.
Getting mem-bound kernels to speed-of-light isn't a dark art, it's just about getting the a couple of details right. We wrote a tutorial on how to do this, with code you can directly use. Thanks to the new CuTe-DSL, we can hit speed-of-light without a single line of CUDA C++.
Very cool thread about the CS336 Language Models from Scratch course at Stanford taught by @percyliang et al. Makes me wish I was a student again!
Very cool thread about the CS336 Language Models from Scratch course at Stanford taught by @percyliang et al. Makes me wish I was a student again!
Ilya Sutskever, U of T honorary degree recipient, June 6, 2025 youtu.be/zuZ2zaotrJs?si… via @YouTube This is a must watch speech, the wisest words you could hear. Congratulations 🙌 @ilyasut @UofT was very fortunate to have you and so many other amazing students and…
Dear PhD students now regretting taking offers at US schools: If you turned down PhD offers in Canada, but want to rethink that, email the professors who were trying to recruit you. They might be able to pull some strings. Your sane neighbor to the north, Canada
Strong recommend for this book and the JAX/TPU docs, even if you are using Torch / GPUs. Clean notation and mental model for some challenging ideas. github.com/jax-ml/scaling… github.com/jax-ml/scaling… docs.jax.dev/en/latest/note…
🚀 Breaking: SGLang provides the first open-source implementation to serve @deepseek_ai V3/R1 models with large-scale expert parallelism and prefill-decode disaggregation on 96 GPUs. It nearly matches the throughput reported by the official DeepSeek blog, achieving 52.3K input…
The SGLang guys @lmsysorg are always doing such incredible work. This is also what the open-source community has made possible! 🚀
4/ Our AI leadership was on display at @googlecloud Next, where we launched Ironwood, our most powerful TPU yet — 10X compute boost, optimized for inference at scale. And we’re first to bring NVIDIA’s next-gen Blackwell GPUs to customers. Now with tools for building multi-agent…
Full talk of Ilya here 👇
Full talk of Ilya here 👇
> go to Claude store > ask the man at the counter if it is TPU Claude or Trainium Claude > he doesn't understand > pull out illustrated diagram explaining the differences between TPU and Trainium > it's a good Claude sir > get the membership > Trainium Claude
When a product is 10x better, it doesn't matter if it starts from a small niche and has no marketing. It will grow steadily, and eventually it will win. That's how I feel about the current trajectory of JAX.
Congratulations to @geoffreyhinton for winning the Nobel Prize in physics!!
I gave a talk at the MLSys conference in May this year, touching on various topics, including large-scale ML training systems, abstractions for embedding ML choices in computer systems, and CO2e emissions of language model training. mlsys.org/virtual/2024/i…
An index of my llama3.1 405b related posts this week: - Memory size matters for inference: x.com/jiayq/status/1… - Lepton endpoint release: x.com/jiayq/status/1… - Measurement and qualitative analysis of the model: x.com/jiayq/status/1… - Running 405b on a poor 4090? A fun…
An index of my llama3.1 405b related posts this week: - Memory size matters for inference: x.com/jiayq/status/1… - Lepton endpoint release: x.com/jiayq/status/1… - Measurement and qualitative analysis of the model: x.com/jiayq/status/1… - Running 405b on a poor 4090? A fun…
they did it. these crazy bastards actually released a full and detailed technical report. the llama 3 paper is going to be up there with the deepseekv2 paper in terms of detail and quality
Lorena @0UJM2F21vcIZm
13 Followers 636 Following My hobbies include eating and complaining that I’m getting fat.
EV_BatteryBets🇺�... @Ugega608
40 Followers 2K Following 15-30% Monthly | 2 High-Conviction Stocks.Short-Term Gains: 15-20% in Days/Weeks.DM "JOIN" for WhatsApp Alerts. Live Trade Signals • Market Analysis
Arc Jax @arcjax7
172 Followers 2K Following
Guanghao Ye @guanghao_ye
405 Followers 819 Following PhD student @MIT_CSAIL working on optimization and LLM
Rhougaw @Rhougaw5396
59 Followers 2K Following
Kan Wu @flyflyf04862062
676 Followers 404 Following Inference @xai, PhD @WisconsinCS. Prev: Google, USTC. Machine Learning Systems, Distributed Systems. Tweets are my own.
Sreeraw @Sreeraw6776150
52 Followers 2K Following
Afroz Mohiuddin @afrozenator
1K Followers 5K Following @OpenAI, ex @Google, @AIAtMeta. Interested in Science, Psychology, Investing and generally everything. Good Thoughts, Good Words, Good Deeds.
Andrew Curran @AndrewCurran_
34K Followers 13K Following 🏰 - I write about AI, mostly. Expect some strange sights.
Jonathan Lai @_JLai
500 Followers 186 Following Post training @GoogleDeepMind, Gemini Reasoning, training algorithms, opinions are my own
Yuchen Zhuang @yuchen_zhuang
891 Followers 359 Following Research Scientist @GoogleDeepMind | Gemini Thinking & Coding | LLM Agent | Prev: PhD @MLatGT | Opinions are my own.
Yufan Song @YufanSong98
196 Followers 523 Following Build Post-Train RL Agent Infra at ByteDance Seed, openhands @allhands_ai
Chris 🇨🇦 @llm_wizard
1K Followers 490 Following Working on cool open-source AI stuff @ NVIDIA Views my own.
Seeteighn @SeeteighnWujN_
140 Followers 3K Following
Teendus @TeendusQRq2Y
43 Followers 2K Following
Timothea @050s8aI6Arf6o
85 Followers 7K Following
traveling @shangrilatrave1
131 Followers 850 Following
MaryDoherty @P0163RcQERpUx
56 Followers 7K Following
HilaryClemens @gs5zIqui33u1Wk
68 Followers 7K Following
Bolian Li @lblaoke
55 Followers 91 Following PhD Student @PurdueCS | LLM Alignment, Bayesian Deep Learning, Imbalanced Learning
Boutear @boutear84525
83 Followers 7K Following A strong woman is one who is determined to do what others are determined not to do.
DonnaSpenser @RMIb01P1w14A60B
57 Followers 7K Following
SetllaHal @Vk5yodZsg6A4A0
62 Followers 7K Following
PhoebeMay @38V31FWDRunVmh
53 Followers 6K Following
Zane @takagirimi4869
83 Followers 7K Following
Erica @hazamataka20877
73 Followers 7K Following
MarthaRuskin @M00Zbt6u379pC80
92 Followers 7K Following
HildaWalter @7rG143nZtFQI2
93 Followers 7K Following
Alina @Suhay8708294103
88 Followers 7K Following She is a force to be reckoned with and she knows it.
Jieun Han @z_eunie
519 Followers 243 Following PhD student @kaistcsdept, interested in NLP application & Language learning
MonaRamsden @B454z7JCEW6Y05
76 Followers 7K Following
Thoynnairs @Thoynnairsqpi7
49 Followers 5K Following
Seatas @Seatasc8h
15 Followers 1K Following
Neethet @NeethetpzPi
43 Followers 5K Following
Tinghao Xie @VitusXie
662 Followers 452 Following 3rd year ECE PhD candidate @Princeton | Prev Intern @Meta GenAI
Yueqi Xie @XieYueqi
275 Followers 474 Following Postdoctoral research associate @Princeton, AI and Society, Responsible AI, Computational Social Science, prev PhD @hkust, BS @PKU1898
Bodun Hu @BodunHu
131 Followers 265 Following CS Ph.D. student @UTAustin. Student researcher @Meta. Working on systems for ML
QuintinaII. @0VHHF7PJ2XVr0
17 Followers 1K Following
Hannah @fbRX3P982X099F
8 Followers 505 Following
HoneyCarrie @8Vtk2buVPR6BkN4
19 Followers 1K Following
Kauan @kauankleos
193 Followers 4K Following
Daniel Han @danielhanchen
28K Followers 2K Following Building @UnslothAI. Finetune train LLMs faster. LLMs bug hunter. OSS package https://t.co/aRyAAgKOR7. YC S24. Prev ML at NVIDIA. Hyperlearn used by NASA.
Heiga Zen (全 炳河... @heiga_zen
9K Followers 192 Following Principal Scientist (Director) @GoogleDeepMind / GDM Tokyo site lead．波瀬小⇒一志中⇒鈴鹿高専⇒名工大 (IBM TJ Watson intern)⇒東芝欧州研⇒Google (Speech🇬🇧⇒Brain🇯🇵) ⇒GoogleDeepMind
Takahiro Miki @ki_ki_ki1
3K Followers 876 Following @GoogleDeepMind / prev ETH Zurich RSL. Reinforcement learning Locomotion. DARPA SubT Challenge winner.
Jarrod Kahn @kahnvex
427 Followers 209 Following Dad, Tech Enthusiast, Problem Solver @GoogleDeepMind
Baizhou Zhang @baizhou_zh83925
68 Followers 69 Following SGLang Contributor | MSCS Student at UCSD | Ex-Intern at Nvidia, Baidu, HPC-AI Tech
Ruihang Chu @RuihangChu
186 Followers 49 Following 🎨 Core developer @Alibaba_Wan | 🎓 PhD from CUHK | Views my own
skcd @skcd42
8K Followers 291 Following Understanding the universe @xai ex hacking @aide_dev ex fb engineer ICPC WF its just code 👨🏼💻
Henry Ko @henryHM_ko
850 Followers 360 Following performance and efficiency in ML | CS @ UC Berkeley, @BerkeleyML
Karthik A Sankararama... @karthikabinav
2K Followers 3K Following AI Research @ Meta Superintelligence Labs. Long-term Affiliations: #iitm, @UMDCS, @facebook, @meta
Szymon Tworkowski @s_tworkowski
10K Followers 659 Following reasoning @xAI | prev. @GoogleAI @UniWarszawski | LongLLaMA
Shen Zhuoran @CMS_Flash
542 Followers 173 Following Reasoning/coding @xai. Ex-@GoogleAI Resident/@augmentcode. Alum @HKUniversity. 💎 Terran @StarCraft II.
liang hu @lianghu349103
461 Followers 14 Following LLM Eval Researcher at Bytedance Ex-investment banker Progress is measured in better questions, not better answers
Gabriele Berton @gabriberton
7K Followers 1K Following Postdoc @Amazon working on VLM - ex @CarnegieMellon @PoliTOnews @IITalk
Anmol Gulati @anmol01gulati
2K Followers 977 Following Research Scientist @ Google Deepmind. TL on Agents and Project Mariner in Gemini. Prev: cofounder @AdeptAILabs, Google Brain. Working on creating AGI.
Jasmine @j_asminewang
7K Followers 1K Following alignment @OpenAI. past @AISecurityInst @verses_xyz @kernel_magazine @readtrellis @copysmith_ai
Clive Chan @itsclivetime
11K Followers 3K Following intelligence per picojoule @openai / prev led dojo workload @tesla
Boyuan Chen @BoyuanChen0
4K Followers 507 Following Researcher @OpenAI, core member of GPT image generation and member of Sora video generation. PhD @MITEECS. I do world models, RL, and robotics.
Quentin Gallouédec @QGallouedec
3K Followers 674 Following PhD - Research @huggingface 🤗 TRL lead maintainer 🇫🇷 in 🇨🇦
Hanson Wang @hansonwng
752 Followers 277 Following @OpenAI Codex // previously cofounder @arcwisedata
Standard Kernel Co. @Standard_Kernel
802 Followers 1 Following Building AI Infrastructure with AI; fast kernels go brrr
Muyu He @HeMuyu0327
993 Followers 224 Following Post-training @CollinearAI | Trying to be an expert of mixtures
Eric Zelikman @ericzelikman
21K Followers 2K Following building for humans // was lgtm-ing @xAI, phd-ing @stanford
Yuanjing Shi @shingjan_
314 Followers 339 Following GenAI at Apple. Ex-Nvidia/Snowflake. Opinions are my own.
Charlie Snell @sea_snell
8K Followers 6K Following PhD student @berkeley_ai; research @cursor_ai; prev @GoogleDeepMind. My friend told me to tweet more. I stare at my computer a lot and make things
Sam Schoenholz @sschoenholz
7K Followers 672 Following @thinkymachines previously: @openai, google brain.
Dinghuai Zhang 张鼎... @zdhnarsil
4K Followers 2K Following Researcher at @MSFTResearch. Prev: PhD at @Mila_Quebec, intern at @Apple MLR and FAIR Labs @MetaAI, math undergraduate at @PKU1898.
Aditya Makkar @AdityaMakkar000
247 Followers 156 Following post-training & reasoning @cohere | CS @ uwaterloo
Wenting Zhao @wzhao_nlp
5K Followers 607 Following reasoning & llms @Alibaba_Qwen Opinions are my own
Shivalika Singh @singhshiviii
2K Followers 773 Following Research Engineer @Cohere_Labs @cohere | @huggingface fellow 🤗 | “Research means that you don't know, but are willing to find out” ✨
Mudit Verma @v_mudit
548 Followers 605 Following Research Scientist @GoogleDeepMind. RLHF, Agentic LLMs with humans. @dtu_delhi @ASU
Fangru Lin @FangruLin99
4K Followers 467 Following Research Intern @GoogleDeepMind; DPhil student @UniofOxford; Clarendon Scholar; Prev @MSFTResearch, @Microsoft, @turinginst; Computational Linguist
Hongyuan Mei @RoverHM
1K Followers 162 Following Core Contributor to Grok 4 & Grok 4 Heavy. Member of Technical Staff @xAI. Training knowledgeable AI reasoners. ex-@GoogleDeepMind, @TTIC_Connect, @jhuclsp.
Bert Maher @tensorbert
3K Followers 373 Following I’m a software engineer building high-performance kernels and compilers at Anthropic! Previously at Facebook/Meta (PyTorch, HHVM, ReDex)
Marzieh Fadaee @mziizm
2K Followers 594 Following seeks to understand language. head of @Cohere_Labs. phd from @UvA_Amsterdam. https://t.co/YI5NC5J5e4.
Nikos Kolotouros @nikoskolot
1K Followers 295 Following Research Scientist @GoogleDeepMind working on Veo. Veo Ingredients (I/O 2025). CS PhD from @Penn.
Max Vladymyrov @mvladymyrov
1K Followers 1K Following MTS at Anthropic. Previously: Research @ {Google DeepMind, Yahoo Labs} 🇺🇦Trends for United States
You might like
