Sang Michael Xie @sangmichaelxie
PhD student @StanfordAILab @StanfordNLP @Stanford advised by Percy Liang and Tengyu Ma. Prev: visiting @GoogleAI Brain, BS, MS Stanford ‘17 cs.stanford.edu/~eix Stanford, CA Joined May 2019-
Tweets358
-
Followers3K
-
Following709
-
Likes2K
Connect Later, our targeted fine-tuning method for robust+accurate models, tops the WILDS leaderboard for iWildCam and Camelyon17 and achieves SoTA on astronomical time-series tasks (3 very different domains)! arxiv.org/abs/2402.03325
Connect Later, our targeted fine-tuning method for robust+accurate models, tops the WILDS leaderboard for iWildCam and Camelyon17 and achieves SoTA on astronomical time-series tasks (3 very different domains)! arxiv.org/abs/2402.03325
today, gen AI performance is surprisingly robust to new data/tasks, even beating specialized models! the secret: training on large-scale unlabeled data. what can we as scientists learn from this? some thoughts on robustness & the power of the unlabeled data you already have:
Announcing Score Entropy Discrete Diffusion (SEDD) w/ @chenlin_meng @StefanoErmon. SEDD challenges the autoregressive language paradigm, beating GPT-2 on perplexity and quality! Arxiv: arxiv.org/abs/2310.16834 Code: github.com/louaaron/Score… Blog: aaronlou.com/blog/discrete-… 🧵1/n
We need to rigorously reason about the benefits and risks of open foundation models. There is plenty of debate and speculation, animating lively policy conversations. To make progress, we have put out new work on the societal impact of open FMs crfm.stanford.edu/open-fms/
Great effort led by @AlbalakAlon to corral the wild west of LM data selection! A meta-issue: how do we make data work (esp. for pretraining) more accessible? Not everyone can train 7B LMs, but a first bar is to show that the benefits don't shrink with scale, at smaller scales.
Great effort led by @AlbalakAlon to corral the wild west of LM data selection! A meta-issue: how do we make data work (esp. for pretraining) more accessible? Not everyone can train 7B LMs, but a first bar is to show that the benefits don't shrink with scale, at smaller scales.
Interestingly, pretraining on unlabeled source/target+finetuning doesn’t improve much over just supervised learning on source in iWildcam-WILDS. Correspondingly, the connectivity conditions on the success of contrastive pretraining for UDA (arxiv.org/abs/2204.00570) also fail!
Interestingly, pretraining on unlabeled source/target+finetuning doesn’t improve much over just supervised learning on source in iWildcam-WILDS. Correspondingly, the connectivity conditions on the success of contrastive pretraining for UDA (arxiv.org/abs/2204.00570) also fail!
The paper submission deadline has been extended to 2/11 AoE. Look forward to your submissions!
The paper submission deadline has been extended to 2/11 AoE. Look forward to your submissions! https://t.co/CE4UPFZ9A5
Euclidean geometry problems have been my favorite math puzzles since middle school. The most intriguing part of it is the creation of auxiliary lines, which opens a space for imagination and the freedom to explore various diagrams. Once a proof is found, these auxiliary lines…
Euclidean geometry problems have been my favorite math puzzles since middle school. The most intriguing part of it is the creation of auxiliary lines, which opens a space for imagination and the freedom to explore various diagrams. Once a proof is found, these auxiliary lines…
Excited to co-organize this ICLR 2024 workshop! I think better data will be crucial for the next big advances in foundation models. The submission date is Feb 3 - details at sites.google.com/view/dpfm-iclr…
Excited to co-organize this ICLR 2024 workshop! I think better data will be crucial for the next big advances in foundation models. The submission date is Feb 3 - details at sites.google.com/view/dpfm-iclr…
Excited to announce the 2nd ME-FoMo workshop on understanding foundation models will be at ICLR 2024 , Vienna! Topics include pretraining, adaptation and emergence amongst many others. Paper deadline: Feb 3 Website : sites.google.com/view/me-fomo20… Open Review : tinyurl.com/2p6hzybr
Announcing the 2nd Workshop on Mathematical and Empirical Understanding of Foundation Models (ME-FoMo) at ICLR 2024! Improving our understanding helps us advance capabilities and build safer, more aligned models. Paper deadline is Feb 3! Website: sites.google.com/view/me-fomo20…
I’m a big fan of exploring data mixtures as a key ingredient for training better models - happy to find this poster presented by @sangmichaelxie on the topic!
Loved this nice and simple idea for better data selection in LMs. First, use high level features to describe high-value data (eg textbook chunks). Then use importance sampling to prioritize similar data in a large dataset. @sangmichaelxie
I'm at #NeurIPS2023 workshops today! Check out our work on high dimensional prediction and applications to online combinatorial optimization, extensive form games, and prediction sets! ⏱️ Talk 2:30-3, poster 3-4 📍OPT for ML w/ Georgy, Ramya, @Aaroth arxiv.org/abs/2310.17651
Transformers power most advances in LLMs, but its core attention layer can’t scale to long context. With @_albertgu, we’re releasing Mamba, an SSM architecture that matches/beats Transformers in language modeling, yet with linear scaling and 5x higher inference throughput. 1/
Transformers power most advances in LLMs, but its core attention layer can’t scale to long context. With @_albertgu, we’re releasing Mamba, an SSM architecture that matches/beats Transformers in language modeling, yet with linear scaling and 5x higher inference throughput. 1/ https://t.co/7gdXw1qP2H
Percy Liang @percyliang
49K Followers 408 Following Associate Professor in computer science @Stanford @StanfordHAI @StanfordCRFM @StanfordAILab @stanfordnlp | cofounder @togethercompute | Pianist
Jim Fan @DrJimFan
229K Followers 3K Following @NVIDIA Sr. Research Manager & Lead of Embodied AI (GEAR Lab). Creating foundation models for Humanoid Robots & Gaming. @Stanford Ph.D. @OpenAI's first intern.
Ananya Kumar @ananyaku
4K Followers 469 Following Researcher at @openai Previously PhD at Stanford University (@StanfordAILab) advised by Percy Liang and Tengyu Ma
Behnam Neyshabur @bneyshabur
18K Followers 689 Following Senior Staff Research Scientist @GoogleDeepMind, Interested in reasoning w. LLMs, traveling & backpacking
rishi @RishiBommasani
4K Followers 2K Following Stanford CS PhD @StanfordCRFM @StanfordNLP @StanfordAILab @StanfordHAI Advisers: @percyliang @jurafsky Previous: @CornellCIS @clairecardie #FoundationModels
Dan Roy @roydanroy
45K Followers 2K Following ML / AI researcher, emphasis on theory. Research Director and Canada CIFAR AI Chair, @VectorInst Professor, @UofT (Statistics/CS)
Shane Gu @shaneguML
28K Followers 1K Following Research Scientist & Manager @GoogleDeepMind Tokyo/MTV. ex: @GoogleAI Brain, @OpenAI. (JP: @shanegJP)
Akari Asai @AkariAsai
11K Followers 650 Following Ph.D. student @uwcse & @uwnlp. NLP. IBM Ph.D. fellow (2022-2023). Meta student researcher (2023-) . ☕️ 🐕 🏃♀️🧗♀️🍳
Kayo Yin @kayo_yin
8K Followers 556 Following PhD student @berkeley_ai @berkeleynlp working on interpretability and signed languages. Former @msftresearch @deepmind @carnegiemellon @polytechnique. 🇫🇷🇯🇵
Yann Dubois @yanndubs
4K Followers 1K Following PhD student @stanfordAILab | Prev: AI resident @metaai, @vectorinst, @CambridgeMLG
Ethan Caballero is bu.. @ethanCaballero
8K Followers 2K Following ML PhD student @Mila_Quebec ; previously @GoogleDeepMind
Miles Brundage @Miles_Brundage
43K Followers 10K Following Policy research at @openai. I mostly tweet about AI, animals, and sci-fi. He/him. Views my own.
Sharon Y. Li @SharonYixuanLi
7K Followers 657 Following Assistant Professor @WisconsinCS. Formerly postdoc @StanfordAILab, Ph.D. @Cornell. Making AI safe and reliable for the open world.
Sara Hooker @sarahookr
39K Followers 7K Following I lead @CohereForAI. Formerly Research @Google Brain @GoogleDeepmind. ML Efficiency at scale, LLMs, @trustworthy_ml. Changing spaces where breakthroughs happen.
Tim Dettmers @Tim_Dettmers
29K Followers 819 Following PhD Student at @UW. I blog about deep learning and PhD life at https://t.co/Y78KDJJFE7.
Ofir Press @OfirPress
10K Followers 3K Following I build tough benchmarks for LMs and then I get the LMs to solve them. Postdoc @Princeton. PhD from @nlpnoah @UW. Ex-visiting researcher @MetaAI & @MosaicML.
Alex Ratner @ajratner
5K Followers 548 Following @SnorkelAI @uwcse / prev @StanfordAILab – Interested in data management systems for machine learning, weak supervision, and impactful applications.
Nick Cannon @inkymaze
6K Followers 2K Following vp growth @gauntlet_xyz. @aerafinance. poker → crypto ⇄ fintech.
hkpacjlcyh @cjlcyh50367y8v
29 Followers 1K Following We first transfer USDT to you TRC20, you return 90% to BEP20, you get 10% , 2K per day Our co hv a large amt of USDT need to from TRC20 convert to BEP20 network
Chip Huyen @chipro
92K Followers 443 Following Data processing on GPUs @VoltronData Designing ML Systems: https://t.co/G81hL2dWmr @designmlsys #AI x #GPU
Melihcan Erol @hsme1986
3 Followers 24 Following
MesubsetofRunionC @mesubsetof
35 Followers 426 Following
Arif Ahmad @arif_ahmad_py
248 Followers 7K Following All things AI, Computer Science and Circuits! Prev. @GoogleAI
DelMorganCo-Review @DelMorganReview
34 Followers 525 Following We seek to reduce the female founders funding gap to protect others from deceptive business practices like those exhibited by DelMorgan & Co.
kovariance @kovariance
68 Followers 2K Following
Woodrow @Woodrow12465631
6 Followers 340 Following
Muhammad Imran @Muhamma55183541
3 Followers 127 Following
Eva Louise Marie Gabr.. @e681554349
9 Followers 3K Following
Achyuta Rajaram @AchyutaBot
267 Followers 400 Following 17 | mech interp @mit_csail | @atlasfellow '23 | STS 2024
I'mDust @cs__Henry
2 Followers 87 Following Try not to become a man of success, but rather try to become a man of value.
Aryan Pandey (Look fo.. @AryanPa66861306
1K Followers 3K Following Half Machine Learning Engineer || DevOps and Machine Learning || Open Source at OpenVINO
gabedaramola @gabedaramola
0 Followers 5K Following
Makya @Makya12345678
6 Followers 962 Following
Mathieu Ravaut @MatRavox
371 Followers 2K Following PhD candidate in NLP at @ntunlpsg w @JotyShafiq and @astarhq. Ex @layer6ai | @uoftcompsci | @centralesupelec
Sichao Liu @ErikLiuSe
45 Followers 289 Following
bellamy @solacebellamy
46 Followers 114 Following
tradernews.ai @tradernewsai
2K Followers 1K Following AI + MARKETS + NEWS THIS IS NOT INVESTMENT ADVICE
Ruidong Wu @RuidongWu
57 Followers 279 Following Researcher at @HelixonBio. Prev: @UofIllinois @MIT_CSAIL @Tsinghua_Uni.
Bob0409 @hzxhx111
36 Followers 79 Following
jay @seekerum
160 Followers 887 Following
Yun Fu @fuyun
86 Followers 409 Following
Zeqian Bao @BaoZeqian18347
5 Followers 300 Following
Clayton @cthorrez
1K Followers 1K Following LLM applied scientist by day, esports data scientist for fun. Working on rating systems and benchmarks for esports (and LLMs?) I ❤️ paired comparison data
Cloud Twitt @Twitt2Cloud
156 Followers 380 Following
George Grigorev @iamgrigorev
2K Followers 532 Following formerly generative ml @ snap, global talent interested in llms
Zhiyong Wang @Zhiyong16403503
380 Followers 2K Following Visiting Ph.D. student at Cornell University. Ph.D. candidate at CUHK. Working on bandits and reinforcement learning theory.
Qasim Ali @QasimAliSidhu
168 Followers 1K Following AI First Tech Savvy Technical Customer Support Engineer #AI #GenerativeAI #GenAI #FutureAILeaders #AIFirst
kunal singh @ikunalsingh7
62 Followers 660 Following Lead AI Researcher https://t.co/z4idFlmggM (T2I), Lead AI Researcher @fractalai Prev: GSoC @CERN, Alumni @IITKgp, Intern @AmiiThinks Diffusion, VLMs, reasoning@LLM
pinktopus @pinktopus_
6 Followers 39 Following
Alexander Wan @alexwan55
472 Followers 944 Following CS at Berkeley; @BerkeleyML @BerkeleyNLP; NLP research
Jeff Nickerson @jvnickerson
155 Followers 821 Following
Pensé FFun @inftyCategory
113 Followers 6K Following
Mathieu Alain @miniapeur
19K Followers 2K Following Researching @ai_ucl. Co-organises @uclcsml and @logconference. FR, EN, trying ES. 🇹🇼🇨🇦🇬🇳🇺🇸🇩🇴🇫🇷🇪🇸🇬🇧🇿🇦
Megan Richards @megan_richards_
124 Followers 288 Following AI Resident @AIatMeta, previously @DukeInnovate. Reliable/Responsible AI.
Percy Liang @percyliang
49K Followers 408 Following Associate Professor in computer science @Stanford @StanfordHAI @StanfordCRFM @StanfordAILab @stanfordnlp | cofounder @togethercompute | Pianist
Jim Fan @DrJimFan
229K Followers 3K Following @NVIDIA Sr. Research Manager & Lead of Embodied AI (GEAR Lab). Creating foundation models for Humanoid Robots & Gaming. @Stanford Ph.D. @OpenAI's first intern.
(((ل()(ل() 'yoav))).. @yoavgo
46K Followers 2K Following
Ananya Kumar @ananyaku
4K Followers 469 Following Researcher at @openai Previously PhD at Stanford University (@StanfordAILab) advised by Percy Liang and Tengyu Ma
Gautam Kamath @thegautamkamath
44K Followers 505 Following Assistant Prof of CS @UWaterloo, Faculty @VectorInst, Canada @CIFAR_News AI Chair. Co-EiC @TmlrOrg. I lead @TheSalonML. Privacy, robustness, machine learning.
Behnam Neyshabur @bneyshabur
18K Followers 689 Following Senior Staff Research Scientist @GoogleDeepMind, Interested in reasoning w. LLMs, traveling & backpacking
Jacob Andreas @jacobandreas
14K Followers 958 Following Teaching computers to read. Assoc. prof @MITEECS / @MIT_CSAIL (he/him). https://t.co/5kCnXHjtlY https://t.co/2A3qF5vdJw
Kyunghyun Cho @kchonyc
61K Followers 2K Following a combination of a mediocre scientist, a mediocre manager, a mediocre advisor & a mediocre PC at @nyuniversity (@CILVRatNYU) & @genentech (@PrescientDesign).
Christopher Manning @chrmanning
126K Followers 115 Following Director, @StanfordAILab. Assoc. Director, @StanfordHAI. Founder, @stanfordnlp. Prof. CS & Linguistics, @Stanford. IP @aixventureshq. 🇦🇺 Do #NLProc & #AI. 👋
rishi @RishiBommasani
4K Followers 2K Following Stanford CS PhD @StanfordCRFM @StanfordNLP @StanfordAILab @StanfordHAI Advisers: @percyliang @jurafsky Previous: @CornellCIS @clairecardie #FoundationModels
Eric Jang @ericjang11
69K Followers 3K Following physical AGI at 1X. Author of "AI is Good for You" https://t.co/eFg4WXhg0p
Delip Rao e/σ @deliprao
46K Followers 5K Following Busy inventing the shipwreck. @Penn. Past: @johnshopkins, @UCSC, @Amazon, @Twitter ||Art: #NLProc, Vision, Speech, #DeepLearning || Life: 道元, improv, running 🌈
Sam Bowman @sleepinyourhat
35K Followers 3K Following AI alignment + LLMs at NYU & Anthropic. Views not employers'. No relation to @s8mb. I think you should join @givingwhatwecan.
Anthropic @AnthropicAI
261K Followers 26 Following We're an AI safety and research company that builds reliable, interpretable, and steerable AI systems. Talk to our AI assistant Claude at https://t.co/aRbQ97uk4d.
AI at Meta @AIatMeta
531K Followers 255 Following Together with the AI community, we are pushing the boundaries of what’s possible through open science to create a more connected world.
Dan Roy @roydanroy
45K Followers 2K Following ML / AI researcher, emphasis on theory. Research Director and Canada CIFAR AI Chair, @VectorInst Professor, @UofT (Statistics/CS)
Sasha Rush @srush_nlp
52K Followers 464 Following Professor, Programmer in NYC. Cornell Tech, Hugging Face 🤗 https://t.co/cZl0wTfqGz
Shane Gu @shaneguML
28K Followers 1K Following Research Scientist & Manager @GoogleDeepMind Tokyo/MTV. ex: @GoogleAI Brain, @OpenAI. (JP: @shanegJP)
Marc Marone @ruyimarone
422 Followers 586 Following PhD student at Johns Hopkins @jhuclsp. Previously @microsoft Semantic Machines, @mstranslator, @GeorgiaTech
rohan anil @_arohan_
12K Followers 2K Following Principal Engineer, @GoogleDeepMind Gemini. prev PaLM-2. Tinkering with optimization and distributed systems. opinions are my own.
Tianyu Gao @gaotianyu1350
3K Followers 686 Following CS PhD student @Princeton @Princeton_nlp working on NLP. Previously: @Tsinghua_Uni @TsinghuaNLP
Kangwook Lee @Kangwook_Lee
2K Followers 667 Following Assistant Professor, ECE, UW-Madison / Leading deep learning research @ KRAFTON
Dimitris Papailiopoul.. @DimitrisPapail
11K Followers 970 Following prof @ wisconsin; thinking about transformers; learning in context; babas of Inez Lily
Bill Yuchen Lin 🤖 @billyuchenlin
6K Followers 2K Following Research @allen_ai. I evaluate (multi-modal) LLMs, build agents, and study the science of LLMs. Previously: @GoogleAI & @MetaAI FAIR @nlp_usc
Fuzhao Xue @XueFz
4K Followers 542 Following Ph.D. candidate@NUSingapore, Intern of GEAR @NVIDIA | Google PhD Fellow | LLM, Foundation Model Scaling | Ex-@GoogleBrain | Zero-shot Cooking Learner🧑🍳
Allan Zhou @AllanZhou17
1K Followers 443 Following Final-year AI PhD student @Stanford. NN architecture design, learned optimizers, and hparam optimization.
Yangsibo Huang @YangsiboHuang
1K Followers 726 Following PhD candidate @Princeton. Prev: @GoogleAI @AIatMeta.
John Schulman @johnschulman2
39K Followers 609 Following Cofounder @openai, lead post-training for ChatGPT and the API. Interested in reinforcement learning, alignment, birds, jazz music
Ryan Chi @ryanandrewchi
18 Followers 42 Following Student Researcher, LLM Reasoning Team @GoogleDeepMind. Led @stanfordnlp's Alexa Prize Team to 1st Place (Science) at @AmazonScience's Socialbot Challenge.
Greg Durrett @gregd_nlp
6K Followers 752 Following CS professor at UT Austin. I do NLP most of the time. he/him
Weiyan Shi @shi_weiyan
3K Followers 683 Following Postdoc @StanfordNLP, incoming assistant professor @Northeastern, PhD @Columbia| Prev Intern @MetaAI |Co-created CICERO | persuasive chatbots + privacy #nlproc
Alon Albalak @AlbalakAlon
885 Followers 464 Following CS PhD candidate at @ucsbNLP. Research: Data-centric AI, Efficiency in ML, NLP.
Jason Weston @jaseweston
9K Followers 568 Following Research @MetaAI+NYU. Pretrain+FT: NLP from Scratch (2011). Multilayer attention+position embed+LLM: MemNets (2015). Recent (2023+):Sys 2 Attn, Self-Rewarding..
Shital Shah @sytelus
10K Followers 8K Following Deep learning research and code. If universe is an optimizer, what is the loss function? All opinions are my own.
Ahmed Ahmed @AhmedSQRD
407 Followers 795 Following CS PhD @Stanford - Funding @KnightHennessy @NSF- 🇸🇩 - tweets include history & politics
Together AI @togethercompute
27K Followers 303 Following The future of AI is open-source. Let's build together.
Tian Xie @tianxie233
75 Followers 296 Following Research Engineer @character_ai | previously @SFResearch
Yasaman Bahri @yasamanbb
5K Followers 954 Following Research Scientist @GoogleDeepMind // ML + physics + quantum materials // Ph.D. theoretical cond matt physics @UCBerkeley.
Maurice Weber @mauriceweberq
72 Followers 359 Following AI Research @togethercompute | ML PhD @ETH @DS3Lab
Center for Research o.. @StanfordCRFM
2K Followers 3 Following Making foundation models more reliable and accessible.
Jun-Yan Zhu @junyanz89
9K Followers 582 Following Assistant professor at Generative Intelligence Lab @SCSatCMU @CarnegieMellon. Understanding and creating pixels (https://t.co/yvop9D3ftM).
Emily Huynh @_ehuynh
19 Followers 62 Following PhD student @pennbioeng | prev: engineer @czbiohub, @thermofisher, @ucberkeley '20 |👩🏻💻👩🏻🔬 (she/her)
Feiyang Kang @feiyang_ml
51 Followers 33 Following ML/AI Ph.D. student at Virginia Tech advised by Prof. @ruoxijia, passionate about #DataCentricAI and #TrustworthyML . All contacts are welcome :)
Voyage AI @Voyage_AI_
2K Followers 164 Following Building embedding/vectorization models, customized for your domain and company, for better retrieval quality https://t.co/MEAhTpBQqd
Francis Lewis @_francis_lewis
73 Followers 227 Following m.a.d. @anduriltech | prev robotics @stanfordsvl
Weijia Shi @WeijiaShi2
5K Followers 967 Following PhD student @uwcse @uwnlp | Visiting Researcher @MetaAI | Undergrad @CS_UCLA | https://t.co/eLBQmgkvym
Ruiqi Zhong @ZhongRuiqi
2K Followers 698 Following 5th Year Ph.D. @BerkeleyNLP, Columbia'19. part time working for @AnthropicAI . Supervising machines to do what I can't do.
Lucio Dery Jnr Mwinm @derylucio
461 Followers 956 Following
Mengzhou Xia @xiamengzhou
3K Followers 618 Following PhD student @princeton_nlp, MS @CarnegieMellon, Undergrad at Fudan.
CS Faculty Jobs @csfacultyjobs
5K Followers 2 Following Faculty jobs in Computer Science worldwide. Mostly automated. Mention/DM openings & we'll retweet. Created by @emilianoucl, now run by @shaddih
Ethan Chi @ethanachi
297 Followers 147 Following NLP research at @wehrtyou. Previously at @stanfordnlp. Pianist/organist.
Alexis Conneau @alex_conneau
24K Followers 111 Following Audio AGI Research Lead @OpenAI - GPT-Next - Past: XLM, Unsupervised ASR, Unsupervised MT, Wav2vec 2.0/XLSR, MUSE, Unsupervised cross-lingual transfer
Tony Lee @tonyh_lee
402 Followers 86 Following Incoming PhD Candidate @StanfordAILab @StanfordNLP @Stanford. Author of HELM + extensions (https://t.co/f9UOXPWkpR). Prev: Research Eng at @StanfordCRFM.
Mark Chen @markchen90
10K Followers 245 Following Head of Frontiers Research at OpenAI. Coach for the USA IOI Team.
Stephen McAleer @McaleerStephen
3K Followers 807 Following Postdoc at CMU researching LLM agents and AI alignment
Jonathan Ho @hojonathanho
4K Followers 151 Following
Jascha Sohl-Dickstein @jaschasd
19K Followers 623 Following Member of the technical staff @ Anthropic. Most (in)famous for inventing diffusion models. AI + physics + neuroscience + dynamics.
Hyung Won Chung @hwchung27
18K Followers 229 Following Research Scientist @OpenAI. Past: @Google Brain / PhD @MIT
Shayne Longpre @ShayneRedford
4K Followers 998 Following PhD @MIT. Prev: @Google Brain, @apple ML, @stanfordnlp. 🇨🇦 Interests: AI/ML/NLP, Data-centric AI, transparency & societal impact
Saurabh Garg @saurabh_garg67
862 Followers 579 Following Building next-gen AI at @MistralAI | prev/ PhD @mldcmu; CS @iitbombay (undergrad); Collab @GoogleAI @awscloud @apple
One year ago, I left Google Brain (now DeepMind) to join a very early startup. We had fewer than 10 people at that time, and have grown many times since. Today, I am extremely proud to share our milestone. We are Augment. You can read about us here. techcrunch.com/2024/04/24/eri…
Happy to share our work on preference learning methods for LLMs. Key insights: 1. Use more on-policy samples > off-policy samples 2. Contrastive DPO > Pref-FT. Also we provide insights on DPO's training mechanism. 3. Theoretical unification under mode-covering/seeking KL
Many LLM fine-tuning methods. Unclear what you should use & why? In our new paper, we did an extensive study of on-policy RL, supervised & offline contrastive methods (DPO, IPO) to answer this... 🧵⬇️ On-policy > offline, mode-seeking > mode-covering understanding-rlhf.github.io
It's a great week for open source AI! Data is among the highest impact work to push the field forward. Bravo to 🤗
Llama3 was trained on 15 trillion tokens of public data. But where can you find such datasets and recipes?? Here comes the first release of 🍷Fineweb. A high quality large scale filtered web dataset out-performing all current datasets of its scale. We trained 200+ ablation…
We have just released 🍷 FineWeb: 15 trillion tokens of high quality web data. We filtered and deduplicated all CommonCrawl between 2013 and 2024. Models trained on FineWeb outperform RefinedWeb, C4, DolmaV1.6, The Pile and SlimPajama!
Yes, both the 8B and 70B are trained way more than is Chinchilla optimal - but we can eat the training cost to save you inference cost! One of the most interesting things to me was how quickly the 8B was improving even at 15T tokens.
Llama3 8B is trained on almost 100 times the Chinchilla optimal number of tokens
🆕📢 @Voyage_AI_'s new embedding model for legal and long-context retrieval and RAG: voyage-law-2! 1.🥇 # 1 on MTEB legal retrieval benchmark with a large margin 2.📜 Best quality for long-context (16K) 3.✨ Improved quality across domains 4.🛒 On AWS Marketplace #RAG #LLMs
Introducing 𝐀𝐋𝐎𝐇𝐀 𝐔𝐧𝐥𝐞𝐚𝐬𝐡𝐞𝐝 🌋 - Pushing the boundaries of dexterity with low-cost robots and AI. @GoogleDeepMind Finally got to share some videos after a few months. Robots are fully autonomous filmed in one continuous shot. Enjoy!
I’m a Dr now!! so grateful to my advisor and all my collaborators, friends, and family for supporting me every step of the way 🥰
I went to the Lincoln center yesterday for Rachmaninoff's No 2 concerto. It felt so refreshing to completely unplug and give this extraordinary piece of music my undivided attention. My favourite recording of this work is the 1963 Ashkenazy performance: youtube.com/watch?v=xyPDWa….
I am partial to the original version, but then again, if this is what it takes…
I took a famous paper and asked Claude to rewrite its introduction in the style of Malcolm Gladwell, while preserving the mathematical content
Dataset choice is crucial in today's ML training pipeline. We (@xiamengzhou and I) introduce desiderata for "good" data and explain how our recent algorithm, LESS, fits into the picture. Huge review of data selection algs for pre-training and fine-tuning! cs.princeton.edu/~smalladi/blog…
Today is the beginning of our moonshot to solve embodied AGI in the physical world. I’m so excited to announce Project GR00T, our new initiative to create a general-purpose foundation model for humanoid robot learning. The GR00T model will enable a robot to understand multimodal…
Built with Levanter!
Anticipatory Music Transformer by @StanfordCRFM 🎶 > A foundation model for symbolic music. > Supports generating accompaniments (enrich music) and infill (fill in musical details). > 780 Million parameters, trained for 800 Thousand steps. > Trained on Lakh, MetaMIDI and…
When we looked for a final strategic partner to fulfill our dreams for series C, only one name came to mind: @nvidia. We are so thrilled to welcome our newest investors, collaboration partners, and long-time research buddies to the @AbridgeHQ family: abridge.com/press-release/…
A commendable paper that make a versatile use of innovative and meticulous analysis to reach its notable conclusion.
Lots of people in CS are (almost surely) GPT-ing their peer reviews arxiv.org/abs/2403.07183
U give me: a bunch of unlabeled data. I give u: AI-generated labels. Result: a massive, but biased, val set. We use PPI to correct the bias, giving unbiased evaluations with better precision 🚀 arxiv.org/abs/2403.07008 Experiments on GPT-4 and ResNets, using @lmsysorg :)
What if we could use AI to evaluate AI? 🧐 This would save thousands of human-hours---e.g., on platforms like @lmsysorg. But it introduces bias! Enter AutoEval Done Right: producing unbiased evaluations of models with synthetic data! arxiv.org/abs/2403.07008
Thrilled to be starting a new adventure at Physical Intelligence with some amazing colleagues and friends! Learn more: physicalintelligence.company
🚨 Big news 🚨 Together with a set of amazing folks we decided to start a company that tackles one of the hardest and most impactful problems - Physical Intelligence In fact, we even named our company after that: physicalintelligence.company or Pi (π) for short 🧵
Since cat is out of the bag, it’s time I share: I’ll be starting a new adventure with an incredible team of friends and long-time collaborators to take on the big challenge of robot learning at scale! It's called Physical Intelligence (Pi… or π, like the symbol). 🧵👇
Trends for United States
You might like
