Anthony Chen @_anthonychen
nlp research @googledeepmind anthonywchen.github.io little worm in big apple Joined May 2017-
Tweets162
-
Followers416
-
Following495
-
Likes5K
New from @GoogleDeepMind: When can you trust your LLM? We show that LLMs consistently overestimate their own accuracy on some topics (eg nutrition) while underestimating it on others (eg math). Our Few-shot Recalibrator fixes LLM over/under-confidence: arxiv.org/abs/2403.18286 🧵
The next chapter about transformers is up on YouTube, digging into the attention mechanism: youtu.be/eMlx5fFNoYc The model works with vectors representing tokens (think words), and this is the mechanism that allows those vectors to take in meaning from context.
Introducing Gecko 🦎, a new text embedding model from Google DeepMind! Distilled from LLMs, Gecko offers powerful embeddings for various NLP tasks. Gecko is now available in Google Cloud API 👉bit.ly/google-gecko-a… Paper: bit.ly/google-gecko Colab: bit.ly/google-gecko-c…
Thanks @USC_ISI and @HJCH0 for having us! Check out a recording of our talk "The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI"👇
📢 Week of Mar. 18th is a bonus week with another seminar! On Mar. 21st, Thursday 11AM-12PM PST, we have @_anthonychen and @ShayneRedford give us a talk on "The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI" @USC_ISI @cutelabname_nlp
New Resource: Foundation Model Development Cheatsheet for best practices We compiled 250+ resources & tools for: 🔭 sourcing data 🔍 documenting & audits 🌴 environmental impact ☢️ risks & harms eval 🌍 release & monitoring With experts from @AiEleuther, @allen_ai,…
We at @allen_ai know our fine-tuned models are not particularly close to state of the art right now, but at least they're the best models where you know everything that went in every step along the way. OLMo Instruct v1 is here. Lot's more to come. huggingface.co/allenai/OLMo-7…
Gemini 1.5 Pro launched last week and already we're seeing the community produce some amazing interactions with long context. Below 👇 are some highlights and cool posts from folks who have gotten early access. A few thoughts from discussions / reactions from the community so…
ByteDance v OpenAI⚠️, LAION-5B CSAM☢️ & NYT v OpenAI🛑 illustrate rising lockdown + legal risk on data. Need more informed training data selection? 🔗 dataprovenance.org Detailed licenses, terms, sources, properties. 📢 Come help us build it! All open sourced. 1/ 🧵
Optical illusions with diffusion models. There are so many good gifs on this page but honestly I would like several million more. dangeng.github.io/visual_anagram…
We at @GoogleDeepMind are excited to announce #GNoME - an AI tool that has discovered 2.2 million new materials, and helps to predict material stability. We're releasing 381K stable materials to help scientists pursue materials discovery breakthroughs. dpmd.ai/PK-materials
🧵Announcing GPQA, a graduate-level “Google-proof” Q&A benchmark designed for scalable oversight! w/ @_julianmichael_, @sleepinyourhat GPQA is a dataset of *really hard* questions that PhDs with full access to Google can’t answer. Paper: arxiv.org/abs/2311.12022
I am the first author of the Galactica paper and have been quiet about it for a year. Maybe I will write a blog post talking about what actually happened, but if you want the TLDR: 1. Galactica was a base model trained on scientific literature and modalities. 2. We approached…
I am the first author of the Galactica paper and have been quiet about it for a year. Maybe I will write a blog post talking about what actually happened, but if you want the TLDR: 1. Galactica was a base model trained on scientific literature and modalities. 2. We approached…
📢 We are expanding the instruct/align datasets in the 🌟Data Provenance Collection🌟 Are there any great/new ones not covered? Available at: github.com/Data-Provenanc…
wake up babe, the year’s biggest data data set research project just dropped The Data Provenance Initiative analyzed 1,800+ popular fine-tuning text data sets and found a crisis of confusion. W/insights from @ShayneRedford @sarahookr washingtonpost.com/technology/202…
It is hard to overstate how huge this is. Data laundering is a huge problem in AI, and doing a systematic review and audit of licenses is a massive contribution in and of itself, let alone the additional exploration and filtering tools. This is the best NLP data work of 2023.
It is hard to overstate how huge this is. Data laundering is a huge problem in AI, and doing a systematic review and audit of licenses is a massive contribution in and of itself, let alone the additional exploration and filtering tools. This is the best NLP data work of 2023.
Many of these trends don't hold. Last week we celebrated @geoffreyhinton's retirement, and a few weeks earlier saw @kkariko receive the Nobel Prize. Their research took decades to come together, and had enormous impact at a world scale. We'd be much worse off if they'd pivoted!
Many of these trends don't hold. Last week we celebrated @geoffreyhinton's retirement, and a few weeks earlier saw @kkariko receive the Nobel Prize. Their research took decades to come together, and had enormous impact at a world scale. We'd be much worse off if they'd pivoted!
Better way to do interpretability:♟️Interpretability has been my passion for more than a decade. Most of time however, I was frustrated; many method don't seem to meet their promise, some even provably wrong*. I felt stuck in this impossible task.
(1/6) 🚀🚀 Thrilled that our paper arxiv.org/abs/2305.14907 has been accepted to #EMNLP2023 findings! 🎉 tl;dr: Selecting in-context examples that together cover all the salient aspects of the test input yields training-free methods that beat even trained SoTA methods! 💪🔥
Sameer Singh @sameer_
7K Followers 2K Following Cofounder @SpiffyAI and Assoc Prof at @UCIrvine, working on reliable LLMs, explanations for AI+ML, adversaries for NLP, and debugging/evaluation.
Ofir Press @OfirPress
10K Followers 3K Following I build tough benchmarks for LMs and then I get the LMs to solve them. Postdoc @Princeton. PhD from @nlpnoah @UW. Ex-visiting researcher @MetaAI & @MosaicML.
Luyu Gao @luyu_gao
1K Followers 241 Following PhD candidate @CarnegieMellon @LTIatCMU On the job market for full-time industry position.
Mike Lewis @ml_perception
6K Followers 227 Following Llama3 pre-training lead. Partially to blame for things like the Cicero Diplomacy bot, BART, RoBERTa, kNN-LM, top-k sampling & Deal Or No Deal.
Tim Dettmers @Tim_Dettmers
29K Followers 821 Following PhD Student at @UW. I blog about deep learning and PhD life at https://t.co/Y78KDJJFE7.
Gabriel Ilharco @gabriel_ilharco
4K Followers 1K Following Building cool things @xAI. Prev. PhD at UW, Google AI
Harsh Trivedi @harsh3vedi
264 Followers 487 Following #NLProc PhD candidate in @stonybrooku. Past intern @allen_ai & student research visitor @CILVRatNYU
Ekin Akyürek @akyurekekin
2K Followers 726 Following graduate student in computer science @MITEECS/@MIT_CSAIL
Shayne Longpre @ShayneRedford
4K Followers 997 Following PhD @MIT. Prev: @Google Brain, @apple ML, @stanfordnlp. 🇨🇦 Interests: AI/ML/NLP, Data-centric AI, transparency & societal impact
Michi Yasunaga @michiyasunaga
3K Followers 868 Following CS PhD @Stanford working on language models and multimodal models. Previously @Meta @GoogleDeepMind @Yale
Luca Soldaini 🎀 @soldni
6K Followers 1K Following I like tokens! Lead for OLMo data team at @allen_ai (Dolma 🍇), OSS is fun, @QueerInAI organizer 🤖☕️🍕they/them (views mine, not my employer’s)
Nandan Thakur @nandan__thakur
2K Followers 2K Following PhD @uwaterloo | Author of BEIR, Upcoming: TREC-RAG | Prev: @GoogleAI, @UKPLab | Undergrad @BitsPilaniGoa | Interested in IR and NLP 🇨🇦🇩🇪🇮🇳
Shruti Rijhwani @shrutirij
4K Followers 499 Following * Research Scientist @GoogleDeepMind * #NLProc research * PhD from @LTIatCMU * Amateur woodworker, scuba diver, foosball player
Tesitews @Tesitews6YX5G
0 Followers 87 Following
Erlinda Teachout @ErlindaTea87662
66 Followers 5K Following
LeonaMalory @15GqV5NQTielRYg
1 Followers 122 Following
KristinLongman @s46CA4jAV693qg
0 Followers 73 Following
Candice Pam @candice_pa62985
88 Followers 5K Following
Nylah Lamascolo @NLamascol
84 Followers 5K Following
Arlo Tyger @ArloTyger63245
81 Followers 5K Following
Devorah Eaby @DevorahEab44277
82 Followers 5K Following
Terresa Lamana @terr_lama
79 Followers 5K Following
Arif Ahmad @arif_ahmad_py
275 Followers 7K Following All things AI, Computer Science and Circuits! Prev. @GoogleAI
Urvashi Khandelwal @ukhndlwl
2K Followers 611 Following Research Scientist @GoogleDeepMind, Stanford CS PhD @stanfordnlp
Nilda Hoving @hovi_nil
81 Followers 5K Following
Keeley Dellasanta @DellasaKeel
36 Followers 5K Following
Chaitanya Malaviya @cmalaviya11
99 Followers 121 Following PhD student at UPenn | currently @GoogleDeepMind
Heike Robleto @roble_hei
28 Followers 5K Following
Yiğit Polat @dyigitpolat
138 Followers 561 Following Average C++ enjoyer and a Ph.D. student @NUSingapore, School of Computing. Hardware-aware neural network discovery for neuromorphic AI accelerators. vi/vim.
Sasikanth Kotti @kotti_sasikanth
559 Followers 4K Following Graduate Student @iitjodhpur @CSEIITJ1 / Computer Vision, Trusted AI, Deep Learning. Volunteer Research Engineer @openminedorg. @ml_collective and @forai_ml
Jiacen Xu @JiacenXu
241 Followers 350 Following Ph.D. Candidate @UCIEngineering | Ex-Intern @MSFTResearch | Master and Undergrad @sjtu1896
Krish Dasgupta @officialKrishD
879 Followers 4K Following Forever Learner | Building Reinforcement Learning Systems | Healthcare | Robots and Brains | Graph ML for Health
HashHakim @hash_hakim
125 Followers 4K Following
Isabelle Lee @i_g_lee
132 Followers 239 Following ML/NLP PhD student @nlp_usc interested in emergence, interpretability, and reasoning. 한american, she, phase transition enthusiast, sagittarius.
Soumya Sanyal @ssanyal8
439 Followers 532 Following Ph.D. Candidate @USC | Research Assistant @iiscbangalore | Bachelor's @IITKgp | Working on #NLProc
Nara-simba @narasimba7
156 Followers 2K Following Vews are personal; do not reflect opinion of the place I work. Retweets draw attention, not all retweets are endorsements
OneHundred @OneHundred12733
1 Followers 396 Following
Gaurav Singh Tomar @gtomar_google
216 Followers 82 Following Research and Machine Intelligence Engineer @GoogleResearch
Vera_US_ @VeraUS255128
28 Followers 2K Following
bamfit @bamfit516751
56 Followers 439 Following
Kyle Marieb @kylemarieb
738 Followers 5K Following Profoundly deaf with cochlear implants 🦻🤖 YouTube Backend SWE 📺
ajikangelo @ajikangelo
121 Followers 1K Following Electronics and Computer Engineering Student||Tech Enthusiast||Web & App Dev||Software and AI guy ||Technical Writer
Manan Dey @manandey
97 Followers 2K Following
Dan Alexandru @TukeysFence
9 Followers 159 Following A data scientist, a product manager and many other things
Theawhough @theawhough71230
19 Followers 2K Following
Raphael @rahoff8
17 Followers 134 Following
Nooghoson @nooghoson85020
64 Followers 2K Following
bagofwords.ai @bagofwordsai
282 Followers 4K Following All About NLP and Its Applications #safenlp #NLProc #ai #ml
Chen Cai@NeurIPS2023 @ChenCaiUCSD
301 Followers 428 Following CS PhD at UC San Deigo. Work on geometric deep learning.
Anshuman Sahoo @anshuML264
410 Followers 5K Following Senior ML engineer at BenchSci; University of Toronto
Justin Cho 조현동 @HJCH0
747 Followers 688 Following NLP PhD candidate @cutelabname_nlp @nlp_usc @USC_ISI Interned @amazonscience, @AIatMeta x2, @stitchfix CS B.Eng @hkust Cofounder @auto_lang
Akari Asai @AkariAsai
11K Followers 650 Following Ph.D. student @uwcse & @uwnlp. NLP. IBM Ph.D. fellow (2022-2023). Meta student researcher (2023-) . ☕️ 🐕 🏃♀️🧗♀️🍳
(((ل()(ل() 'yoav))).. @yoavgo
46K Followers 2K Following
Sam Bowman @sleepinyourhat
35K Followers 3K Following AI alignment + LLMs at NYU & Anthropic. Views not employers'. No relation to @s8mb. I think you should join @givingwhatwecan.
Sasha Rush @srush_nlp
52K Followers 464 Following Professor, Programmer in NYC. Cornell Tech, Hugging Face 🤗 https://t.co/cZl0wTfqGz
Yann LeCun @ylecun
712K Followers 719 Following Professor at NYU. Chief AI Scientist at Meta. Researcher in AI, Machine Learning, Robotics, etc. ACM Turing Award Laureate.
Percy Liang @percyliang
49K Followers 408 Following Associate Professor in computer science @Stanford @StanfordHAI @StanfordCRFM @StanfordAILab @stanfordnlp | cofounder @togethercompute | Pianist
Sameer Singh @sameer_
7K Followers 2K Following Cofounder @SpiffyAI and Assoc Prof at @UCIrvine, working on reliable LLMs, explanations for AI+ML, adversaries for NLP, and debugging/evaluation.
Ofir Press @OfirPress
10K Followers 3K Following I build tough benchmarks for LMs and then I get the LMs to solve them. Postdoc @Princeton. PhD from @nlpnoah @UW. Ex-visiting researcher @MetaAI & @MosaicML.
Najoung Kim 🫠 @najoungkim
2K Followers 493 Following At @BULinguistics and visiting @GoogleAI part-time. 🤖🔠🐱
AK @_akhaliq
310K Followers 3K Following AI research paper tweets, ML @Gradio (acq. by @HuggingFace 🤗) dm for promo follow on Hugging Face: https://t.co/q2Qoey80Gx
Yoav Artzi @yoavartzi
13K Followers 162 Following Research/prof @cs_cornell + @cornell_tech🚡 / https://t.co/9YnWry7yHs / https://t.co/3VmRSyYm2d / asso. faculty director @arxiv / building https://t.co/f9QkzO5kaC
Christopher Manning @chrmanning
127K Followers 116 Following Director, @StanfordAILab. Assoc. Director, @StanfordHAI. Founder, @stanfordnlp. Prof. CS & Linguistics, @Stanford. IP @aixventureshq. 🇦🇺 Do #NLProc & #AI. 👋
Christopher Potts @ChrisGPotts
11K Followers 620 Following Stanford Professor of Linguistics and, by courtesy, of Computer Science, and member of @stanfordnlp and @StanfordAILab. He/Him/His.
Luyu Gao @luyu_gao
1K Followers 241 Following PhD candidate @CarnegieMellon @LTIatCMU On the job market for full-time industry position.
Andrej Karpathy @karpathy
979K Followers 905 Following 🧑🍳. Previously Director of AI @ Tesla, founding team @ OpenAI, CS231n/PhD @ Stanford. I like to train large deep neural nets 🧠🤖💥
Kyunghyun Cho @kchonyc
61K Followers 2K Following a combination of a mediocre scientist, a mediocre manager, a mediocre advisor & a mediocre PC at @nyuniversity (@CILVRatNYU) & @genentech (@PrescientDesign).
No Priors @NoPriorsPod
2K Followers 81 Following @saranormous and @eladgil host your guide to the AI revolution. podcast and YouTube links: https://t.co/KaLZIjm131
unusual_whales @unusual_whales
1.7M Followers 2K Following Stocks/Options/Crypto/Market News +Tools. Not advice 🐳 who changed 🏛️. Get $50-$5000 to trade: https://t.co/wGf2ZdlXpw Discord: https://t.co/0xJ9e0ZYYG More: https://t.co/nsxZlPV0pC
Nikolay Savinov 🇺�.. @SavinovNikolay
1K Followers 0 Following Research Scientist at @GoogleDeepMind Work on LLM pre-training in Gemini ♊ 10M context length in Gemini 1.5 Pro 📈
Logan Kilpatrick @OfficialLoganK
92K Followers 2K Following Lead product for @Google AI Studio and working on the Gemini API, helping developers build with AI, my views!
Chaitanya Malaviya @cmalaviya11
99 Followers 121 Following PhD student at UPenn | currently @GoogleDeepMind
Keller Jordan @kellerjordan0
1K Followers 199 Following Independent research Prev MLE @ Hive AI, math @ UCSD
Isabelle Lee @i_g_lee
132 Followers 239 Following ML/NLP PhD student @nlp_usc interested in emergence, interpretability, and reasoning. 한american, she, phase transition enthusiast, sagittarius.
Jacob Austin @jacobaustin132
3K Followers 798 Following @Google @DeepMind researcher. AI for math and science. Coding. Gemini. I also play piano. NYC. Opinions my own
Karan Goel @krandiash
3K Followers 882 Following Founder @cartesia_ai, Machine Learning PhD at @StanfordAILab, CMU / IIT-Delhi alum.
Sepp Hochreiter @HochreiterSepp
10K Followers 395 Following Pioneer of Deep Learning and known for vanishing gradient and the LSTM. I mostly tweet about random ArXiv papers which sparked my interest.
BlinkDL @BlinkDL_AI
7K Followers 90 Following RWKV = 100% RNN with GPT-level performance. https://t.co/TkdxOJSFWX and https://t.co/86DzS6arA0
François Fleuret @francoisfleuret
31K Followers 457 Following Prof. @Unige_en, Adjunct Prof. @EPFL_en, Research Fellow @idiap_ch, co-founder @nc_shape. AI and machine learning since 1994. I like reality.
Nikos Pappas @nik0spapp
731 Followers 667 Following Senior Applied Scientist at @awscloud #NLProc #ML 🤖 Previously Postdoc @uwcse, @Idiap_ch, PhD @epfl_en.
Nathan Lambert @natolambert
25K Followers 690 Following Figuring out AI @allen_ai, "rl boi" DM me papers. Writes @interconnectsai, talks @retortai Has phd and some credentials
Keith Stevens @fozziethebeat
219 Followers 173 Following Helping LLMs solve meaningful human problems. On the hunt for a new job in the United States (and leaving Japan).
Shruti Rijhwani @shrutirij
4K Followers 499 Following * Research Scientist @GoogleDeepMind * #NLProc research * PhD from @LTIatCMU * Amateur woodworker, scuba diver, foosball player
lmsys.org @lmsysorg
37K Followers 172 Following Large Model Systems Organization. We created Vicuna and Chatbot Arena! Compare 30+ LLMs (GPT-4/Claude/Llamas) side-by-side at https://t.co/IDFeIDIOtm
Gaurav Singh Tomar @gtomar_google
216 Followers 82 Following Research and Machine Intelligence Engineer @GoogleResearch
Arthur Mensch @arthurmensch
40K Followers 873 Following Co-founder and CEO @MistralAI. Apply https://t.co/yHGRZAtjcx
Columbia NLP @columbianlp
2K Followers 29 Following Natural language processing group at Columbia University. @Zhou_Yu_AI, Kathleen McKeown, Julia Hirschberg, Smaranda Muresan, @dnlbauer
Melvin Johnson @melvinjohnsonp
980 Followers 280 Following Researcher @ Google Research. Multilingual NLP and MT. Previously, Stanford CS.
Cartesia @cartesia_ai
1K Followers 8 Following Cartesia is training next-gen foundation models with subquadratic deep learning architectures. Sign up for early access at https://t.co/c5og0yF1Pz
Albert Gu @_albertgu
9K Followers 90 Following assistant prof @mldcmu. chief scientist @cartesia_ai. leading the ssm revolution.
Illia (root.near) (�.. @ilblackdragon
82K Followers 398 Following Co-Founder @NEARProtocol. Working on bringing 1B users into web3. Previously builder of #Tensorflow & ML researcher.
Jakob Uszkoreit @kyosu
4K Followers 276 Following
david rein @idavidrein
2K Followers 983 Following Sentio ergo sum. AI alignment research at NYU, early employee @cohere
sarah guo // convicti.. @saranormous
91K Followers 3K Following startup investor and builder, founder @w_conviction. accelerating AI adoption, interested in progress. tech podcast: @nopriorspod
Justin Cho 조현동 @HJCH0
747 Followers 688 Following NLP PhD candidate @cutelabname_nlp @nlp_usc @USC_ISI Interned @amazonscience, @AIatMeta x2, @stitchfix CS B.Eng @hkust Cofounder @auto_lang
Julian Michael @_julianmichael_
1K Followers 122 Following Researching stuff @NYUDataScience. he/him
typedfemale @typedfemale
23K Followers 477 Following a really exciting new account "have you ever though you might be like scott alexander? very smart, but can't do math" - anon
Jürgen Schmidhuber @SchmidhuberAI
107K Followers 0 Following Invented principles of meta-learning (1987), GANs (1990), Transformers (1991), very deep learning (1991), etc. Our AI is used many billions of times every day.
Andrew Ng @AndrewYNg
1.0M Followers 913 Following Co-Founder of Coursera; Stanford CS adjunct faculty. Former head of Baidu AI Group/Google Brain. #ai #machinelearning, #deeplearning #MOOCs
Robert Mahari @RobertMahari
90 Followers 22 Following JD-PhD @medialab and @Harvard_Law. Computational lawyer.
Jad Kabbara @jad_kabbara
1K Followers 731 Following NLP Postdoc @MIT Center for Constructive Communication (CCC). PhD from McGill University @rllabmcgill & @Mila_Quebec. @AUB_Lebanon alum.
Will Brannon @wwbrannon
579 Followers 2K Following PhD @MIT. Recently intern @ Amazon. Interests: NLP, graph deep learning, computational social science.
Maithra Raghu @maithra_raghu
17K Followers 476 Following Cofounder and CEO @Samaya_AI. Formerly Research Scientist Google Brain (@GoogleAI), PhD in ML @Cornell.
Matthew Peters @mattthemathman
2K Followers 572 Following Cofounder @SpiffyAI. Research Scientist at AI2 (@allenai_org).
Ashish Vaswani @ashVaswani
19K Followers 2K Following
Jay Alammar @JayAlammar
35K Followers 1K Following Machine learning and language models R&D. Builder. Writer. Visualizing AI, ML, and LLMs one concept at a time. @Cohere. https://t.co/TquuQXlLOJ
SoCal NLP Symposium @socalnlp
207 Followers 72 Following ☀️🏝️Annual symposium with students and faculty to promote NLP research in the (Southern) California region 👩💻 #SoCalNLP2023 🔜 @ucla, posts by @BrihiJ
Roy Frostig @froystig
1K Followers 500 Following research scientist at @googledeepmind. co-author of JAX (https://t.co/TaE9kvzZMa)
@nvidia CEO JensenHuang delivered the world's 1st AI supercomputer DGX-1 today to SAIL! @jcniebles @silviocinguetta
PSA TO THE NBA: NIKOLA JOKIC IS COMING OVER THIS YEAR. PICKED BY THE NUGGETS AT #41 IN 2014, HE WILL DOMINATE EVERYTHING YOU HAVE AND LAUGH.
Do models need to reason in words to benefit from chain-of-thought tokens? In our experiments, the answer is no! Models can perform on par with CoT using repeated '...' filler tokens. This raises alignment concerns: Using filler, LMs can do hidden reasoning not visible in CoT🧵
Some personal updates: I joined OpenAI a few months ago, working on all things robustness/safety/privacy. Also, we are working to publish more of our safety work. See my first project here below, where we make initial progress on prompt injections and other attacks!
Introducing the Instruction Hierarchy, our latest safety research to advance robustness for prompt injections and other ways of tricking LLMs into executing unsafe actions. More details: arxiv.org/abs/2404.13208
@drillling_up @zhangir_azerbay @moinnadeem Oh! So, that's what was meant with deep learning hit a wall
Introducing Arena-Hard – a pipeline to build our next generation benchmarks with live Arena data. Highlights: - Significantly better separability than MT-bench (22.6% -> 87.4%) - Highest agreement to Chatbot Arena ranking (89.1%) - Fast & cheap to run ($25) - Frequent update…
Today a polymath public intellectual wandered into my domain of expertise (game theory), and I discovered they were just smoke and mirrors. Ah, well.
Llama3 was trained on 15 trillion tokens of public data. But where can you find such datasets and recipes?? Here comes the first release of 🍷Fineweb. A high quality large scale filtered web dataset out-performing all current datasets of its scale. We trained 200+ ablation…
We have just released 🍷 FineWeb: 15 trillion tokens of high quality web data. We filtered and deduplicated all CommonCrawl between 2013 and 2024. Models trained on FineWeb outperform RefinedWeb, C4, DolmaV1.6, The Pile and SlimPajama!
I'm seeing a lot of questions about the limit of how good you can make a small LLM. tldr; benchmarks saturate, models don't. LLMs will improve logarithmically forever with enough good data.
Yes, both the 8B and 70B are trained way more than is Chinchilla optimal - but we can eat the training cost to save you inference cost! One of the most interesting things to me was how quickly the 8B was improving even at 15T tokens.
Excited to share a preview of Llama3, including the release of an 8B and 70B (82 MMLU, should be the best open weights model!), and preliminary results for a 405B model (still training, but already competitive with GPT4). Lots more still to come... ai.meta.com/blog/meta-llam…
There’s a coffee shop down the road from my apartment in London and I’m obsessed with it. It’s one of those genuinely independent shops where there’s only one location and they’re very much doing their own thing and don’t intend of mass expansion despite its popularity.
🥁 Llama3 is out 🥁 8B and 70B models available today. 8k context length. Trained with 15 trillion tokens on a custom-built 24k GPU cluster. Great performance on various benchmarks, with Llam3-8B doing better than Llama2-70B in some cases. More versions are coming over the next…
Check out our new work on few-shot recalibration of LMs with our amazing intern @XiangLisaLi2!
New from @GoogleDeepMind: When can you trust your LLM? We show that LLMs consistently overestimate their own accuracy on some topics (eg nutrition) while underestimating it on others (eg math). Our Few-shot Recalibrator fixes LLM over/under-confidence: arxiv.org/abs/2403.18286 🧵
New from @GoogleDeepMind: When can you trust your LLM? We show that LLMs consistently overestimate their own accuracy on some topics (eg nutrition) while underestimating it on others (eg math). Our Few-shot Recalibrator fixes LLM over/under-confidence: arxiv.org/abs/2403.18286 🧵
So proud of my brilliant spouse @vibhuti_ramach !!!
Congrats to Vibhuti Ramachandran, @UCIrvine global & international studies, who's received the @AIISIndia Joseph W. Elder Prize in the Indian Social Sciences for her forthcoming book, “Immoral Traffic”: An Ethnography of Law, NGOs, & the Governance of Prostitution (@CUPAcademic)!
The famous LSTM paper has reached 100k citations on Google Scholar. We therefore surprised the one and only @HochreiterSepp with some cake 🎉🎉
Our 12 scaling laws (for LLM knowledge capacity) are out: arxiv.org/abs/2404.05405. Took me 4mos to submit 50,000 jobs; took Meta 1mo for legal review; FAIR sponsored 4,200,000 GPU hrs. Hope this is a new direction to study scaling laws + help practitioners make informed decisions
We are witnessing greatness 👨🍳
SWE-agent is our new system for autonomously solving issues in GitHub repos. It gets similar accuracy to Devin on SWE-bench, takes 93 seconds on avg + it's open source! We designed a new agent-computer interface to make it easy for GPT-4 to edit+run code github.com/princeton-nlp/…
Excited to see our 🍮Flan-Palm🌴 work finally published in @JmlrOrg 2024! Looking back, I see this work as pushing hard on scaling: post-training data, models, prompting, & eval. We brought together the methods and findings of many awesome prior works, scaled them up, and…
Trends for United States
You might like
