Alex Wettig @_awettig

PhD @Princeton trying to make sense of language models and their training data; trying to train agents @cursor_ai cs.princeton.edu/~awettig/ Joined July 2022

Tweets

186
Followers

2K
Following

584
Likes

2K

Charlie Snell @sea_snell

3 days ago

yolo run summer is over scaling laws fall has arrived

Preet Sojitra @Preet_Sojitra03

6 months ago

yolo run summer is over scaling laws fall has arrived

2 0 26 22K 1

Download Image

1 1 63 7K 4

🔍 How do we teach an LLM to 𝘮𝘢𝘴𝘵𝘦𝘳 a body of knowledge? In new work with @AIatMeta, we propose Active Reading 📙: a way for models to teach themselves new things by self-studying their training data. Results: * 𝟔𝟔% on SimpleQA w/ an 8B model by studying the wikipedia…

15 157 1K 126K 1K

Download Image

Stuart Sul @stuart_sul

3 weeks ago

MoE layers can be really slow. When training our coding models @cursor_ai, they ate up 27–53% of training time. So we completely rebuilt it at the kernel level and transitioned to MXFP8. The result: 3.5x faster MoE layer and 1.5x end-to-end training speedup. We believe our…

30 100 860 123K 410

Download Image

Alex Wettig @_awettig

2 months ago

Presenting two posters at ICML over the next two days: - Both at 11am - 1:30pm - Both about how to improve pre-training with domains - Both at stall # E-2600 in East Exhibition Hall A-B (!) Tomorrow: WebOrganizer w/ @soldni & @kylelostat Thursday: MeCo by @gaotianyu1350

1 9 50 11K 7

Download Image

Albert Gu @_albertgu

2 months ago

Tokenization is just a special case of "chunking" - building low-level data into high-level abstractions - which is in turn fundamental to intelligence. Our new architecture, which enables hierarchical *dynamic chunking*, is not only tokenizer-free, but simply scales better.

Sukjun (June) Hwang @sukjun_hwang

2 months ago

98 745 5K 736K 4K

Download Gif

61 192 1K 197K 768

Download Image

Anthropic @AnthropicAI

2 months ago

Anthropic staff realized they could ask Claude to buy things that weren’t just food & drink. After someone randomly decided to ask it to order a tungsten cube, Claude ended up with an inventory full of (as it put it) “specialty metal items” that it ended up selling at a loss.

65 210 4K 844K 330

Download Image

Alex Wettig @_awettig

3 months ago

New paper cutting through the thicket of KV cache eviction methods!

Adithya Bhaskar @AdithyaNLP

3 months ago

New paper cutting through the thicket of KV cache eviction methods!

2 36 233 31K 194

Download Image

0 1 17 1K 1

Alex Zhang @a1zhang

3 months ago

Can GPT, Claude, and Gemini play video games like Zelda, Civ, and Doom II? 𝗩𝗶𝗱𝗲𝗼𝗚𝗮𝗺𝗲𝗕𝗲𝗻𝗰𝗵 evaluates VLMs on Game Boy & MS-DOS games given only raw screen input, just like how a human would play. The best model (Gemini) completes just 0.48% of the benchmark! 🧵👇

23 77 544 122K 233

Download Video

Aman Sanger @amanrsanger

4 months ago

Claude Sonnet 4 is much better at codebase understanding. Paired with recent improvements in Cursor, it's SOTA on large codebases

32 44 860 92K 114

Download Image

Kilian Lieret @KLieret

4 months ago

Massive gains with Sonnet 4 on SWE-agent: Single-attempt pass@1 rises to 69% on SWE-bench Verified! Sonnet 4 iterates longer (making it slightly more expensive) but almost never gets stuck. Localization ability appears unchanged, but quality of edits improves.

4 14 85 9K 35

Download Image

Ofir Press @OfirPress

4 months ago

Great results from the Claude team- the 80% result is pass@1!! They ran the model in parallel multiple times and had an LM judge pick the best patch to submit.

5 7 120 15K 14

Download Image

Alex Wettig @_awettig

4 months ago

Big arrow time! We can make huge progress on open-source SWE agents by scaling up the creation of virtual coding environments 🚀

John Yang @jyangballin

4 months ago

Big arrow time! We can make huge progress on open-source SWE agents by scaling up the creation of virtual coding environments 🚀

24 138 655 99K 377

Download Image

0 4 17 3K 4

Cursor @cursor_ai

4 months ago

Cursor is now free for students. Enjoy!

2K 4K 42K 11.2M 15K

Xindi Wu @cindy_x_wu

4 months ago

Introducing COMPACT: COMPositional Atomic-to-complex Visual Capability Tuning, a data-efficient approach to improve multimodal models on complex visual tasks without scaling data volume. 📦 arxiv.org/abs/2504.21850 1/10

6 47 158 52K 75

Download Image

John Yang @jyangballin

4 months ago

@ weekend warriors - DM me a GitHub repo that you like / maintain, and I'll train you a 7B coding agent that's an expert for that repo. Main constraints - it's predominantly Python, and has a testing suite w/ good coverage. (example of good repo = sympy, pandas, sqlfluff)

19 8 116 17K 66

Jacob Springer @jacspringer

5 months ago

Training with more data = better LLMs, right? 🚨 False! Scaling language models by adding more pre-training data can decrease your performance after post-training! Introducing "catastrophic overtraining." 🥁🧵+arXiv 👇 1/9

17 184 819 162K 655

Download Image

Alisa Liu @alisawuffles

6 months ago

We created SuperBPE🚀, a *superword* tokenizer that includes tokens spanning multiple words. When pretraining at 8B scale, SuperBPE models consistently outperform the BPE baseline on 30 downstream tasks (+8% MMLU), while also being 27% more efficient at inference time.🧵

94 328 3K 363K 1K

Download Image

Logan Engstrom @logan_engstrom

6 months ago

Want state-of-the-art data curation, data poisoning & more? Just do gradient descent! w/ @andrew_ilyas Ben Chen @axel_s_feldmann @wsmoses @aleks_madry: we show how to optimize final model loss wrt any continuous variable. Key idea: Metagradients (grads through model training)

9 32 175 33K 143

Download Image

Zhiyuan Zeng @ZhiyuanZeng_

6 months ago

Is a single accuracy number all we can get from model evals?🤔 🚨Does NOT tell where the model fails 🚨Does NOT tell how to improve it Introducing EvalTree🌳 🔍identifying LM weaknesses in natural language 🚀weaknesses serve as actionable guidance (paper&demo 🔗in🧵) [1/n]

5 92 264 61K 147

Download Image

Jeremy Bernstein @jxbz

6 months ago

I just wrote my first blog post in four years! It is called "Deriving Muon". It covers the theory that led to Muon and how, for me, Muon is a meaningful example of theory leading practice in deep learning (1/11)