We revisited Independent Component Analysis (ICA), a decades-old statistical method, and found that it works far better for interpreting LLM activations than we expected!
This old baseline deserves a second look.
Across GPT-2 Small, Gemma 2 2B, and Qwen 3.5 9B Base, we find ICA is actually competitive with the public SAEs across interpretability, sparse reconstruction, concept prediction, and targeted probe perturbation, while fitting in minutes per layer and using 7–43× less parameter memory.
We're releasing ICA Lens, which lets you quickly discover, annotate, and evaluate interpretable directions in LLM residual-stream activations. No need to train another dictionary!
More broadly, our results suggest that non-Gaussianity is a useful inductive bias for efficiently finding interpretable directions.
Try it: pip install icalens pypi.org/project/icalen…
Paper: arxiv.org/abs/2606.11722…
1/8
We open source SEA-ASR-ID a fast speech-to-text for Bahasa Indonesian (Apache 2.0 364M params open model + 2,287.4 h speech dataset)
- 1.5 s to transcript 79 minutes of Indonesian parliament (3,582× real time and 26–42x cheaper than Whisper large-v3 on 1x H100)
- 5.5% WER with a 1B rescoring LM (on par with Whisper large-v3 1.5B), 6.1% WER with its n-gram LM (ties Whisper large-v3-turbo)
Finetune on Google's EmbeddingGemma 2 inspire by @TslShahir@stablequan
In reading group reviewing flow models, I realized I didn't understand the change-of-variables rule as fully as I probably should. So I put together this blog post and collection of visuals to help me more fully understand the Jacobian-inverse term. 💪
michaeljohncooper.com/writing/change…
Menurut saya indikasi negara maju itu salah satunya adalah ini:
"Kita bisa neliti hal yang 'tidak berguna' dengan tenang"
'Tidak berguna' = hal yang, oleh awam, terlihat tidak bisa diaplikasikan langsung dan/atau belum ada gunanya saat ini. This applies to stem maupun soshum
super interesting to see anthropic's roadmap in 2021, i think they were very early in everything related to RL/tool use
for the pre-training part, they mention "periodic batched backprop" and "skeletokenization" which might correspond to these two papers
arxiv.org/abs/1711.10563arxiv.org/abs/2105.09352
Anthropic released their 2021 research roadmap earlier this year (because of copyright litigation); I think some people might find it interesting: i.sm4.ca/concise-resear…
I wrote a blog post detailing the writing approach I took while working on my recent paper: sophielwang.com/blog/paper-wri…
I describe the behind the scenes for how this paper came to be, why and how I storyboarded figures before running experiments, and some figure design tips! :)
Can “chicken” make a base model reason? 🐔
Yes!
Why? With the right first tokens, base models can match RL in reasoning. We trace this to learned training data associations and use a simple data edit to make “chicken” elicit reasoning too.
blog: sophielwang.com/cues
There was a time when @nvidia and @Google weren't into AI. There was a time when the big cloud providers didn't have GPUs. There was a time before GPUs and deep learning, and AI, ...
During that time professors and students created the foundation of the AI we have today.
If doomscrolling X is part of your research workflow, we built something for you.
Introducing Paperscrolling, now on iOS and Android 🚀
Follow the latest research with key ideas, figures, and audio explanations from alphaXiv Briefs
Check it out here!
play.google.com/store/apps/det…
Ramanujan wasn’t skipping proofs. He was hiding his chain of thought to prevent distillation.
Hardy: “Show your work.”
Ramanujan: “Sorry, reasoning tokens aren’t exposed via the API.”
900 Followers 4K FollowingLLM Post-Training/ Recursive Self-Improvement for Reasoning
Co-founder/ MTS @ReasonCoreAI
PhD @UCLA, Bachelors @iitbombay
Led a team of ~500 PhD researchers
4K Followers 1K FollowingAI Consultant @amazon @aws | Alumni: @gojektech @CMU_Robotics @itbandung | Tweets are my own and not the view of my employer | ga accept akun profpic ga jelas
2K Followers 1K FollowingLLM theory and interpretability.
Professor at Saarland University @LstSaar @SIC_Saar.
Previously PhD at Stanford @stanfordnlp.
Hiring PhD students and Postdocs
81 Followers 205 FollowingPhD Student doing XAI for NLP at @ANITI_Toulouse, IRIT, and IRT Saint Exupery.
🪄 Interpreto & Xplique libraries dev member.
https://t.co/R75ZM3vrPh
163 Followers 589 FollowingCS PhD student @VT_CS. Previously a visiting researcher @NYU_Courant. I work on AI Security, and Multilingual LLMs. Sapere aude.
339 Followers 608 FollowingAI Safety, NLP PhD student @mbzuai, currently visiting @sutdsg
Prev: @univ_indonesia and JAIST
Only create X because AI Researchers are on X
572 Followers 805 FollowingCS PhD @UMDCS, working on LLMs and actionable interpretability. Prev @UpennNLP @Google
RedNote: https://t.co/offEkZNZic
Opinions are my own
85K Followers 401 FollowingStudent of causal inference, human reasoning, and history of ideas, all viewed through the sharp lens of artificial intelligence.
857 Followers 78 FollowingWe're a part-time, virtual research program that gives students and early career professionals an opportunity to work with professional AI safety researchers.
815 Followers 212 FollowingBy-design interpretability researcher @guidelabsai
driven by a curiosity to understand the inner workings of AI and make it safer
73K Followers 63 FollowingStudent of mind and nature, libertarian, chess player, cancer survivor. @ https://t.co/pioQVVjSXz, UAlberta, amii, https://t.co/vPRUv44glx, The Royal Society, Turing Award
600 Followers 211 FollowingComputer scientist @Yale. KI, der KI erschafft. Ex-farmer in 🇮🇹, ex-teacher in🇨🇭. All tweets are wrong, but some are useful. No need to follow back :-)
81 Followers 205 FollowingPhD Student doing XAI for NLP at @ANITI_Toulouse, IRIT, and IRT Saint Exupery.
🪄 Interpreto & Xplique libraries dev member.
https://t.co/R75ZM3vrPh
2K Followers 6 FollowingInterpretability Researcher @ Google DeepMind, ARENA founder. Guinness World Record holder for the largest number of clothes pegs held in one hand (yup, really)