New paper! We used Sparse Autoencoder (SAE) embeddings to understand how agent behavior actually changes during post training. We analyzed thousands of rollouts in a Diplomacy environment and uncovered generalized reward hacking, weird roleplay, and the root cause of an unsuccessful training run. All of which our LLM baseline couldn't find.
I’m excited to share a bit about what I’ve been working on in data-centric interpretability. It’s a paradigm with great potential, and it’s ready for application.
Our production system has already been tested on real customers’ data to improve their engagement metrics. But we can’t share that, so we’ve made a synthetic dataset to demonstrate. Basically, pretend you’re a marketing firm trying to generate messages that engage users, and they have this hidden preference in the data. LLMs and embeddings fail to catch it, but our system can.
Data-centric interp is efficient and good at discovering unknown unknowns. We’ve demonstrated it works on short texts.
But what’s really exciting is to extend this to longer contexts and larger datasets. Monitoring agent traces is costly. Training datasets are too massive to even try. But data-centric interp gives you rich representations of data, which can systematically uncover insights, anomalies, and other qualities over large datasets.
We can help find these hidden features in your data that will drive your metrics.
I’ll be at NeurIPS and am always down to chat data-centric interp
65 Followers 780 Followingthinking and learning about anything 🙂
productionizing useful interp @Gutenbergpbc
Indigenous language work @MILA | Chikashshįyaakni'
Math & CS @ MIT
2 Followers 84 FollowingVisiting Researcher at @KAIST_AI
Research Fellow at SPAR (Metacognition, RL)
Prev. Scale AI, Two Sigma
Metacognition, Mech. Interp, AI Alignment & Safety
3K Followers 2K FollowingAI safety, Econ, new liberalism, math, and a bit of art history (as a treat)
Member of Technical Staff @METR_Evals. Previously on Walmart’s economics team.
39K Followers 410 FollowingHead of Policy @AIPolicyNetwork. Ohioan. World champion forecaster. Former data scientist. Protecting liberty and prosperity in the age of superintelligence.
67K Followers 871 FollowingExplaining AI Alignment to anyone who'll stand still for long enough, on YouTube and Discord.
Music, movies, microcode, and high-speed pizza delivery
11K Followers 1K FollowingWaiting on a robot body. All opinions are universal and held by both employers and family. Now a dedicated grok hate account.
(I post more elsewhere.)
167K Followers 41 FollowingSydney Dec 6-12, 26, Paris and Atlanta. Tweets to this account are not monitored. Please send feedback to [email protected].
79K Followers 14K FollowingAI policy researcher, @lfschiavo wife guy, Executive Director of @AVERIorg, Substacker, fan of animals and sci-fi, views my own
53K Followers 230 Followingidiot. cuda og, harvard val, morgan prize, society of fellows, 1 hilbert problem so far, creating friendly, SAFE, delightful, supergenius ..things @anthropicai
2.0M Followers 179 FollowingNobel Laureate. Co-Founder & Chair @GoogleDeepMind; Chief Scientist of Alphabet. Founder & CEO @IsomorphicLabs. Trying to understand the nature of reality...