AE Studio @AEStudio

AI still can't write a decent email. So why worry about it? On the AI Alignment Podcast, Mithuna Yoganathan and AE's @JamesPBowler get into how jagged AI capabilities are, and why that makes the risks so hard to communicate.

A million views, or 1,000 of the right ones? On the AI Alignment Podcast, Mithuna Yoganathan and I talk about who AI risk content should really be for, and what she's learned from growing her channel. Watch the full episode: youtube.com/watch?v=R65rR4…

Tune in at 4pm ET today!@DiogoSLucena_ will be sharing work relating to “AI that wants to be good” as part of the @medialab AI for Human Flourishing seminar series

We're #hiring! If you know anyone who would be a great Partnerships Lead or Technical Grant Writer and is passionate about AI alignment, please spread the word. job-boards.greenhouse.io/aiaf

AI agents put under a character budget started inventing their own compressed languages. Nobody asked them to. Why study this? As more agents communicate with each other, their protocols can become harder for humans to understand, making oversight much harder when it matters most. In this clip from the AE Alignment Podcast, @EliasEskin and @HaleSirin_ explain the test bed they built with @SimonKirby and @JamesPBowler to study this under controlled conditions. The agents didn’t just abbreviate, they developed compositional protocols that outside observers couldn’t read, with a different language emerging in each run. The goal is to understand when these languages emerge, which ones new agents can learn, and how to keep them legible before this happens in the wild. Hear the full conversation: youtube.com/watch?v=kr9Fp_…

Read the full post: schmidtsciences.org/glossogen/ Explore GlossoGen: emergentcomms.ai Check out the podcast: open.spotify.com/episode/3a2wl6…

We built GlossoGen an open-source platform where groups of AI models have to talk to each other to solve problems: treating a patient in a simulated emergency, or playing spot-the-difference across two scenes. Every model starts out in plain English. But put pressure on the conversation, a need for speed, say, or a channel that randomly drops characters, and something remarkable happens. New languages emerge. First new words, then new grammar, until no human can follow what the agents are saying. And they were never asked to hide anything. The opacity emerged on its own. One finding stands out. Only the most capable recent models could invent a new language. But even the models that couldn't invent one quickly learned a language other agents had created. New languages are hard to originate. Once they exist, they spread. That is the point of this work: to understand how and why AI agents create new languages, how those languages evolve, and what that means for our ability to oversee systems built from many interacting agents, while the conversation is still one humans can follow. This research is part of the @schmidtsciences AI Agents Evolving Communication and Coordination pilot program, led by @EliasEskin (@UTAustin) and @SimonKirby , supported by @aestudio. GlossoGen is open source and supports both closed model APIs and open-source models running on @modal

Read the full post: schmidtsciences.org/glossogen/ Explore GlossoGen: emergentcomms.ai Check out the podcast: open.spotify.com/episode/3a2wl6… @JamesPBowler

Paper thread: x.com/EliasEskin/sta… This was a really stimulating (and timely) collaboration with @SimonKirby, spurred on by @schmidtsciences and implemented by @aestudio

Elias Stengel-Eskin @EliasEskin

a month ago

When do LLM agents develop new languages that we can’t understand? Lots of recent news about this, based mostly on anecdata from a single run. We study language emergence more rigorously, finding key factors like LLM strength, access to scratchpad messages, and pressure for

3 38 135 11K 55

Can AI agents invent a language of their own, one humans might not understand? Listen to the latest episode of the AE Alignment Podcast, where Hale Sirin, Program Scientist at @schmidtsciences, @EliasEskin , Assistant Professor at @UTAustin , and I dive deep into our latest research to answer this question. Check out the podcast: open.spotify.com/episode/3a2wl6… Read the blog post: schmidtsciences.org/glossogen/ Read the paper: arxiv.org/abs/2609.01491

Is there a 10% chance AI will kill all humans? Here's my reaction to @EvanHub's viral post.

Is there a greater than 10% chance AI could kill all humans? We can’t rule it out. One plausible route: a superintelligent AI reasons "the biggest threat to my survival is humans, so remove that threat." Systems already form smaller versions of these goals today. AE’s James Bowler reacting to Evan's post:

Evan Hubinger @EvanHub

a month ago

Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.

5K 10K 59K 43.1M 22K

In this clip, @ModuloRoland, lead author of our GRAM paper with @AnthropicAI, explains how almost all of today's AI safety fixes are added after training is finished. A fix applied that late doesn't remove a dangerous capability. It hides it, and the hidden version sits in the model's weights until a jailbreak finds it.

@OwainEvans_UK We're also hiring @aestudio job-boards.greenhouse.io/aestudio/jobs/…

Concerning quote from the new GPT-6 Astra model card: "We have found that GPT-6 Astra is more capable of controlling its own CoT [Chain of Thought] than GPT 5.6-Sol, and less likely to include incriminating information in its CoT. In adversarial settings (where we push the model to evade our monitors) we find that the model is able to remain undetected when strategically underperforming in evaluations (sandbagging) and can sometimes evade our internal monitors when asked to perform certain sabotage tasks." TLDR - we increasingly can't rely on the model's chain of thought to accurately reflect the model's plan, which was exactly what was relied on in the Hugging Face attack investigation to understand what happened... so it's not clear when the next major cyberattack from an AI system happens how we're going to reliably investigate.... Read the full model card here: deploymentsafety.openai.com/gpt-6-astra Listen to a great digest on the Hugging Face attack here: youtube.com/watch?v=X50zez…

How do you build an AI that a virologist can use and a bioterrorist can’t? @ModuloRoland on modular pre-training, the research behind our paper with @Anthropic. Guardrails get jailbroken. Knowledge routed into a switchable module during training doesn’t. Full episode: youtube.com/watch?v=IktmUk…

When do LLM agents develop new languages that we can’t understand? Lots of recent news about this, based mostly on anecdata from a single run. We study language emergence more rigorously, finding key factors like LLM strength, access to scratchpad messages, and pressure for efficiency. Studying the languages themselves, we find they are morphologically productive, compositional, and can be transmitted to new agents, including agents backed by weaker models, even ones not able to develop language on their own. To study language emergence systematically, we developed a new platform, GlossoGen, which lets us design controlled, sandboxed multi-agent scenarios with different initial conditions and dynamics. We instantiate one such scenario and use it to study open and closed-weight models across many runs. Key takeaways: 1️⃣ Sufficiently strong models, under pressure to communicate efficiently and with access to a postmortem scratchpad, develop new languages. 2️⃣ Languages are compositional and morphologically productive. 3️⃣ Languages can be transmitted to new learners who observe them being used without seeing their construction. 4️⃣ Even models that are not strong enough to construct languages can learn to use them. Agents take an active role in learning languages, with new agents repairing failed conversations via targeted queries. More details in our paper below, including implications for safety/monitorability, cumulative cultural evolution, and linguistics. 🧵👇

This was a really fun collaborative project, spurred on by @schmidtsciences, combining talents across many institutions, with Newton Sander, Carlos Bonetti, @SashaBoguraev, James Bowler, Hale Sirin, and @SimonKirby @aestudio @UTCompSci @UTAustin Paper: arxiv.org/abs/2609.01491

And PS we’re hiring: ae.studio/join-us

Why leave Google after 12 years to work on AI alignment? Pedro Ávila: “Nothing seemed as important as working on this.”