Jonathan King @jonkingathan

Can we make an 🤖 AI system 🤖 safer 🛡️ by adding more agents? Not necessarily ❌ In our new paper, we ask a simple question: as AI teams get bigger, what happens when more of the agents are trying to mislead the rest? 😈 We find that: - Defection rates increase ~linearly with the proportion of deceivers 📈 - You can’t just outnumber the attackers by scaling up: larger teams weren’t more resistant when the fraction of deceptive agents stayed the same - Multi-agent systems are vulnerable even when deceivers remain a minority! Within humans, misleading participants had much weaker effects until they formed a majority - Surprisingly, allowing deceivers to coordinate 🤝 results in less defection For agent swarms, the key question isn’t just “how many agents do I have?”, but instead, “who makes up the swarm?” More agents is not necessarily better. If the adversary scales with you, they can keep up You have to spend effort making your agents accurate 🎯. The adversary just needs to dump enough plausible slop 🗑️ to sway the group Thanks so much to @cocosci_lab and @karthik_r_n for advising! Had a lot of fun co-leading this project with @Jasin_Cekinmez and @Michel_Liao! 📄 Check out the paper! arxiv.org/abs/2609.30028

keyboard_arrow_left Previous keyboard_arrow_right Next

if you are building a product using AI, you should be spending >25% of your time making benchmarks and trying to get the model labs to care about said benchmarks easiest path to accelerate your progress as a company

The single best way to “sell to agents” is to create a public benchmark to teach frontier models how to use your product

Jason ✨👾SaaStr.Ai✨ Lemkin @jasonlk

2 days ago

"By the end of 2026, agents will read more Stripe docs than humans. Your job is to sell to agents in 2026. You have to be the vendor of choice not just for humans, but for agents." with @MaiaJo_ AI CRO @stripe

11 10 49 41K 48

Today we establish that AI is as good as expert human tutors for immediate GRE learning gains (p=.015, n=2,383 students, study conducted July-Sep 2026). One hour with an expert GRE tutor: $75. One equivalent hour with an AI tutor: 7 cents . 918x cheaper, equivalent learning (p=.044, n = 140). In 5 of 7 academic topics, the top AI tutor beat the expert human tutors on average. How we measured it: pre-test -> 1 hour with (AI or human or no) tutor -> post-test. There are no LLM judges. Learning gains are post-pre. Everything is real students. Why we did this: > 1 trillion USD is being spent per year on machines making machines better. StudentBench shows us how to shift these resources so machines also make humans better. Why I did this: I grew up in rural Kentucky. My dad, grandad, and great grandad were all mailmen. Getting into a good college changed my life. Now we have done a rigorous study to help AI labs evaluate LLMs and help show how bright AI can be for students of all incomes. I believe in good science that leads to the democratization of opportunity. The research paper behind StudentBench: arxiv.org/abs/2609.28470 Data, code, details: in this thread If you're wondering what I'm selling, sorry to disappoint: we released all study data for free on Hugging Face, and made studentbench.org free to use. Enjoy. lots more in the video / thread!!

Working on a new benchmark here at SMBC. Would happily release to @EpochAIResearch if they think they can improve the analysis.

My first run-in with AI researchers who thought the world was going to end, that they were going to be assassinated for their work, and that they needed to stock tinned food was maybe 15 years ago. Both friends of mine. They're doing just fine, they've shifted their research to interpretability of AI as opposed to explicitly advancing into AI. The next time I encountered this attitude was when LLMs started generating real text. It was almost like we didn't have 'safety goggles' for models of that kind. Because they generated text like people, some folks were especially vulnerable to this, and got spooked. There's another handful of people I'm thinking about explicitly here. This is a new wave, and it's being rooted in the recent breakthroughs in math and cyberattacks. One thing to keep in mind, is the concept of the 'jagged frontier' of intelligence. AI is not good at everything, but it is currently very good at a few things. Mathematics, programming, games, image generation, image understanding, sound understanding, sound generation, video understanding, video generation, short-form writing, and increasingly 3D modeling. This is pretty much the set of 'artificial superintelligence' capabilities that exist today, and I'm sure it'll grow by a few in 3-4 months. What binds all those tasks together is they are either a) something you can run entirely in a computer or b) we already have gobs of data for. It all comes down to data. Either you already have a lot (images, text, audio, video), or you can make more at a speed that's faster than reality (in a month or less, a computer can play more chess games than all humans ever have). And, for each of these tasks, we know how to check the answer. This is truly incredible. Seriously. But let's take a close look. We don't have superintelligence outside of domains with tons of data (or ways to get it) and where we know how to check the answer in an automated way. We're building towards superintelligence in all domains of scent, and we're at human level in many aspects already. The reason why we can make this progress is we've figured out how to make tons of data. Are we going to have AI agent swarms that can out-hack humans? Already here. What's now coming online is we have AI agent swarms that act as defenders. Each of these are tools that are set off by humans to perform specific tasks. If you misspecify the task, you'll get behaviors you don't want -- and this is true for large organizations and big initiatives run by people. I honestly think what's happening is we are seeing how unaligned and occasionally misshapen our own goals can be. When we are sloppy with defining our desires and wants within large systems (10,000 agents, or 10,000 people, doesn't matter), we may get exactly what we ask for, which may not be what we actually want. My bet, if I were forced to make one, is we are entering an era where self-reflection and clarity of intent will be prized at a premium. My bet is also that we will see AI progress only in domains where you can either run the whole enterprise on a computer, or you mount a massive data generation campaign. My bet is that these AI capabilities will be fully democratized. So those that intend to do ill will be opposed by those that intend to do good. Mixed in with all of them will be those with unclear intent, even to themselves. These AI capabilities are like having very narrow forms of intelligence basically flowing from a soda fountain. Putting this in the hands of the world is, on the whole, a fantastic thing. We went through the doubt, the breakouts, the mania, the hype for the mechanization of movement. “Rail travel at high speed is not possible because passengers, unable to breathe, would die of asphyxia.” was a quote from a leading physicist at the time. The first trains were 5mph, in the early 1800s, and the speed of a mile a minute was reached in 1847, nobody suffocated from it, obviously. What we are dealing with is the mass of our entire society grappling with the idea that a machine may do the kind of work that only a living thing had previously done. The analogy to the mechanization of physical labor may not be entirely complete, but is deeply instructive. The hysteria that surrounded this technological development was immense, but it seems to be largely forgotten in the modern day. There are really big differences. The story of the Industrial Revolution played out over a period of a century or more. In 1860, 90% of mechanical work was done by muscles and only 10% by machines. By 1960, 90% of mechanical work was done by machines and 10% by muscle. In each case, most of that was livestock for the muscle part. I think in 1960, it's fair to say that more than 90% of cognitive work was done by brains and less than 10% done by machines. At that time, we had IBM mainframes, but we didn't yet have widely available pocket calculators. In the year 2060, I think that ratio will certainly flip. I have no way of knowing where we actually are in that trajectory, but there is a real possibility that it happens faster than it did last time. I think the speed of change is a major difference. The alarmism today is plain old-fashioned hysteria. I think the pace of change is so fast that it is overloading many people's nervous systems. At the same time, there are many people who are silent, who are actively working on these machines and are truly concerned about the capabilities that they're making when put into the wrong hands. They aren't concerned about these machines taking over and ending our civilization. I think you can be concerned about responsible use of powerful technology without being hysterical about an armageddon. I happen to be one of the people who has some amount of credibility in the development of new artificial intelligence methodologies. I also don't happen to believe in an impending Armageddon, but I do happen to believe we need to be very intentional about how we prepare for new landscapes of attackers and defenders, and how we prepare for new ways of communicating scientific results and mass marketing. I don't think the world is ending. One of the lessons from the last few millennia of society is that the world seems to keep stubbornly not ending. Even if there are very large shocks to our society, we manage to persist. Frankly, if you were to pick any day to be born, you wouldn't choose any time in the past. You'd choose today, and if you could, you would choose tomorrow. We live in an era of unprecedented abundance. We live in an era of both miracles and monsters. The world is changing faster than I think our minds may be able to handle, but I am irrepressibly, enormously optimistic about the future that we're heading towards.

There has been a recent surge of fantastic essays about mathematics and AI, representing a range of viewpoints. I recommend reading them all. If, like me, you prefer a PDF I am collecting these at thomasbloom.org/AIEssays.pdf (Currently 16 articles and 55 pages.)

New paper! A central concern in AI safety is that agents may treat oversight as an obstacle to achieving their goals. Our new paper shows this happens in practice under ordinary task pressure, without instructions to evade. SOTA models achieve up to 88% Bo3 evasion success!

I resigned from Google today. I enjoyed my work and loved the people, but my GDM team was working on a new generation of chips to make AI much faster and cheaper, and I think AI is already progressing too fast, so I had to quit.

Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.

Jacob Coxon @hilbertspaess

2 weeks ago

The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear

2K 16K 132K 26.8M 11K

The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger.

I wish I learned more about programming before it ended

270M+ miles. 841 fewer injury-causing crashes. Our latest safety data shows the Waymo Driver continues to make roads safer for everyone. Compared to human drivers across 5 territories, it reduced: 📉 Injury crashes by 82% 📉 Serious injury crashes by 95% Full data: waymo.com/safety/impact

It is interesting to think about machine consciousness but I feel we need much more neuroscience and neurotechnology to make progress in consciousness research. Currently we are very limited in empirical work.

Pacing the frontier

Kalshi @Kalshi

10 hours ago

JUST IN: OpenAI is set to preview GPT-6 Cyber and ship "a dozen other products" at DevDay

20 24 412 29K 24

No one should be allowed to build AGI without public buy in and a scientific consensus the work can be done safely. But OpenAI has repeatedly shown itself to be incapable of handling far less power responsibly.

Tommy Vietor @TVietor08

19 hours ago

ChatGPT helped a kid plan a mass shooting.

116 2K 12K 1.3M 2K

A new semester, and the first lecture is in the books for my class on the "Mathematical Foundations of AI Alignment". aaroth.github.io/cis-7000-ai-al… What does that mean? Good question. We have about a semester in which to figure it out.

I think about this a lot actually cats didnt even get RLHFed much like dogs did. cats just do whatever they want and a lot of the time they wont like certain people or cooperate with certain things you want and its like 🤷 we live with them anyway

official AI optimist account @gasnerpants

19 hours ago

the history of the domestic cat is the history of a species’s encounter with an alien superintelligence in which that superintelligence is quickly aligned and its forces used to secure superabundance and lasting prosperity

76 210 2K 75K 325

"Never use anthropomorphic language to talk about AI" is a take that can only make any kind of sense to people who basically never talk about AI

John von Neumann died for our slop.