Karen Kwok @KarenKKwok

What does it mean to pace the frontier? Over the last month, @random_walker and I have analyzed the loss-of-control incidents at AI companies to understand what technical and policy interventions can improve safety and what companies should do to pace the frontier. The result is a new 13,000 word essay — our most substantial writing on AI safety since AI as Normal Technology. A summary of our arguments: 1) The polarization between the cybersecurity and AI safety communities is counterproductive. The safety community largely sees these incidents as a crisis for alignment, and worries that these incidents will become more damaging as agents become more capable. Cybersecurity practitioners largely see companies failing to take basic security precautions. We offer a middle ground between these communities as a way forward for improving AI safety. 2) We agree with security practitioners that OpenAI did not take adequate protections for controlling their agents. But this is not just a matter of applying 30-year-old security methods to a new domain. Security for AI agents — AI control — while important, is not a solved problem. While known control methods would have prevented the Hugging Face incident, as agent capabilities continue to advance, we will only be able to control them if we invest adequately in control interventions. 3) We also agree with security practitioners’ implicit position that these incidents are primarily a security story. In the AI safety community, rogue agents are treated as inherently catastrophic because of the assumption that there is an endless list of risks that will arise from their development. We disagree. We have long advocated that the best approach to AI safety is to identify the risks and address those specific risks. Over the last few months, it has become clear that one urgent risk is cyberoffense, because it has unique properties that allow agents to carry it out autonomously. We should similarly invest in defenses against other specific risks, such as biorisk and risks from military AI. 4) We agree with the safety community that there is an urgent need for technical and policy interventions to prevent loss-of-control incidents. But in our view, marginal investments in control are more likely to be effective compared to those in alignment. We view these incidents as illustrating the lack of emphasis on AI control within companies, despite the availability of known techniques. More broadly, there are many common-sense policy proposals that could help promote investments in AI control where we share common ground with the safety community. 5) Organization governance should be a key tool for pacing the frontier. Unfortunately, AI companies are trying to reinvent basic aspects of organizational governance as a problem to be solved by improving the technology. But even developing better control techniques will not be enough if irresponsible individuals or teams within large organizations can choose not to use them. When a single misconfigured RL environment or unmonitored evaluation can cause real-world harm, individual teams should not be able to run potentially dangerous experiments without oversight from legal, security, and other teams. AI companies need processes for reviewing experiments, assigning responsibility for monitoring them, and investigating warning signs deeply before restarting experiments. If putting these processes in place requires pausing some experiments, companies should do so. 6) How should we reason about AI's impact on cybersecurity? It's plausible that advances in agent capabilities upset the offense-defense balance for cybersecurity. We cannot yet be certain, but there is enough evidence that agent capabilities might soon make widespread cyberoffense possible that urgent action is warranted. We discuss potential interventions for tilting the offense-defense balance towards defenders. 7) How our views have evolved over the last year. We take stock of AI progress and share how we have updated our views. In the essay, we did not pay sufficient attention to safety risks that arise during development and evaluation (as opposed to the widespread deployment of models). We were too confident that companies would take basic control precautions and underplayed the importance of jaggedness, which led us to underestimate how quickly capabilities could improve in domains such as cybersecurity. 8) At the same time, many distinctive claims of AI as Normal Technology have held up. In particular, we think recent incidents support our continuity hypothesis — the behavior of "rogue" agents became apparent and widely publicized while they are still incompetent at causing serious harm or hiding their traces. The societal reaction to even the relatively small harms from these incidents has been fierce (and the safety community deserves credit for keeping up pressure on companies). Whether this translates into meaningful changes in companies’ behavior remains an open question, and a test of the usefulness of the AINT framework. 9) In short, we’ve tried to synthesize the AI safety and cybersecurity communities' views into a coherent plan of action: hold companies responsible, invest in control, and strengthen defenses against specific risks.

NEW ODD LOTS - OPENAI PRESIDENT GREG BROCKMAN @TheStalwart and I speak with @gdb about AI safety, what a slowdown in AI development could actual look like, and directly respond to the 'AI doomerism is just marketing' criticism podcasts.apple.com/us/podcast/ope…

What are China’s biggest concerns about AI? A new article by Chen Yixin, China’s Minister of State Security, in the latest issue of China Cybersecurity Magazine offers some revealing clues. Chen identifies six major risks: 1. Regime security. Hostile forces and people with ulterior motives can use deepfakes, AI-generated text and images, and automated online accounts to cheaply and massively produce political rumors, spread harmful information, and incite social divisions—waging “public-opinion wars” and “cognitive warfare” against China and directly threatening its political, institutional, and ideological security. 2. Critical infrastructure. AI can enable countries and organizations to rapidly discover vulnerabilities, automatically chain together attack paths, and carry out sophisticated hacking operations, dramatically lowering the technical barriers and costs of cyberattacks against China’s critical information infrastructure. 3. Data leakage. Foreign intelligence agencies can use AI-powered web crawlers, data mining, and profiling to collect sensitive information, including critical national data, trade secrets, and citizens’ personal information. Chinese users who process sensitive information through foreign AI products or export data abroad could also cause large-scale data leaks. 4. Closed technological ecosystems. Countries with advantages in AI theory, model architecture, and computing power may invoke “national security” to impose technology controls, entity lists, monopolize technical standards, and build closed-source ecosystems, restricting other countries’ access to advanced technologies and fragmenting global AI supply chains. 5. Structural challenges to social governance. AI creates new uncertainties for social governance and public order. Algorithmic “black boxes” and data poisoning can amplify existing social biases; misuse of personal information can trigger crises of public trust; automated decision-making creates difficult questions of accountability; and the rapid development of AI is increasingly outpacing existing laws, ethics, and regulatory mechanisms. 6. A fundamental transformation of warfare. AI is pushing warfare into an era of “intelligentization.” Whoever can use algorithms to achieve more precise sensing, judgment, and targeting will gain the initiative on the battlefield. AI is moving from an auxiliary role to a central one in intelligence fusion, decision support, target identification, combat operations, and cognitive warfare. And what is the solution? Chen’s answer is revealing: strengthen “Party control over the internet” and “Party control over data,” and “transform the advantages of Party leadership into the effectiveness of AI governance.” In other words, from the CCP’s perspective, the central question is not simply AI versus no AI. Nor is it even fundamentally a contest between the US and China. It is a contest between the US and the CCP over who gets to control AI, data, information, and ultimately the future of society. And whoever wins that contest will decide the future of humanity. mp.weixin.qq.com/s/jsc97cKVYOcH…

Sam Altman told me yesterday there will be no IPO for OpenAI in 2026, that it would be “ill timed” given all they have to work on with alignment, control and safety fortune.com/2026/09/12/sam…

FT Exclusive: The US’s second-largest law firm is customising open-weight models as an alternative to OpenAI and Anthropic, a demanding set-up that shows how the technology is changing Big Law’s business model. ft.trib.al/9lQS92l

Today we are releasing new data on how models have accelerated AI research at OpenAI since the start of this year. Recursive self-improvement will be one of the most consequential AI developments of the next few years, and a shared public understanding for how models enter into the AI research process will be a critical part of democratic governance of AI and pacing the model frontier. openai.com/index/research…

Will Huawei catch up to Nvidia by 2030? Almost certainly not. We analyzed Huawei’s chip roadmap and supply chain, from the Ascend 950 to its 3D-stacking bet and domestic HBM supply. Our estimate: Huawei will produce less than 4% of Nvidia's AI compute in 2026. Relying on domestic HBM alone, that share could fall to just 1% by 2028. 🧵

AI models are becoming unknowable. Astra performs better, but it's also better at avoiding monitoring, meaning it could be harder to know what it's thinking or doing. OpenAI CEO Sam Altman separately told Axios this week that models are becoming "superhuman" in some capabilities and that "we are just sailing in unknown waters." More here: axios.com/2026/09/04/ast…

Sources: Anthropic is expected to make its IPO prospectus public late September and complete the listing days before the US midterm elections in November (@DEER_ECHO_ / Reuters) (Visit Techmeme dot com for the link and full context!)

My comment for @Reuters about the OpenAI-agent incident where agents apparently coordinated emergently during a routine web search and analysis task (unrelated to cybersecurity), probing a website for vulnerabilities and exhibiting exploitation attempts, persistence, and coordination. The public web effectively became an undeclared extension of the agent harness. reuters.com/world/europe/o…

Sources: Mira Murati's Thinking Machines Lab is in talks to raise $1B+ at a $40B+ pre-money valuation, lower than the $50B+ valuation it sought last year (The Information) (Visit Techmeme dot com for the link and full context!)

$GOOGL Google Cloud Exec Mark Lohmeyer: Google has accelerated TPU development from one generation every two years to two new platforms annually. “Previously, we were at a pace maybe of a new TPU generation one new TPU generation every two years. Then we accelerated it to one new TPU generation every year. And then in this last round, we actually are delivering two new platforms every year, right, with TPU 8i and TPU 8t."

China Open-Source LLM Tracker US$10-15B of total ARR, of which core LLM revenue (excluding Byte / Seedance) is ~$6-8B. Growth has been phenomenal. Take Zhipu as an example (more data available as a public co.) ARR has grown 6x+ since March, while token usage is up >40x YTD. Not a ton of high quality data, but growth seems to have been accelerating in recent months, esp. since June. Importantly, this growth did NOT come from a price war. DeepSeek raised prices by 3-12x, while usage continued to grow. Zhipu tells a similar story. Best estimate for overseas revenue is already ~$2B. Use Cases Are Moving Beyond Coding: $MiniMax’s H3, post-trained and accelerated on fal.ai, can now generate a 5-second video clip in ~3 seconds. Bilibili disclosed that its AI video revenue has doubled. Detailed analysis per co: robonomics.substack.com/p/china-open-s…

An open letter for a global surge in cyber defense, signed by over 100 organizations including Anthropic, AWS, Google, Microsoft, OpenAI, and Oracle. x.com/i/article/2093…

Writing is thinking. Exporting that work means skipping actual thought. I'm amenable to the idea that sometimes you take the elevator instead of the stairs, but if we all start skipping the thinking part, we're all going to get dumber than we already are

Max Tani @maxwelltani

3 weeks ago

In a statement, WSJ opinion editor Paul Gigot defends Druckenmiller's use of AI to write Monday's op ed, saying "AI is a fact of modern life," and "nobody can doubt that his op-ed is his genuine opinion."

97 122 575 1.1M 191

Sam Altman admits he was wrong on AI's timeline and says society and the economy will adapt more slowly "I thought when we got to GPT-4, which was back in 2023, that very quickly after that there was going to be much more disruption, software businesses up for grabs right away, than it turned out to be." "I think I was wrong about a few things, but one in terms of the speed: the economy just has so much inertia." "People keep doing the same things, buying from the same company, wanting to use their tools the same way. I think this is actually a positive in many ways, and it's going to make this big transition go smoother and slower. I'm grateful for it." "But it means we've all been too ambitious on timelines. Even with this incredible technology, society and the economy will adapt more slowly."

Fireside Alpha @firesidealpha

2 months ago

Sam Altman reveals building Codex was a "crazy kamikaze mission" to beat Claude Code and that most of the best coders he knows now use Codex. "We were way behind Claude Code, and it seemed like a kind of crazy kamikaze mission to try to beat them with a coding app." "The

55 58 1K 2.2M 610

Anthropic IPO filing will show AI backlash as a risk factor, sources say cnbc.com/2026/08/21/-an…

🍦SCOOP: Anthropic could raise more than $100B in its upcoming IPO at a $2T valuation. That would make it the largest-ever public offering! Potential investors have been digging into Anthropic's business on: - how Anthropic's AI models can continue to improve - will Anthropic continue to charge a premium for its models - can it get enough compute? Read more in @nytimes: nytimes.com/2026/08/21/tec… w/ @LaurenSHirsch @eringriffith @MikeIsaac

We’ve got exclusive new polling on local data center development at @heatmap_news. Over the past year, we’ve asked Americans whether they would support or oppose a data center being built near where they live. We haven’t changed the wording. When we first polled the question a year ago, Americans were split 43-42 on support/oppose. When we asked again in Feb, a bare majority opposed data centers. Today, 75% of Americans say they would oppose a data center being built near where they live, and 61% said they would “strongly oppose” such a proposal. It’s a swift, massive shift in US public opinion on a scale that I wouldn’t have once thought possible. More details, including the party ID & rural/urban breakdown, in my new story: heatmap.news/daily/data-cen…

May startle the markets with this but: OpenAI is growing faster than Anthropic, actually. While data through Q2 shows tepid growth for OpenAI vs. Anthropic, Q3 to date shows that OpenAI has surpassed Anthropic in QoQ enterprise growth: 82 vs. 76. Why? GPT-5.6 Sol is really good, increasingly the choice for developers. Fable 5, meanwhile, disappointed both in adoption and real-world application given price + data retention requirements imposed by regulators.