Does intelligence need a hard cap?

Calls for a new kind of slowdown were the talk of The Curve. PLUS: Slinking back to X

(Shutterstock)

This is a column about AI. My fiancé works at Anthropic. See my full ethics disclosure here.

Today let’s talk about a new front in the conversation about whether to slow the pace of AI development: an emerging push to consider limits on just how smart a large language model can be.

I spent the past weekend at The Curve, an annual conference that brings together executives from top AI labs, leaders of nonprofit organizations, government officials, and a small contingent of journalists. Inside a converted hotel in Berkeley, the group spent a few days loudly debating questions of economics, politics, and safety. And while the conference has had one eye on existential risk for each of the three years it has existed, the questions felt notably more urgent this year.

I see two explanations. One is the ongoing fallout of the OpenAI-Hugging Face incident, whose implications have alarmed people across the industry. The other is recent blog posts from OpenAI and Anthropic outlining their progress toward recursive self-improvement: AI systems that can research and train their successors, leading to ever-faster releases and increased risk that the systems will escape their control.

At The Curve, I heard multiple speakers say we may want to limit how intelligent a future system is allowed to become. Depending on how far labs take any restrictions, the result could be a de-facto ban on systems reaching superhuman intelligence.

Whether that seems meaningful to you depends on whether you believe recursive self-improvement or superintelligence is even possible given the architecture of the current models. And a lot depends on how you would define and assess “intelligence” here. But it was the most striking idea I heard at a conference full of noteworthy discussions, and its airing in the relatively friendly confines of The Curve may be a preview of a more public discussion sometime soon.

Over the weekend, though, these comments were made at sessions being conducted under the Chatham House Rule. Accordingly, I can’t identify the speakers. But the fact that there appears to be agreement on this subject among speakers at The Curve struck me as newsworthy.

So what does it mean to place limits on intelligence? The speakers I heard not offer many details, though we can gather clues from other remarks made recently by AI executives.

For one, Anthropic CEO Dario Amodei has called for “some kind of ‘speed limit’” on recursive self-improvement. For another, the company’s “responsible scaling policy,” which has been copied in some form by most of its leading rivals, seeks to impose limits on the training and deployment of more powerful systems as they develop new capabilities.

Other approaches could involve restrictions on using frontier models to conduct AI research; limits on how much compute is available to systems and how many copies of itself a system is allowed to run; or preventing labs from deploying models past a certain level of capability. (That last one is arguably already in action.)

Notably, though, any effort to restrict models’ advancement in this way would require enforcement capabilities that don’t exist yet. Individual labs can’t impose restrictions like this, nor can individual countries, and the current US government actively opposes them.

There are other, easier methods to manage AI’s development: Anthropic has already adopted the idea of embedded evaluators, and OpenAI has said it will follow. An antitrust waiver that explicitly allows labs to collaborate on safety questions might be useful enough that it could overcome fears that it would allow leading companies to consolidate their power even further.

Still, speakers'’ comments over the weekend suggested to me that nothing proposed so far meets their own definition of “enough” — including the “morally binding” accord AI leaders signed last week with the president.

For the moment, lab leaders and the US government differ sharply on one all-important question: how close are we to actual danger? The former say we may see a catastrophe as soon as next year; the latter has vacillated between creating a de facto licensing regime for frontier models and publicly encouraging US companies to go even faster.

But the Trump administration remains frustratingly obtuse on the question of where all this is headed, even as more warning signs blink red. A few days before speakers mused about the need to limit intelligence, the president was insisting that everyone start calling it “super intelligence” now. But “super intelligence” is already worth of more than a branding exercise, and the limits that matter most today are among the officials at the White House.

A MESSAGE FROM OUR SPONSOR

Infrastructure for scaling AI monetization

Intelligence is everywhere. Identifying scalable business models is the next frontier. Chargebee delivers billing, metering, and monetization primitives that enable ambitious AI-native startups and enterprises to ship pricing as a product. Learn more.

Slinking back to X

Now let’s talk about something truly unpleasant.

About three years ago, I quit posting on X in protest of the new ownership. For a while there, every day brought some fresh outrage from Elon Musk: inciting dangerous harassment of his former employees; rigging the platform against journalists and their work; posting support for anti-semitic conspiracy theories; dismantling the content-moderation apparatus, and so on.

Mostly I left because I could not stand the thought of being there. I also hoped that I could use whatever minor influence I had to help kick-start new platforms. I encouraged Meta executives to challenge X with a Twitter clone, and have used Threads daily since it launched. I covered the launch of Bluesky closely and have continued to pour energy into it (and the Fediverse) even as they decline. While Musk was suing OpenAI in an effort to effectively destroy it, I repeatedly told Sam Altman to start a rival to X and take his AI posting there.

Whatever virtue these actions may have had, they did not succeed at creating a true rival to X in the category that matters most to me: a forum for daily tech and business conversation. The community of people working in and around artificial intelligence product and policy simply refuses to move away from X, no matter how bad it gets.

Along the way, X was absorbed twice, first into xAI and then into SpaceX. While I never held much hope that my boycott of X would affect it economically, the fact that it is now a small division of one of the world’s most valuable companies made my protest feel even more abstract and quixotic than it was when I began.

I am among those who believe that the next year will be decisive in shaping the future of AI policy, regulation, and safety. And increasingly, attempting to influence the AI policy conversation without participating on X (or Substack) has made me feel like I am fighting with both hands tied behind my back.

And so, in the coming days, I plan to resume posting on X. I am doing so not because I think it’s any better — it isn’t — but because I have exhausted all of the alternatives. I continue to believe that anyone who can avoid posting on X should. But my calculus about the t.radeoffs involved for my own work has shifted: as the stakes of the AI conversation have risen, boycotting X over my disagreements with management has grown both more costly and less effective.

Should the AI crowd ever move elsewhere, I will happily follow them. But I’ve become convinced that it’s important I engage with them, even if doing so makes me feel like I’m coworking inside a Port-a-Potty.

I’m thinking of this as an experiment, and I may abandon it at any time. I’ll continue to post on and otherwise support alternative social platforms. But I also need to spend time on the platform that still — despite my best efforts — matters most to my beat. I’m not going there for a good time, and I hope not to be there for a long time. For the moment, though, I will be there, and I wanted you to hear it here first.

Feedback welcome: casey@platformer.news.