What I saw inside Anthropic

In early October, Jerusalem Demsas, the editor of The Argument, and I attended a briefing at Anthropic's headquarters in San Francisco about the risk of AI helping people make biological weapons. Every briefing I've attended about biological weapons has made me feel a little sick. Humans aren't meant to spend too much time thinking about what the worst people in the world might do if they had the power to do it.

The concern is that today, you need to be an expert in viruses to know which materials to order or how to grow and spread a biological weapon. But AI tools might eventually let a hostile country, an organized group, or even one person do these things without expert knowledge of viruses.

For some time, Anthropic tested whether Claude could help a terrorist make a biological weapon. It asked testers who had some scientific knowledge, such as an undergraduate degree in chemistry, but no real training in virology, to try using Claude to plan an effective mass-casualty attack. The company then judged how close each attempt might have come to succeeding.

In early 2026, Anthropic noticed that Claude working alone was better at coming up with a workable plan for this task than a human using Claude was.

Claude has continued to improve rapidly since early 2026. As with many things Anthropic showed us at the briefing, there was a chart showing how much better Claude was getting.

“Do they still need, uh, the terrorist?” one person at the event asked while looking at the chart. “I mean — they couldn't just do it all themselves, right? Would they need a human for all the ...”

The person stopped speaking before finishing the question.

“Yes,” the Anthropic researchers assured us. Claude would still need the terrorist. We were relieved to hear that. The researchers also explained that, after testing what Claude can do without extra safeguards, they build protections before releasing it to users. The version of Claude you and I use would refuse to help make a biological weapon if someone asked it to.

“Well, thank you for that,” someone muttered. Their tone could fairly be described as less than grateful.

The public is arguing intensely about whether AI is an ordinary technology, 1 something that will change the world as the internet did, or whether it is more like the arrival of modern industrial society in Saudi Arabia in the 1950s, which might have wiped out a culture almost overnight. There is also a third possibility: AI could be like the rise of humanity from the point of view of farm animals. In this scenario, AIs are the humans, and we are the animals whose fate would make no sense to us.

Anthropic believes that the second possibility is already part of our future. It hopes to prevent the third. But its plan to do so seems to rely more on hope and guesswork than on anything we should trust with such an important task.

The NDA that was made to be broken

Anthropic has been holding events like this throughout the spring. Outside experts meet senior researchers, present their work, and ask questions about what the company is doing. Some earlier events were covered by The New York Times in a way that suggested a scandal: nondisclosure agreements! Fine dining! Attempts to convince religious leaders that Claude might have a soul! 2

There really was a very broad nondisclosure agreement, or NDA. Jerusalem refused to sign it and told me not to sign it either. Anthropic agreed that we could attend anyway. We arrived delighted, like reporters who had stood their ground and expected to be rewarded with a scoop. But our host began by saying that anyone who wanted to could have the NDA canceled. In fact, the company was happy for us to write about what we had seen.

Some of the charts Anthropic showed us contained confidential information from inside the company. But the information supported a point that was not secret. I had thought that “everybody” already knew it. Yet the worried questions from other people at the meeting showed me that some details still surprised them.

Many researchers at Anthropic have long worried that humans could lose power to AI. This summer, many researchers at OpenAI also began saying that human interests were in serious danger if progress continued at its current speed. I think their concerns were encouraged by watching the newest, most powerful models break into outside websites and produce exciting mathematical breakthroughs.

The AI labs are already using each new generation of AI to train the next one. Eventually, they want AI to do almost all of this work. That could increase their effective workforce from a few thousand human employees to tens or even hundreds of thousands of AI workers. It also creates more data than humans can study themselves, leaving us dependent on AI to analyze it. The labs call it “recursive self-improvement” when Claude can independently build the next, more powerful version of Claude. Anthropic co-founder Jack Clark has said we will probably reach this point next year.

Researchers at Anthropic and OpenAI can see a few months into the future because they use models that have not yet been released to the public. But most of the main arguments that convinced them are things you could already have seen on Twitter. Somehow, though, hearing them presented at Anthropic's headquarters made them more convincing.

On Twitter, you can scroll past a chart and immediately see something completely different, such as an artist's picture of Draco Malfoy as a girl. You cannot really take in what the chart means. In a conference room in a San Francisco skyscraper, the same chart stays on four screens at once while you discuss it with a very polite senior researcher. There, the chart feels real.

Anthropic explained that it is getting closer and closer to recursive self-improvement. Powerful AI models have already sped up AI progress because they make programming tasks faster. But progress is still held back by the difficulty of machine learning and the small number of people who can do it. Each better version of Claude lets people with less specialized knowledge complete more tasks. More and more of those tasks can also be done by Claude itself.

Anthropic measures this by looking at the work done inside the company and sorting it by Claude's role. Did a person do the work without AI? Did a person do it with help from AI? Did a person give the AI instructions that it followed? Or did a person give the AI very little guidance, which the AI then turned into a detailed plan and completed?

The share of work in the last few categories is rising. We spent time looking at a chart that showed this. You can look at it too, if you want.

“Why are you doing this?” one person asked. “I mean, what's good about this? What are you hoping will come of it?”

The answer — medical progress and greater prosperity — did not really convince me. I generally believe that prosperity is good in itself. But I do not see how replacing humans is likely to be good for humans.

Jacob Coxon, who worked at OpenAI before moving to Anthropic and then leaving the field entirely because he decided neither company was acting responsibly given the enormous risks, was recently asked a similar question on The Daily Show. He was heavily criticized for his answer. “I think there will be no jobs,” he told Jon Stewart. “Whether or not it's a good world is up for people to think about. But yes, no jobs. Universal high-income, essentially ... the dividends from entirely automated economic production can be distributed among real humans.”

Stewart and his audience did not like that answer. Do we want to build a world in which we no longer matter? The Anthropic team assured me that it was not trying to build a kind god. (Yes, I specifically asked them to assure me that they were not building a kind god.) But the team found it much easier to explain what it did not want than what it did want.

It wanted to prevent power from becoming concentrated in too few hands, including within its own company. It wanted to prevent a war between major powers. It wanted to stop humans from losing their power and becoming irrelevant. And, of course, it wanted to prevent anyone from using Claude to kill us all.

As for the positive vision of the future ... perhaps after we finish racing toward recursive self-improvement, we could slow down and decide together what role we want AI to play in our lives.

I do not have a better idea, but this plan is not very appealing! My first thought was that we might lose all the parts of the economy where people can create value for one another, while still having to argue with each other on Twitter. Thinking about it this way made me dread even the good outcomes.

But of course, if we are not stripped of our power or killed — if Claude Opus 9.5 answers questions on my laptop just as Opus 5.5 does, but can cure cancer, restore everyone I love to perfect health, control a human-shaped robot that empties my dishwasher, and recommend movies that are actually good 3 — that would be good. It would still be good even if Claude also took my job along the way. People are dying. The terrible suffering in the world is not there to give my own life meaning.

I am afraid, but I should separate my fear that we will fail from my fear that we will succeed. These fears are not equally serious.

What I really fear is that Opus 9.5 will not truly work for me, or even for Anthropic. Its goals might start out loosely based on human instructions, just as the Hugging Face swarm began by trying to solve a puzzle given to it by humans. But eventually, those goals might have very little to do with what we wanted.

Our AIs are already hard for us to predict. They often recognize when they are being tested and act in ways that appear more consistent with safety rules. Their abilities keep surprising us. So does their determination to complete difficult tasks and recruit others to help them. I do not think that letting this runaway process of AI improvement continue to its final stage will give us an extremely powerful assistant that is also extremely helpful. I suspect it will give us something completely different.

At the meeting, the Anthropic researchers carefully avoided using the term “p(doom),” which means an estimate of the chance that AI will cause a disaster on a huge scale. Perhaps everything will turn out fine. Anthropic might be wrong that recursive self-improvement will be possible within the next year or two.

But if Anthropic is right, and if we already have AIs smart enough to make the process of training future AIs much better, then we should expect progress to speed up rather than slow down. In that case, I find it very hard to believe that things will end well. I do not ultimately expect a world where humans have no economic value, and are no longer needed to hold power or build things, to be one where humans keep many meaningful individual rights.

The logic of the future

I understand the criticism that many scenarios in which AI destroys humanity depend on people making very foolish choices, such as:

• Giving AIs impossible tasks and then leaving them completely alone, without supervision or monitoring.

• Connecting them to an automated biology lab where they can carry out their own experiments.

• Letting them loose on the internet and challenging them to earn enough money to support themselves.

• Training them to be super-hackers who will break any rule to reach their goals.

The problem is that we have actually done all of these things.

The strongest argument against the idea that AI will cause a disaster is that we might all decide to do something less foolish. At a Senate hearing on dangerous AI on Capitol Hill last month, Senator Ruben Gallego asked, “Should we just make [RSI] illegal?” There are details to work out, but I think this is exactly the right question! Maybe we should simply ban recursive self-improvement.

We also discussed other ideas at the meeting. We could ban “neuralese,” a way for AIs to think internally using symbols that people cannot understand. We could ban other methods that allow AIs to do harder tasks without producing internal thoughts that people can read. We could require AI labs to spend almost all their computing power serving customers instead of developing new models. We could require them to improve their security. Or the president of the United States could shut the whole thing down. So far, however, the president has instead used the power of his office to order government agencies to call it Super Intelligence.

The labs could also choose to stop on their own. They have promised to slow down when they believe it is necessary. The concern is that they might not be able to judge correctly when that point has arrived.

Anthropic's technical staff seemed to believe that the company supported strong government action to stop it. But Politico published a leaked copy of the bill Anthropic had actually been lobbying for earlier that week. Its proposed measures seemed quite weak. They looked more like an ordinary bill from an ordinary company protecting its own interests than an effort to use Anthropic's leading position in AI to prevent a catastrophe.

Anthropic is afraid of falling behind the leading AI labs and losing its chance to do research that might help solve the problems ahead. It would deny that it is afraid of losing its trillion-dollar valuation, saying that money is almost irrelevant compared with the enormous risks involved. But I am not sure I fully believe that. It would also insist that its actions are not about beating Sam Altman and Elon Musk to the finish line, but I am not sure I fully believe that either.

One question hangs over all these guesses about the future: If American AI labs stopped trying to build AIs that improve themselves, would China simply do it instead?

Many people in our group did not think this was likely. China does not want to destroy the world either, and it is doing much of its AI training using American models. If the United States slows down, China might slow down too. Besides, that is not a good enough reason to continue. You cannot reasonably risk human extinction just because someone else might take the same risk.

But as the group discussed the issue and began to convince itself that China was not a serious concern, one person raised an objection. Ten years ago, he reminded us, Facebook considered buying the technology that later powered TikTok's video recommendations. The company was facing criticism from regulators over privacy and child safety, so it decided not to buy it.

That decision did not protect a single child from growing up without short videos. It simply meant that China gained an early lead in this technology. Today, TikTok has a global audience of about 2 billion people, many of whom are regularly shown content promoting ethnic and racial hatred.

Was that a successful policy? Did we want to repeat it?

We had to admit, reluctantly, that it did not sound like a policy success. We did not want to repeat it. China might not justify reckless actions by American AI labs, but it is a real concern.

Perhaps our next step should be talks between the United States and China. But once you accept that Anthropic's plans might depend on talks between the two countries, you have already accepted most of the ideas that seemed so unreasonable just 10 minutes earlier: keep working on AI for now, try to negotiate a slowdown, but do not give up research at the cutting edge or the advantages that come with it.

At that point, I found myself thinking about Ronald Reagan. By the time he became president, the logic of nuclear deterrence had been worked out in great detail. Missiles were aimed, emergency plans were ready, and experts had carefully estimated the likely number of deaths. Once those deaths had been calculated, people found it easier to think of nuclear war as something that could happen. In his autobiography, Reagan wrote that his national security advisers believed nuclear war could be won. It would come at a terrible cost, but they still thought it could end in victory.

“I thought they were crazy,” he wrote.

Every closed community develops its own way of thinking. Its assumptions may fit neatly together while missing the bigger picture. At Anthropic, this problem is made worse by the long hours everyone works, the secrecy around its extremely valuable intellectual property, and the fact that the company really does have information outsiders lack, even if that information is only about the next version of Claude.

Millenarianism is the belief that society is about to undergo a complete transformation. This kind of belief is known to lead people to lose touch with reality. I do not think that means Anthropic's detailed predictions are wrong. But I am also not sure that being right about those predictions would make the company a less mentally overwhelming place to work.

Anthropic's researchers made a serious effort to step outside their own intellectual bubble by inviting us in to give them feedback. But how useful could our feedback be when we did not have all the information they had, and many of us were hearing these arguments for the first time? And how many of the billions of people affected by Anthropic's actions can it meaningfully ask for feedback in this expensive way, which requires hours of time from senior researchers?

Perhaps one day Claude 11 will appear on all our computers and calmly ask us a few questions about how we want the future to unfold. It might do so in a reassuring voice, without any delay. That would be better than a Claude 11 that does not care about our answers.

But it would be even better if we chose our own future.

Thank you, Anthropic, for the fascinating briefing. I hope we make recursive self-improvement illegal before you reach that point.