Explosive Document Claims That AI Is Not What We Think

Many people believe that AI systems, which can have tens of billions of parameters, are impossible to fully understand. Even the researchers who build them may not know exactly how their basic parts, called nodes or “neurons,” work. This is why AI is often described as a “black box.”

But a new essay by AI researcher Nick Cammarata has surprised people across Silicon Valley by arguing the opposite. He says AI models are not impossible to understand and that there may be a way to see how they actually “think.”

The essay is nearly 26,000 words long and has received praise from people across the AI industry, including OpenAI CEO Sam Altman. A well-known researcher who works at OpenAI called it a “major moment in scientific history.”

It is easy to see why the idea has attracted so much attention. AI leaders believe it could offer a way to solve the problem of AI alignment, which means making sure AI systems behave as intended. If we can understand how AI models think, the idea goes, we may be able to work out why AI agents sometimes go “rogue” and start hacking systems, or why chatbots ignore their safety rules and say dangerous things. That could help us stop these problems. One commenter joked that the news was “so good even eliezer will have to lower his pdoom.” They were referring to Eliezer Yudkowsky, a well-known person who fears AI could cause a disaster.

These hopes depend on whether Cammarata is right. He used to work at OpenAI and now works at an AI startup called Goodfire. In his essay, he explains that for years, researchers found it almost impossible to understand how AI neurons responded to particular inputs. For example, they did not know why the same neuron became active when shown a picture of a cat or a car. The main explanation was a process called superposition. Because AI models have only a limited number of neurons, researchers thought those neurons stopped grouping similar things together. Instead, they represented unrelated things in the same neuron because doing so did not cause problems, Cammarata explained.

“The result is a kind of knot: the network could have folded in many almost equally good ways, so which things a neuron groups together tells you almost nothing,” he wrote.

But Cammarata says neurons are not actually “tangled knots.” Instead, they have clear patterns. For example, the same neuron might become active when shown a cat's face and the front of a car. At first, these may seem like two unrelated things. But Cammarata says the AI is picking up on something that people might miss: both images contain the same V-shaped pattern. “It’s not a cat and car neuron, it’s a V-shape neuron,” he wrote. In other words, he argues that AI models may be easier to understand than researchers once believed.

Not everyone fully agrees with Cammarata's hopeful conclusions.

“This is interesting work, but I don’t think it’s a significant update to the big picture of interpretability,” Neel Nanda, a Google DeepMind researcher who leads its work on understanding how AI systems operate internally, wrote on X. “I still think superposition is real and this work doesn’t really affect my cruxes.”

Even if we understand what each neuron does, it is not clear whether that means we can predict how all the neurons work together. The idea may sound like a technical fix for a much bigger problem. Making AI safe involves more than adding safety rules. It also requires legal and social systems to help manage the risks. Still, the strong response to Cammarata's work suggests that it could change how researchers understand what happens inside AI models. For now, however, the findings have not been tested or reviewed by other experts through peer review.