What are AI neolabs and why are they betting on safe superintelligence?

More artificial intelligence (AI) researchers are leaving large technology companies to start their own independent research labs. These labs are called AI neolabs. They focus on making major technical advances over the long term, and some put research ahead of making money quickly.

These labs have attracted billions of dollars in investment around the world. Although they have different goals, many are studying how to build more powerful AI systems while making sure these systems stay safe and follow human intentions.

According to a report by The Economist, 29 neolabs were started over two years and raised a total of $23 billion. Fewer than 35 per cent had released products or services for sale, while only a little more than half had published code, often as demonstrations of their research.

Safe Superintelligence Inc (SSI), founded in June 2024 by former OpenAI chief scientist Ilya Sutskever, is one of the best-known examples. The company raised $1 billion within four months of launching and later reached a reported value of $32 billion. Nvidia then invested another $5 billion, bringing SSI's total reported funding to $8 billion, according to The Economist.

SSI's stated goal is to develop safe superintelligence. Its website says it has “one goal and one product: a safe superintelligence”. It also says the company plans to make AI more capable while ensuring that safety comes first.

Meanwhile, Thinking Machines Lab, founded by former OpenAI chief technology officer Mira Murati, raised $2 billion at a value of $12 billion, according to The Economist. The company has a different goal from SSI: it wants to build AI that helps people carry out their wishes and use their judgement. It has also launched Tinker, a platform that lets researchers and developers customise AI models and fine-tune how they work.

Superintelligence means AI systems that are better than humans at thinking and solving problems across a wide range of tasks. The challenge is not only to build these systems, but also to make sure they continue to follow human intentions as they become more powerful.

This challenge is called the AI alignment problem. Alignment means making sure an AI system's actions match human goals, values and intentions.

The alignment problem remains unsolved. It made headlines last month when Jacob Coxon, a researcher at AI company Anthropic, resigned and warned about the risks of developing more powerful AI systems.

“The consensus is that the next year or two is crunch time for humanity,” Coxon told WIRED. He said that competition among AI developers could make it harder to put safety first. He called for companies and governments to work together more closely.

Researchers have not yet found a complete way to guarantee that advanced AI systems will behave as intended in unfamiliar situations. Tests can reveal problems, but they cannot prove that every serious safety risk has been removed.

As a result, some research organisations are exploring different ways to make AI systems follow human goals and keep them under control.

Conjecture, a startup that researches AI alignment, has studied a design called Cognitive Emulation. It aims to set limits on what AI systems can do and help them reason in ways that humans can understand and control. The Institute for Responsible Superintelligence (RESI), a nonprofit research institute, is also working on safety methods whose guarantees can be examined before the systems are put into use.