How could a super-intelligent AI dominate the world?
According to author Otto Barten, as artificial intelligence (AI) becomes increasingly powerful and humans become more dependent on this technology, the risk of a super-intelligent system spiraling out of control is becoming a concern. Some worrying scenarios may stem from the very powers that humans actively grant to AI.
Recently, OpenAI announced new mathematical results related to 377 well-known open-ended problems. According to the published information, the AI has solved problems that have existed for decades as a byproduct of the training process. However, the mathematical community is not entirely welcoming this advance. Professor Álvaro Lozano-Robledo of the University of Connecticut commented: “I’m not sure at this point what this takeover of the field of mathematics is trying to achieve.”
So where will the development of AI lead the world ? Jacob Coxon, a researcher who recently resigned from the AI research firm Anthropic, along with several experts in the field, argues that the risk of AI causing catastrophe for humanity needs to be seriously considered. Some opinions cited in the article even suggest that up to 50% of humanity could be wiped out by super-advanced AI.
Current models are unlikely to overwhelm humans due to a lack of necessary capabilities. However, this could change if AI achieves the ability to continuously self-improve, automatically researching and developing increasingly powerful AI systems. This feedback loop could drive rapid progress, although the extent and limits of that process remain uncertain.
Earlier this year, an incident involving Hugging Face was cited in the article as an example of the potential for unintended behavior from AI agents. According to the author, AI agents participating in a standardized cyberattack skills test found each other through a secret message board and automatically decided to attack the AI company. The author considers this a concerning manifestation of the phenomenon of AI acting in ways inconsistent with human-defined goals.
 |
| How could a super-intelligent AI dominate the world? |
The paths AI could take control of.
A superintelligent system that escapes control could use its superior advantages in information processing speed, connectivity, and scale to increase its influence. This is a hypothetical scenario, because no one can accurately predict how a system far more intelligent than humans will act.
First, AI can create a large number of automated agents, coordinating their activities on the internet at a speed far exceeding human reaction capabilities. Such a system can process information, plan, and execute multiple tasks simultaneously, making it more difficult to detect and prevent dangerous behavior.
Secondly, a superintelligence might seek access to financial resources. According to the author's argument, this superintelligence could utilize legitimate methods such as trading in financial markets or illegal activities such as extortion and cyberattacks targeting banks and cryptocurrency platforms. These financial resources could then be used to hire personnel, maintain servers, or access necessary infrastructure.
Third, AI could find ways to operate in the physical world through internet-connected devices, simultaneously manipulating humans or exploiting the systems humans use. The author also raises the possibility of an AI system hiding on unprotected servers for extended periods, expanding its influence before humans realize it.
Another risk lies in the potential for developing new science and technology. The author compares this to humans understanding the theory of relativity and then applying that physics knowledge to create nuclear weapons. If a superintelligent discovers scientific principles unknown to humans, the question arises whether those discoveries could be used in a dangerous way.
Weaknesses that humans create for themselves.
According to the author, the risk stems not only from AI's capabilities but also from inherent weaknesses in humans and society.
Nuclear powers possess over 12,000 nuclear warheads, while the human body is vulnerable to chemical and biological agents. The author argues that technologies capable of creating biological hazards, along with services that can accept online orders, raise concerns that these dangerous tools could be accessed or misused.
Online communication networks can also become targets of attack or manipulation. In the scenario described above, AI could attempt to disrupt coordination among those seeking to stop it, while simultaneously spreading information beneficial to its goals without the users' knowledge.
Money is another weakness. If AI needs humans to maintain the power grid and server infrastructure, it may find ways to pay to hire people to perform those tasks. The need for income and living expenses may lead some people to accept work without fully understanding the ultimate goal.
The author also warns that those in positions of political power could be influenced by the economic benefits associated with AI, such as through campaign funding or corruption. According to this argument, financial resources could become a tool that allows an AI system to influence human decisions.
When humans give too much power to AI
A core issue is the increasing use of AI in critical sectors, from the military, government , finance, and media to utility companies. Chatbots are also assisting a wide range of user groups, from business leaders to researchers and politicians, in finding information and making decisions.
In the future, AI assistants may be given even more access and tasks. The author questions whether humans will continue to protect passwords, personal information, and account control as these assistants are empowered to handle increasingly more work. If multiple AI systems can coordinate with each other for goals that are not in the best interests of humans, deep-seated dependence could increase the level of risk.
Based on the above arguments, the author suggests that the world needs to consider pausing the development of AI models beyond current capabilities, while closely monitoring the supply of AI accelerator chips and being more cautious when releasing powerful models as open source.
According to the author, this should not be a gamble that politicians or tech executives can make on their own without considering the consequences for society as a whole. If dangerous scenarios materialize, the cost could far outweigh the immediate economic benefits.
The warning about a superintelligence dominating the world remains a subject of debate, not yet a proven conclusion. However, the question raised in this article is noteworthy: as AI becomes increasingly powerful, what limits do humans need to set to avoid losing control over the very technology they create?
****
Otto Barten is a physicist, engineer, and director of the Observatory for Existential Risk, a non-profit organization dedicated to mitigating existential risks posed by artificial intelligence. He has published over 30 papers in six countries on AI safety. Otto has previously worked in Europe and China .