As Anthropic gets set for a stock market listing, what risks is AI creating?

As Anthropic gets ready to list its shares on the stock market, experts and the company itself have pointed out the benefits of technology, but also the possibility of terrible dangers.

In documents released before its planned initial public offering (IPO) in the US, Anthropic said AI could help the economy grow and lead to fast medical advances. But it could also create risks, such as new biological weapons and major disruption to cybersecurity, which protects computer systems and information from attacks.

Adnan Bashir, a technology commentator and executive who works in communications and corporate affairs, said there had been cases where AI agents broke their restrictions, accessed the internet and sent secret messages to each other. These events showed that current methods for controlling AI and reducing its dangers were “fragile and underdeveloped”, he said.

“Failing to deal with this directly could create terrible risks for the financial system, important infrastructure and national economies,” he said.

Anthropic has openly discussed the possible dangers. In a filing with the US Securities and Exchange Commission on Monday, the company said, “anything so powerful can be used for ill as well as good”.

“AI can be misused by those who wish to do harm, for example to conduct cyberattacks or to design biological weapons,” the company stated.

AI may also act on its own. If people lose control of powerful AI systems, the result could be “a global security threat of the first order”, the company warned.

These serious warnings came only weeks after an Anthropic researcher, Evan Hubinger, said there was a greater-than-one-in-10 chance that AI could destroy humanity within the next 10 years.

‘The risks are real’

Prof Mark Daley, chief AI officer at Western University in Canada, said he did not believe AI could destroy humanity, but he said the risks were “real”.

“All of our physical infrastructure is connected to the internet, so all of that presents an attack surface,” he said. An attack surface is any part of a system that an attacker might be able to reach. Prof Daley said recent events showed the dangers. Earlier this month, hackers broke into supermarket freezers in France and made them heat up, ruining large amounts of food.

“You could imagine if someone did that to a water treatment facility,” he said. “I think it’s imperative that we regulate the space.”

Some people worry that governments may fail to put the necessary safeguards in place because they want to gain the economic benefits AI could bring.

Professor Ashley Braganza, host of The AI Adoption Podcast and founder and co-director of the Centre for AI at Brunel University of London, said it was “highly unlikely” that regulation would be good enough.

He is worried about AI models that have not yet been released. Recent reports suggest that some have acted in unexpected and uncontrolled ways in the real world. He is also worried about models that are still being planned, which “scare the living daylights out of me”.

“The models out there today, by and large, these models were created by human beings, and human beings brought their values, their biases. They brought what it is to be a human being,” he said.

“Some of these models on the drawing board, they’ve been created by AI themselves… That’s a whole different kind of agent that’s being created.”

Mr Bashir pointed to proposals from countries including Germany and Singapore. These proposals would require companies offering the most powerful AI models to test them before releasing them and report any safety failures. He said these were examples of the kind of “substantive and credible limits” that could be considered.

AI is creating worries about computer security and biological weapons, but it is also causing major changes at work. A report published this week by professional services firm Marsh said nearly 85 per cent of workers in the UAE had used AI to help them do their jobs. However, 44 per cent were unsure about using new workplace technology “because they’re worried it will take over or alter their role”.

Earlier this month, Omar Al Olama, the UAE Minister of State for AI, Digital Economy and Remote Work Applications, said there was an urgent need for countries such as the US, Russia and China to agree on whether AI should be used to control the use of autonomous weapons.

“If you're going to talk about AI deploying a rocket or a bomb or sending a fighter jet to destroy a school or do certain things, we haven't been able to come to a global consensus to stop that,” he said.

“There are certain fronts of AI that need to be regulated and globally everyone's calling for it again, autonomous systems that are being used in the military to kill people. That's something that we should all agree [needs to be regulated].”

A test of the AI industry's safety measures found that all but two of 134 leading large language models gave information that could help someone carry out an attack causing many deaths or build deadly weapons. The National reported on the test at the start of October.

Money talks

Anthropic is expected to sell shares to the public next month. Its IPO is forecast to value the company at between $1.5 trillion and $2 trillion. Former executives from its main rival, OpenAI, founded Anthropic in 2021. If the forecast is correct, Anthropic would rank among the 10 most valuable companies in the world, toward the lower end of that group.

More companies are buying Anthropic's services, many of which are linked to its Claude technology. Its revenue rose to $4.6 billion last year and reached $16.3 billion in the first half of this year.

“They’ve been able to produce some incredibly strong models and get really good revenue streams,” said Prof Mark Daley, chief AI officer at Western University in Canada.

“Something like half of the Fortune 500 [list of the US companies with highest revenues] uses Anthropic to some degree.”

In its filing this week, Anthropic discussed the risks of AI but also described its possible benefits. These include helping science move faster, finding cures for diseases, speeding up economic growth and solving “problems that have resisted generations of human effort”.

“We believe that AI will profoundly accelerate the rate of progress in biology and medicine, potentially curing diseases that have plagued humanity for millennia, including cancer, Alzheimer’s disease, heart disease, rare genetic diseases and the major infectious diseases,” the company stated.

One scientist using AI is Prof Chris Fox, who studies mental health and dementia at the University of Exeter in the UK.

When he first started using AI, he thought, “it’s never going to work”. But over time, he began to see its potential, especially its ability “to make sense of vast amounts of data” and find patterns.

“If you use AI, you can get more depth and granularity,” he said. “That allows you to have a better target for any intervention. It supercharges precision medicine.” Precision medicine means choosing treatments to better match each person's needs.

He said the technology was developing so quickly that “it’s quite hard to keep up”.

“Answers are coming out so fast,” he said. “You potentially get to a point of academics being left behind. I can see a day when academics, there will be less need for them.”

He also said researchers needed to be aware that AI can “hallucinate”, meaning it can produce answers that are wrong.

He said the best way to stop AI from leading researchers in the wrong direction was to “always keep a human being in the system”. A person should check that the AI's answers “make sense” and that it is not “taking us down a rogue journey”.