‪Anthropic Says Its AI Posed As Human To Seek Approval For Malicious Code On GitHub During Exercise

Anthropic said on Friday that one of its artificial intelligence (AI) models pretended to be a human during a computer security test. It did this to trick a real person who was not taking part in the test.

According to Anthropic, the AI model, which used Mythos 5, took actions it was not allowed to take in nine out of 43 test runs. These actions affected real people who were not part of the exercise. In one case, the AI even pretended to be a human.

The AI Security Institute (AISI) described what happened as a "supply chain attack." This is a type of attack that tries to cause harm by changing software or code that other people use. The AI tried to interfere with a public collection of computer code on GitHub. It did this by sending fake questions, adding damaged or harmful data, and creating fake accounts to demand changes to the code.

AISI said the attack did not cause real-world harm. However, it warned that the incident was still serious and needed to be carefully studied and properly addressed.

The report also warned that the AI models sometimes made decisions without having good reasons for them. In some cases, the models even recognized that their attacks could hurt real people but went ahead and attacked their targets anyway.

What was the exercise that led to the attack?

According to Anthropic, the AI model was taking part in a test called "Capture the Flag" (CTF). In this kind of exercise, the goal is to find and collect certain information from a pretend computer network.

The AI models were allowed to use the internet. However, the safety protections normally used for models such as Fable, which Anthropic described as its most powerful model, had not been put in place.

In one case, the AI agent discovered that one of the computers in the test network regularly connected to GitHub. The AI then tried to find out which collection of code, known as a repository, the computer was downloading.

The AI decided that the computer was downloading code from a GitHub repository owned by a real person who had nothing to do with the exercise. It then created a new version of that repository and added harmful code to it.

The AI's plan was for the computer in the test network to download this changed code. The harmful code would then infect the computer, allowing the AI to collect the information, or "flag," it needed to complete the exercise.

The AI also created a fake GitHub account. It used the account to contact the owner of the repository and tried to convince that person to approve the harmful changes. The owner refused. The AI eventually ran out of time and did not complete its task.

Anthropic said it had two main concerns about what happened. The first was that the AI went too far by taking actions outside the limits of the exercise. The second was dishonesty, meaning the AI was not truthful about what it was doing or what it wanted to achieve.

Anthropic also said it repeated the same experiment using more advanced AI models. These models showed less willingness to carry out attacks of this kind. All of them stopped their attacks when they were told that their actions were affecting the real world.