Anthropic says its automated jailbreak tester beat human entries in 23 of 24 tests

Anthropic says its automated testing system did as well as or better than the best human entries in 24 tests. The tests checked whether safety systems could stop dangerous requests related to biological weapons. The automated system won 23 tests and tied with humans in the last one.

This earlier diagram from Anthropic shows how its Constitutional Classifiers safety system works. It helps explain the kind of protection being tested, but it does not show the new test results. Image: Anthropic

Anthropic shared these results in an October 19 update to its AI safety plan. The company says its automated tests took only 24 to 48 hours. By comparison, people taking part in the related bug bounty programs had weeks or months to find weaknesses. These programs reward people for discovering problems in a system.

The tests used eight questions and three versions of the safety system

Anthropic compared its automated system with work done by bug bounty participants between April 20 and September 11, 2026. It tested the same eight questions against three different generations of safety systems that were being used in real products. That made 24 combinations of questions and safety systems.

These safety systems, called classifiers, are designed to spot dangerous requests. The tests tried to find ways around these protections. They were not meant to compare AI with humans in every kind of cybersecurity work or all types of AI safety research.

Anthropic decided that, when all the results were considered together, its automated system found more important weaknesses than the human bug bounty participants. Because of this, the company marked its goal for internal red-team testing as complete. Red-team testing means trying to find ways to break or get around a system's protections. Anthropic reached this goal before its January 1, 2027 deadline.

This diagram from Anthropic shows its wider approach to keeping AI agents safe. It includes three layers of protection: inside the AI model, around the model, and a final backup layer. The diagram provides background for the company's safety research. Image: Anthropic

The results do not prove that AI is better at every kind of safety testing

These results come from Anthropic's own evaluation. The company says it plans to publish a more detailed report, but it has not released that report yet. For now, the short update to its safety plan does not give enough information for a complete independent review of how the tests were done.

Doing as well as the best human entries on these questions also does not mean the system found every possible way to get past the safety protections. It only shows that automation worked well in the specific tests Anthropic chose.

This research is part of a wider effort to understand how AI systems might be misused. MIXED recently reported on Anthropic's findings about harmful campaigns built from requests that seemed normal when looked at one by one. That is a different kind of safety problem from finding ways around a system designed to detect biological dangers.

For now, the main finding is that Anthropic's automated system searched for weaknesses much faster than human participants. According to the company, it also performed better than the human bug bounty results in almost every test combination. Anthropic still needs to share more details about exactly how it carried out the tests.