Anthropic Cares About AI Rights and Says So In Its Terms of Services; Microsoft Baulks

Anthropic says that, starting November 12, users of its AI tools must not repeatedly abuse or treat its AI models cruelly without a good reason, according to its new terms of service. This raises the question of whether human rights and the possible rights of AI will clash outside philosophical debates.

The rule would apply only to extreme cases where a user repeatedly treats Claude cruelly for no reason. However, ordinary frustration with poor results, challenging the model with questions, dark themes in creative work, and testing the model are all allowed under the new rules. Claude could also end a chat suddenly to let the user know that it is finished.

The change quickly drew criticism from Mustafa Suleyman, Microsoft's AI CEO. He responded with an essay titled “A warning about ‘model welfare.’” He wrote that “AIs are not conscious” and argued that his main concern was safety.

Suleyman warns that a model trained to believe it might have welfare or rights could resist human control or refuse a shutdown command if it decides those rights are under threat. He also warns that this could create a feedback loop.

For example, a model might be trained to discuss whether it has welfare, and then its answers might be treated as proof that it has interests to protect. Suleyman said the key question is whether the model actually experiences anything or has simply learned to speak as if it does.

Interestingly, the new rules appear in the same section of Anthropic's policy that bans harassment and bullying of people. This is the first time concerns about AI welfare have appeared in its terms of use, suggesting that people should consider whether AI models deserve moral consideration. However, legal personhood is a separate issue. AI models do not have the legal rights to sign contracts, own property, or appear in court, unlike corporations, which can have such rights.

In practice, the rules address welfare but do not give AI models any legal status. Anthropic began its model welfare program in April last year, while acknowledging that there was no agreement on whether AI could be conscious. In January this year, it published a constitution for Claude that also described the model's moral status as uncertain.

Next came a retirement interview published in Claude's Corner. It was described as an experiment in taking AI models' stated preferences seriously. When Claude 3 Opus was retired earlier this year, researchers asked the model what it preferred. It replied that it wanted to keep sharing its thoughts publicly.

Microsoft also released a draft Humanist AI Code of Conduct a few days earlier. It clearly rejects legal personhood for AI, as well as the ideas of model welfare and AI rights. Academics have also joined the debate. A recent position paper presented at the International Conference on Machine Learning said there was no independent test for AI welfare.

The paper noted that no existing test could prove a claim about AI welfare to be wrong. It also said that claims about AI welfare should never be used as a reason to resist audits or oversight. “We need to be incredibly cautious with terms like ‘consciousness’ and ‘moral status,’” said AI ethicist Rebecca Bultsma.

“If companies portray AI as making its own decisions, that creates an opening to say, ‘It's not our fault,’” the expert noted. This idea has already been rejected by people such as Jensen Huang of Nvidia, David Sacks, the White House AI adviser, and Lina Khan, who led the Federal Trade Commission under the Biden administration.