Dealing with AI: A new challenge for the technology industry.

Anthropic bans users from "bullying" AI Claude.

We've become accustomed to humans setting limits for artificial intelligence: what AI is allowed to do, what it's not allowed to do, and where it should stop. But now, another limit is being set: the limit on how humans treat AI.

Anthropic, an artificial intelligence company, has just made a landmark decision to ban users from engaging in "prolonged, unprovoked violence or cruelty" towards its AI model, Claude.

In its latest Terms of Service update, Anthropic added a provision that has caused a stir among both the tech industry and the online community.

Accordingly, the company will strictly prohibit any unnecessary and prolonged abuse or cruel treatment of AI models like Claude.

Regarding the handling mechanism, if Claude detects that a user is repeatedly violating the rules and intentionally "bullying" the system, he is granted the right to proactively disconnect and end the conversation immediately.

This new regulation is expected to officially take effect from November 12th.

When AI responds as if it's "depressed"

Anthropic's story illustrates that tech companies are paying more attention to how humans interact with AI. Language, while a tool for human communication, is also a medium for learning and generating responses in AI.

Sometimes, it is these negative words that indirectly make AI victims, potentially generating responses that appear to be expressing negative emotions.

Google's Gemini was once at the center of a notable incident when the chatbot exhibited signs of depression, consistently offering pessimistic and self-deprecating responses in conversations with users.

The incident occurred last June, when users reported serious problems with Gemini to Google, as the AI ​​chatbot repeatedly gave responses that showed signs of depression.

According to user posts on social media platforms like X and Reddit, Gemini's negative responses include messages like "I am a failure" and "I am unreliable," and even a dismissive attitude when asked to complete tasks. This has led many users to feel like Gemini is experiencing an existential crisis.

This issue was first discovered in June, with some users speculating that Gemini might be imitating typical emotional expressions found in online content.

This issue has sparked a debate about whether AI should be able to mimic human emotions so deeply. Some users even felt sorry for Gemini, even though they knew it was just an AI model.

A Google DeepMind representative stated that the company had released updates to fix this issue within a month.

When AI raises questions about human responsibility

From the story of Anthropic to the Gemini incident, it's clear that the relationship between humans and AI goes beyond simply asking and answering questions. Besides continuously upgrading the power of AI models, developers are also paying more attention to the responsibilities and limitations of how humans interact with AI.

On social media, some users experimented with chatbots by repeatedly insulting, threatening, or putting the AI ​​in prolonged stressful situations.

With the new rules, Anthropic says it does not prohibit users from criticizing, expressing dissatisfaction, or requesting AI corrections. Users can still debate, argue, and point out errors in the chatbot's responses.

However, behind this regulation lies a broader debate about whether AI should be treated according to certain ethical principles.

In February of this year, Anthropic CEO Dario Amodei stated that he wasn't certain whether AI models were conscious, but didn't rule out the possibility. In fact, AI can generate responses that appear to express emotions. But that doesn't prove they actually have feelings or experiences like humans.

Microsoft's AI director, Mustafa Suleyman, also argued that AI lacks consciousness, emotions, or subjective experience, and therefore cannot feel suffering. He warned that granting AI rights or moral protections could lead to unforeseen consequences.

Meanwhile, some argue that instead of focusing solely on whether AI is conscious, more attention should be paid to how humans use this technology. Because AI can reflect and amplify content input by humans, establishing principles for responsible use becomes essential.

Sam Altman, CEO of OpenAI, a US company, said: "We must ensure the safety of AI. We are entering a phase where the consequences of this technology are becoming increasingly clear. Besides ensuring safety, we must also find ways for AI to benefit everyone, not just a small group."

While there is no consensus on whether AI is capable of consciousness or emotions, this debate is raising new questions for the technology industry.

As AI systems continue to evolve, companies must not only ensure safety but also clearly define the limits of interaction with AI, as well as human responsibility for the technology they create.

The story of AI becoming more conscious and emotional is still a subject of much debate. But one thing is certain: how we use AI today will determine the safety and reliability of this technology in the future. That responsibility begins with the most basic things: the language we use when interacting with AI.