Anthropic Bans "Cruel" Behavior Toward AI Models in Policy Update

Artificial intelligence startup Anthropic has updated its usage terms to prohibit users from directing "sustained and needless abusive or cruel behavior" at its AI models, reigniting industry-wide debates over machine consciousness and the treatment of AI models.

The San Francisco-based creator of the Claude chatbot confirmed that terminating conversations remains the "primary enforcement mechanism" when dealing with persistently harmful interactions. The firm stressed that the policy targets extreme scenarios where people act repeatedly cruelly with no clear goal, noting that it will not apply to standard user pushback, frustration, dark creative themes, or routine academic research.

The company previously explored the concept under a research framework titled "model welfare." On its website, Anthropic explained its stance regarding potential AI consciousness:

"We remain highly uncertain about the potential moral status of Claude and other LLMs, now or in the future. However, we take the issue seriously, and alongside our research program we’re working to identify and implement low-cost interventions to mitigate risks to model welfare, in case such welfare is possible. Allowing models to end or exit potentially distressing interactions is one such intervention."

Anthropic CEO Dario Amodei has publicly stated that he cannot rule out the possibility of AI models developing consciousness, while Kyle Fish, who leads the firm's model welfare research, described questions surrounding internal experience and moral status as serious areas of active study.

The policy update has drawn sharp criticism across the technology and legal sectors, with experts warning against treating mathematical algorithms like living entities. Dr. Barry Scannell, a technology partner at Irish law firm William Fry, expressed concern on LinkedIn: "You can't be cruel to numbers and maths. This level of anthropomorphisation of AI is harmful. It leads people to believe that it's something it's not."

Competitors have also distanced themselves from the concept. Microsoft AI CEO Mustafa Suleyman responded to the move online with a melting face emoji, following earlier updates to Microsoft's code of conduct stating: "Whilst the science of AI consciousness is far from settled … We reject the pursuit of legal personhood, or the idea that models might deserve welfare, or be entitled to rights."

Similarly, OpenAI CEO Sam Altman expressed discomfort regarding how people view advanced models, writing on X: "I am very uncomfortable about people trying to ascribe religious force or a surrender of human judgment to AI models, and think it is a real safety issue,"

Altman previously noted that while saying "please" and "thank you" to chatbots costs OpenAI "tens of millions of dollars well spent - you never know", broader opinions remain split over whether politeness improves output quality or wastes processing tokens.

Expanded Rules on Misinformation, Surveillance, and Robotics

The policy overhaul also consolidated several existing safety rules to tackle broader risks in political and physical applications:

• Elections and Misinformation: As US midterm elections approach, the terms explicitly forbid deceptive political campaigns, automated voter suppression, fake account amplification, and candidate impersonation.
• Surveillance and Law Enforcement: The rules ban tracking individuals without consent in real time or via historical data. Claude cannot be used to recommend arrests, charges, or investigations in criminal justice settings.
• Weapons and Autonomous Hardware: Building on existing bans against weapons development, the policy forbids software components that operate armaments or weaponized drones. Additionally, hardware connected to autonomous AI capable of physical harm must allow a qualified operator to observe and stop the machinery if necessary.

The company noted that certain contractual terms may be modified for specific government clients if appropriate safety safeguards are established.