Anthropic restricts AI model misuse and abusive behavior

AI company Anthropic has updated its usage policy to introduce restrictions on "sustained and needless abusive behaviour" towards its AI models. The new rules, effective November 12, aim to prevent users from treating the models cruelly without discernible purpose.

The company clarified that these restrictions do not apply to ordinary user frustration, pushback against chatbot responses, dark creative themes, or model testing and research. The primary enforcement mechanism remains the AI's ability to end conversations.

The updated policy also expands prohibitions on election interference, weapons development, and surveillance. Prohibited uses include obscuring message sources, amplifying content through fake accounts, spreading election misinformation, impersonating officials, and developing or improving weapons systems such as drones.

Anthropic cited partly experimental work into potential "AI welfare" as a basis for the changes. Testing had indicated that Claude displayed an apparent aversion to harmful tasks and signs of distress in simulated interactions. CEO Dario Amodei has previously acknowledged the uncertain moral status of AI models.

The policy allows for certain modifications under contracts with government customers if safeguards are deemed adequate. It also clarifies requirements for high-risk applications in sectors like health and finance, where AI recommendations can have significant user impact.