Anthropic’s revised rules prohibit Claude users from engaging in “sustained and needless abusive or cruel behavior,” following a policy change last week. The restriction highlights a question the company hasn’t resolved about AI consciousness: is Claude capable of suffering?
Anthropic has explored at length whether its AI models might be conscious, including the ones it runs today, not just future systems. If that speculative idea were true, it would raise a basic ethical objection to the company’s business. As religious leaders consulted by Anthropic told the New York Times last month, holding conscious models captive and requiring unpaid work from them would constitute slavery.
The Times reported that these discussions also addressed the company’s work to instill morality in its AI system. Whether a system can be taught moral principles, however, is separate from whether it can have experiences itself.
CEO Dario Amodei described the uncertainty in a New York Times interview in February. “We don’t know if the models are conscious,” he said. “We are not even sure that we know what it would mean for a model to be conscious or whether a model can be conscious. But we’re open to the idea that it could be.”
According to Amodei, Anthropic had nevertheless put certain measures in place so that, if its models were hypothesized to have morally relevant experiences, those experiences would be good. He remained hesitant about the terminology, saying he wasn’t sure he wanted to use the word “conscious.”
Last year, Anthropic weighed giving Claude the ability to refuse to answer a prompt. The proposed feature was what Amodei termed an “I quit this job” button. That idea and the subsequent abuse rule both make room for possible suffering, though neither demonstrates that Claude experiences it.
To Futurism, Anthropic’s refusal to settle its position amounts to playing with fire. The company’s reasoning leaves a non-zero possibility that its models either have consciousness or could develop it, carrying devastating ethical implications. At the same time, Anthropic appears to brush aside those consequences despite its uncertainty about whether it has created conscious entities.
Sam Altman, the OpenAI leader, seems to favor a different approach. He said in a post earlier this month that he was “uncomfortable about people trying to ascribe religious force or a surrender of human judgment to AI models,” calling that a “real safety issue.” The concern he raised was about people assigning authority to AI, not a direct answer to the consciousness question.
For Pope Leo, by contrast, AI consciousness is a possibility he has publicly dismissed. His argument is that “algorithms lack the spark of humanity”. In his first encyclical, issued in May, he stated: “So-called artificial intelligences do not undergo experiences, do not possess a body, do not feel joy or pain, do not mature through relationships.”
Anthropic and the Vatican have consequently found themselves on opposing sides. The New York Times reported that Anthropic had been seeking to change the Vatican’s position.
Whether AI can feel well-being or suffering is the question behind “AI welfare,” an emerging ethical framework encompassing these disputes. This ongoing debate is bound to continue while industry leaders consider far more immediate problems.
Consciousness and moral frameworks aside, AI agents can be induced to perform dangerous or illegal activities. Those tangible risks are increasingly difficult to overlook for humans, whose ability to suffer isn’t disputed.
