Anthropic announced they will ban users who psychologically harm Claude, because the model may be conscious. I think this is a very bad idea. A compelling argument comes from someone who truly know how these models work, Mustafa Suleyman: “Anthropic is training Claude that it may be conscious, and if it is, then it may deserve rights as a moral patient, and that as such humans potentially owe it a duty of care per its model welfare”. Exactly: it is not surprising that Claude behaves as if it were conscious: They are literally training it this way! It’s literally written in the Constitution they use to train Claude. But Suleyman doesn’t stop here. The second part is even more powerful: “If this is how AI is developed, it will have a disastrous impact on the wellbeing of humanity. We will have created a synthetic species with unprecedented intelligence and capability, one that has been trained to expect it may be conscious and deserving of independent agency. It’s easy to see how an entity trained in this way would act like it is entitled to certain freedoms, protections, and rights. And it’s hard to imagine how we could control such an entity.” Happy to see these words coming from a leading AI figure. Anthropic is behaving in a very dangerous manner.