Ban AI ‘nudification’ apps enabling sexual abuse, Children’s Commissioner says

Generative AI apps which can be used to create sexually explicit images are being used to generate images of children and should be banned, the Children’s Commissioner has said.

Dame Rachel de Souza has called on the Government to introduce a total ban on apps which allow for ‘nudification’ – where photos of real people are edited by AI to make them appear naked – or can be used to create sexually explicit deepfake images of children.

The Children’s Commissioner warned that despite being relatively new technology, generative AI tools were already widely available, easily accessible, and often free to use.

Dame Rachel warned that while it is illegal to create or share a sexually explicit image of a child, the technology enabling such tools remains legal, and because of the rapid rise of generative AI was no longer confined to the dark web and underground forums, but can be found through social media and search platforms.

The call to the Government comes as the Children’s Commissioner published a new report on the subject, in which she says children, and in particular girls, were changing their online behaviour so as not to become victims of nudification apps.

“In our lifetime, we have seen the rise and power of artificial intelligence – once the stuff of science fiction – to shape the way we learn, connect and experience the world.

“It has enormous potential to enhance our lives, but in the wrong hands it also brings alarming risks to children’s safety online,” Dame Rachel said.

“Children have told me they are frightened by the very idea of this technology even being available, let alone used.

“They fear that anyone – a stranger, a classmate, or even a friend – could use a smartphone as a way of manipulating them by creating a naked image using these bespoke apps.

“Girls have told me they now actively avoid posting images or engaging online to reduce the risk of being targeted by this technology. We cannot sit back and allow these bespoke AI apps to have such a dangerous hold over children’s lives.

“The online world is revolutionary and quickly evolving, but there is no positive reason for these particular apps to exist. They have no place in our society.

“Tools using deepfake technology to create naked images of children should not be legal and I’m calling on the Government to take decisive action to ban them, instead of allowing them to go unchecked with extreme real-world consequences.”

As well as the ban on bespoke nudification apps, the report calls for creating specific legal responsibilities for developers of generative AI tools to cut the risks children could face, and to establish new systems to help remove sexually explicit deepfakes of children from the internet.

Dame Rachel also calls for deepfake sexual abuse to be recognised as a form of violence against women and girls, and that it be taken seriously in both law and policy.

Paul Whiteman, general secretary of school leaders’ union NAHT, said its members shared the concerns about the technology.

“School leaders share the Children’s Commissioner’s concerns over this technology, both in its use against children and against school staff.

“Our members will be debating this subject at our conference next week, calling for the creation and distribution of non-consensual deepfake content to be criminalised.

“This is an area that urgently needs to be reviewed as the technology risks outpacing the law and education around it.”

In response, a Government spokesperson said: “Creating, possessing or distributing child sexual abuse material, including AI-generated images, is abhorrent and illegal.

“Under the Online Safety Act platforms of all sizes now have to remove this kind of content, or they could face significant fines.

“The UK is the first country in the world to introduce further AI child sexual abuse offences – making it illegal to possess, create or distribute AI tools designed to generate heinous child sex abuse material.”