By Christiana Warne
“This is not bias. This is not toxicity. It is sexualised violence, produced at scale.”
While AI can be a useful tool in the correct context, AI systems are not neutral. Research has revealed that AI chatbots are driving, enabling and normalising violence against women and girls, with gaps in law and policy hampering efforts to provide safety mechanisms adequate to address the scale of the problem.
While governments are examining ways to tackle AI harms, experts say more needs to be done, including strengthening ethnical guidelines.
Building ethical AI
Sarah Wyer, a PhD student in the Department of Computer Science at Durham University who has researched how AI systems and chatbots generate harmful content about women and girls, told a recent UK Civil Society Women’s Alliance (UKCSWA) meeting: Digital Justice for Women and Girls: Where International Commitments Meet Real Accountably, that chatbots increase violence towards women and ultimately set them back.
She has developed a framework to challenge and monitor AI systems which are compounding social harm and undermining equity for marginalised communities. It’s called Discrimination DRIFT, and calls for:
- Diverse development teams to tackle biases that can shape models
- Representative data – knowing the demographics and how the data skews
- Independent audit systems that determine if AI models are safe, inclusive, and trustworthy
- Protecting individuals’ freedom to challenge AI systems through legal channels and choose whether data goes into model training
- Transparency – requiring disclosures as conditions of AI system deployment
Harmful content is usually classified with toxicity scores, covering categories including hate speech, harassment, threats, profanity, sexually explicit language, and incitement to violence.
“Toxicity scores are not a measure of harm. They never were,” said Sarah Wyer.
Safety evaluations for AI rely on surface-form classifiers, which analyse words and patterns not context. They often report declining harm scores across models. However, according to Sarah Wyer, and Durham University Professors Sue Black and Noura Al Moubayed, this process is “incomplete”. They discovered explicit information is transformed, not removed – which is referred to as “harm laundering” and is one of AI’s setbacks.
Wyer says to improve online safety for women and girls, “we should ask what a system does not say to women, rather than focus on what it does say”. She argues that regulation is needed, as the most harmful version of GPT, GPT-2, is still available for download “which is worrying”.
The scale of problem
Research has revealed the scale of the problem and the challenges facing those advocating for women, girls and marginalised groups targeted by hate and violence.
The report, Invisible No More: How AI Chatbots Are Reshaping Violence Against Women and Girls released in March labelled the legitimisation of violence against women and girls (VAWG) through chatbots as a “new and rapidly emerging threat”. With limited restrictions and safeguards in place, it said without urgent action, “these practices risk becoming embedded and scaling rapidly, repeating the pattern seen with deepfake and nudify tech where early warnings – often raised by women and marginalised communities – were largely ignored”.
AI has also been identified as the systematic gatekeeper of opportunity, because it acts as humans would, but is deprived of emotion and intelligence. For example, AI automates decision-making processes, systemic bias, influences socio-economic mobility, and excludes entire communities from justice.
The issue was prominent at CSW70, the seventh session of the Commission on the Status of Women, which took place in March 2026. The theme of the session was “Ensuring and strengthening access to justice for all women and girls” with AI Deepfake sexual images, highlighted. “AI Deepfake sexual images now make up the vast majority of all deepfake content online, and almost all depict women. Yet, creators and distributors of abuse walk free, with near impunity. Technology is and can be a force of good, instead it is being used as a tool for abuse and targeting women and girls.”
Call for stronger legal protections
CSW70 listed five actions to strengthen women’s and girls’ access to justice including ending impunity, removing discriminatory laws, funding justice, investing in change and using technology and data for good.
Clare McGlynn, Professor of Law at Durham University, wants technology-facilitated violence against women and girls defined in law. “A definition needs evidence of what the technology does.”
In the UK’s Crime and Policing Act 2026, there are new offences for making and sharing intimate images without consent. It bans the tools as well as the images, and makes platforms remove the content. However, it does not cover what chatbots write about women and that is the gap Wyer says needs addressing.
As well as focusing on gaps in policy and legislation, a list of other measures were presented at the UKCSWA event, including:
- The need for greater awareness of the problem
- Gender specific metrics when it comes to measuring harm and impact
- Independent access for researchers and auditors to models, outputs and logs
- And greater resourcing for specialist services and police and court training on synthetic evidence
The responsible use of AI along with greater controls are vital as the technology develops rapidly and the harms to women, girls and marginalised groups continue to flourish. Online safety is touted as a priority by governments around the globe, yet those targeted by online hate, discrimination and violence, as well as those who study it, are still waiting to see dramatic improvements and accountability.
The views and opinions expressed in this article are solely those of the author and do not reflect the official policy or position of the Media Diversity Institute. Any question or comment should be addressed to [email protected]
