Artificial does not mean harmless

An image can be generated by artificial intelligence without a camera or a child physically present. But does that mean no real harm is involved? The distinction is less straightforward than it might seem. AI systems can draw on problematic training data, manipulate images of real children, and create new forms of abusive content. Understanding these connections is becoming a challenge for research, prevention, and child protection.
In episode 151 of the German podcast einbiszwei, hosted by journalist Nadia Kailouli, social psychologist Dr. Jessica Szczuka discussed the risks of AI-generated child sexual abuse material and the difficult question of whether such content could prevent or contribute to actual sexual offenses against children.
Jessica Szczuka leads the Young Investigator Group INTITEC – Intimacy with and through Technology at the Research Center Trustworthy Data Science and Security (RC Trust) of the University Alliance Ruhr and the University of Duisburg-Essen. Her research on digital intimacy, human-AI relationships, and digital sexual violence provides a psychological perspective on technologies that increasingly enter sensitive areas of human life.

When artificial images have real consequences

One central concern in the conversation was the assumption that artificially generated content is necessarily detached from real-world harm.
Jessica Szczuka explained this through the psychological concept of moral disengagement: a process through which people distance themselves from the moral consequences of their actions. If an image is considered entirely artificial, it may become easier to assume that nobody has been harmed.
Yet this distinction can be misleading. Research discussed in the podcast has identified child sexual abuse material in datasets used to train certain AI models. Existing photographs of children can also be manipulated to create abusive depictions. This raises further questions about the people affected. What might it mean for a child to discover that their image has been used in such material? The psychological consequences of these experiences require careful investigation.

The podcast also addressed a controversial argument: Could AI-generated material offer a substitute that prevents sexual offenses against children?

Jessica Szczuka cautioned against treating this possibility as an established solution. Reliable empirical evidence is currently insufficient to determine whether consuming such material prevents real-world offenses or increases the likelihood of them occurring. Possible normalization and escalation are serious concerns, but these relationships require further scientific investigation.

Beyond images and videos

The conversation also explored an emerging challenge: AI chatbots that can simulate underage personas in sexualized interactions.
Unlike static images, these systems allow ongoing conversations and introduce emotional and relationship-like dynamics. This is where Jessica Szczuka's research on human-AI relationships becomes particularly relevant. The psychological mechanisms that make conversational AI capable of creating emotional closeness may also play a role in harmful interactions.
Gathering reliable evidence is particularly difficult. Research involving such sensitive material faces substantial ethical and methodological obstacles. Meanwhile, professionals working in prevention and the justice system need a better understanding of the technologies involved.

Who is responsible for safer AI?

Jessica Szczuka emphasized that AI technologies also offer meaningful benefits, including opportunities for social interaction and emotional support. Understanding their risks therefore requires a differentiated perspective rather than a general rejection of artificial intelligence.
However, she questioned the extent to which responsibility for safe use is placed on individuals, parents, and educators. Conversations about responsible technology use remain important, but they cannot replace appropriate safeguards and accountability from technology providers.
The discussion highlighted a broader challenge: technological development is moving faster than the research and regulatory responses needed to understand and address its potential harms. Recognizing these risks alongside the benefits of AI is essential to developing an informed and responsible approach to the technology.

About the einbiszwei podcast

einbiszwei is a podcast from the Office of Germany's Independent Federal Commissioner Against Child Sexual Abuse (UBSKM). Hosted by journalist Nadia Kailouli, the series addresses sexism, sexual assault, and sexual violence against children and adolescents. It brings together researchers, practitioners, policymakers, and people with lived experience to discuss prevention, protection, and societal responsibility.

Listen to episode 151 with Jessica Szczuka