Who controls AI conversations?

Every interaction with generative AI includes an invisible layer: system prompts. They are set by other people than the user and shape how AI systems respond, what they prioritize, and ultimately how users experience these technologies. Users are often not aware of their influence.

This is exactly where the award-winning research of Anna Neumann, Yulu Pi, and Jat Singh begins. The paper Who Controls the Conversation? User Perspectives on Generative AI (LLM) System Prompts has been recognized with a Best Paper Award at the ACM CHI Conference on Human Factors in Computing Systems 2026, which is one of the most prestigious venues in human-computer interaction.

Anna Neumann is a PhD researcher in the Compliant and Accountable Systems group at RC Trust, supervised by Prof. Dr. Jat Singh. The work was conducted in collaboration with post-doctoral researcher Yulu Pi and Jat Singh, reflecting the group’s interdisciplinary approach at the intersection of technology, society, and governance.

The paper addresses a fundamental question for the future of AI: Which language is used to control AI, and what do regular users think about this?

To answer this, the research combines qualitative, quantitative, and computational methods. By analyzing 1,309 real-world system prompts and complementing this with user studies, the team provides one of the first investigations into what system prompts look like and how they are perceived. 

The work makes three key contributions:

First, it introduces a taxonomy of seven types of system prompts, offering a structured way to understand how these systems are configured in practice.

Second, it provides empirical insights into user perspectives. The findings show a clear demand for greater end-user consideration: 89% of participants want some form of transparency about system prompts, and 79% want some form of control. At the same time, values such as privacy and freedom from bias emerge as central user values for system prompts.

Third, the paper discusses these findings in relation to design and governance considerations. It outlines how meaningful transparency mechanisms, participatory design processes, and governance structures, such as standards, provenance, and evaluation, could enable more accountable AI systems and greater user agency.

While system prompts already shape the everyday interactions of millions of users, they remain largely invisible and difficult to contest. Making them more transparent and accountable is therefore not just a technical challenge, but a societal one. Taken together, the research opens up a new research direction regarding system prompt design.

With this award, the researchers highlight the importance of interdisciplinary research that connects technical evaluations with analysis of responsibility, governance, and human agency in AI.

Anna Neumann will present the paper at CHI 2026 on Wednesday, 15 April, from 9:12 to 9:24 AM in the session ‘Conversational AI, Agency and Control’.