On 27 April 2026, another seminar in the PAN-Metrics series took place. Dr Cameron S. Kay presented his research aimed at developing methods for detecting synthetic respondents, bots based on large language models, in online surveys. The tools presented make it possible to identify responses provided by such bots in a simple and effective way.
The presentation slides are available here.
Seminar summary:
Dr. Cameron S. Kay (Climate Cognition Lab, Stanford University): Are your online participants human?
Abstract:
Concerns about online data quality have traditionally focused on the possibility that some participants may not be devoting sufficient effort to their responses. However, with the advent and popularization of large language models (LLMs), researchers now face the unsettling possibility that some of their participants may not even be human. In this talk, I share two recent advances in detecting careless and synthetic respondents. First, I introduce CIFR (the Comprehensive Infrequency/Frequency Item Repository), an online database of 660 items designed to identify inattentive respondents. Second, I present ECLAIR (Exploiting Common Limitations of AI Respondents), a novel framework for creating items capable of detecting LLM-generated responses. In the process, I also present evidence demonstrating why detecting careless and synthetic respondents is essential for drawing valid conclusions from online survey research.
Bio:
Cameron S. Kay (he/him) is a postdoctoral scholar in the Climate Cognition Lab at Stanford University. He studies the psychological foundations of antisocial beliefs and behaviours, including conspiracist ideation and climate change denial. To support this work, he develops psychometrically sound scales and tools for improving data quality. Before joining Stanford, Cameron worked as a visiting assistant professor at Union College, completed a PhD in social/personality psychology with a specialization in quantitative research methods at the University of Oregon, and earned a BA in psychology at the University of British Columbia.