W dniu 27 kwietnia 2026 roku odbyło się kolejne seminarium z cyklu PAN-Metrics. Dr Cameron S. Kay zaprezentował swoje badania zmierzające do stworzenia metod do wykrywania syntetycznych respondentów (botów) opartych od duże modele językowe w ankietach online. Zaprezentowane narzędzia pozwalają w prosty i skuteczny sposób na identyfikację odpowiedzi udzielanych przez takie boty.
Slajdy z wystąpienia są dostępne tutaj.
Podsumowanie seminarium:
Dr. Cameron S. Kay (Climate Cognition Lab, Stanford University): Are your online participants human?
Abstract:
Concerns about online data quality have traditionally focused on the possibility that some participants may not be devoting sufficient effort to their responses. However, with the advent and popularization of large language models (LLMs), researchers now face the unsettling possibility that some of their participants may not even be human. In this talk, I share two recent advances in detecting careless and synthetic respondents. First, I introduce CIFR (the Comprehensive Infrequency/Frequency Item Repository), an online database of 660 items designed to identify inattentive respondents. Second, I present ECLAIR (Exploiting Common Limitations of AI Respondents), a novel framework for creating items capable of detecting LLM-generated responses. In the process, I also present evidence demonstrating why detecting careless and synthetic respondents is essential for drawing valid conclusions from online survey research.
Bio:
Cameron S. Kay (he/him) is a postdoctoral scholar in the Climate Cognition Lab at Stanford University. He studies the psychological foundations of antisocial beliefs and behaviours, including conspiracist ideation and climate change denial. To support this work, he develops psychometrically sound scales and tools for improving data quality. Before joining Stanford, Cameron worked as a visiting assistant professor at Union College, completed a PhD in social/personality psychology with a specialization in quantitative research methods at the University of Oregon, and earned a BA in psychology at the University of British Columbia.